Here’s a fun history lesson for you. Don’t worry, we’ll make it quick! In 1966, nearly 51,000 people died on US highways. The overwhelming evidence suggested that many lives could have been saved with modern seat belts, which by then had been around for several years. The automotive industry responded by rallying together, and, in a triumph of self-regulation, voluntarily put the safety feature in every new vehicle that rolled off the line.
Kidding! Of course they didn’t do that. In the fall of 1966, Congress established the Department of Transportation and then passed the National Traffic and Motor Vehicle Safety Act as well as the Highway Safety Act, giving the federal government sweeping new authority to set safety standards. By 1968, seat belts were required in every new car. Despite a massive increase in highway traffic in the last 60 years, far fewer fatalities happen on the road today.
I was thinking about this while watching various AI CEOs huddle around US president Donald Trump on Tuesday, congratulating themselves and each other on having signed an AI safety accord. My colleague Maddy Varner wrote about the specific stipulations here, but the short version is that it’s largely up to the AI labs to keep themselves in check.
Or, as Trump put it even more succinctly in the Oval Office on Wednesday: “They’re gonna police, and they’re gonna police each other, and they’re gonna self-police, they’re gonna police themselves. It’s going to work out very well.”
So this is the current state of AI safety regulation. On one side are the AI executives, employees, and tech luminaries, who are all saying that there need to be clear, thoughtful, tactical rules introduced to avoid potential disaster. (Bill Gates, for instance, has been on a media blitz playing Cassandra about an AI apocalypse.) On the other side is Trump’s magical thinking that if you say something enough times, it must be true.
That’s not to say any of this is simple or easy. The question of how to properly “pace the frontier,” as some of those same executives like to say, remains open. Even if all of the major AI labs were to agree to a slowdown, there’s no good way to keep them to their word, and China would likely continue on its own path regardless. The US economy is increasingly propped up on the AI industry, which itself has become an ouroboros of financing deals and circular investments; any meaningful crackdown risks major instability that reaches well beyond Silicon Valley.
The automotive industry is also admittedly an imperfect analogy for all kinds of reasons. There’s no seat belt equivalent for AI, no straightforward solution that can demonstrably reduce risk overnight. (It also took decades for the majority of states to pass laws mandating that people actually wear them.) The potential harms are less certain and less quantifiable. AI doomers will tell you the technology could result in human extinction in a decade; skeptics argue that’s sci-fi marketing malarkey.
We’ve talked about various AI risk scenarios before, but no one can say for certain what’s going to happen next. If you’re looking for data points to calculate your own personal p(doom), I’d point to a recent CNN report about how the US military supposedly nearly boarded a Chinese ship in part because a false AI-generated report said it was carrying nuclear materials. What, if anything, does the AI accord accomplish to prevent similar incidents from happening in the future? Unclear.
The correct answer to all this uncertainty surely isn’t “I guess we’ll find out.” But that’s exactly what self-regulation encourages. To its credit, OpenAI has delayed and even canceled the release of its most advanced models over security concerns. But not before they went on various, extensive hacking sprees, new details about which are still trickling out at a steady pace.
In an interview with TIME published on Thursday, Trump suggested that the Department of Justice and the Federal Bureau of Investigation were sufficient “guardrails” for the artificial intelligence industry. Are they? It’s far from clear if AI companies can even be held legally responsible when their models go rogue. And it’s not like AI agents themselves are being served subpoenas.
The truth is, no one knows how to solve for AI safety right now. It’s going to take massive amounts of coordination, resources, and regulatory authority to develop the right framework and enforce it, which is exactly the kind of thing governments are designed to provide. Instead of even attempting to work through any of that, the Trump administration is getting out of the way and telling everyone to step on the gas. Don’t bother buckling up.
This is an edition of the Model Behavior newsletter. Read previous newsletters here.
Instagram Engineering
Shopify Engineering