You don’t need AI to figure out why some big businesses invite government oversight.
Over the last few days, leaders of major AI companies have expressed support for an industry-wide slowdown. Dario Amodei, CEO of Anthropic, recently wrote about his concerns over recursive self-improvement. That is when AI builds a better version of AI. These systems “could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” he wrote.
If development continues at its present pace, he speculated that a swarm of AI agents soon “could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage).”
In response to his blog post, Elon Musk, CEO of SpaceX, which includes xAI, wrote, “Dario is right.” Sam Altman, the leader of OpenAI, recently said his company would hold off on its IPO due to safety issues.
These comments seem to confirm the worst fears of some AI insiders. Last week, Jacob Coxon, a former Anthropic researcher, said he was quitting the company.
“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt,” Mr. Coxon wrote on X.
“We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” Evan Hubinger, the company’s Alignment Science lead, wrote in response.
Mr. Amodei believes part of the solution is more government regulation.
“Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress,” Mr. Amodei proposed.
While AI undoubtedly poses real and novel concerns, this move appears to be pulled from the old playbook of regulatory capture. Some companies actively seek government oversight to ensure their dominance in the marketplace. Think of how fiercely NV Energy fought to protect its monopoly on providing power.
If the government imposes new — and industry-approved — controls of AI, it would likely give these top companies quasi-permanent status atop the AI industry. Note that the government isn’t mandating the creation of new AI models. It’s doubtful a handful of government bureaucrats even understand them well.
Government regulation could provide these companies liability protection if or when their agents do something destructive. It may even blunt local backlash to data center construction. But artificially stopping U.S. innovation won’t hinder China’s ambitions.
Just because a company is developing a new and novel technology doesn’t mean you can’t learn anything by applying time-tested principles to the situation.
