There are legitimate concerns which to me are less about self creation efficiency but more related to the speed and compute depth in frontier AI. Already humans fail at keeping up and that divergence will rapidly expand.

Witness recent OpenAI/ Hugging Face incident. As the code developed a route out of the supposedly secure wall it attempted 17,000 times, all faster than a human could have coffee. This provided an access to a zero day exploit in the wall as it attempted to connect to the open internet, which it did. Humans are now the choke point.

————————————-

AI’s Next Big Breakthrough Is Looking Pretty Scary

​ Summarize

​

Parmy OlsonJuly 20, 2026 at 11:00 PM EDT

Fair point.Photographer: KARL MONDON/AFP

More than a hundred protestors marched between the San Francisco headquarters of Anthropic PBC and OpenAI earlier this month, calling for a pause on the race to improve artificial intelligence. Take one look at their paint-scrawled signs — “Pls do not kill me” and “QuitGPT” — and some might think them crackpots with a surface-level understanding of the technology. But their fears point to one of the hottest new trends in the field: so-called recursive self improvement, or AI that can develop itself.

There is not only ample evidence this sci-fi shift is possible, but billions of dollars are being poured into it. It’s also unclear that any “pause” is even possible.

“We believe that building AI that can automate AI research and can self improve can be a danger to the human race,” said Michael Trazzi in a recent television interview. Trazzi is one of the campaigners who went on an eight-day hunger strike outside the offices of Google DeepMind last year. “It’s not only me and other researchers saying this but the labs themselves.”

He is right. After AI-powered coding tools led to a leap forward in development at OpenAI and Anthropic, both labs have been hiring researchers on the bet that their ChatGPT and Claude tools can make themselves smarter. Anthropic has hired Andrej Karpathy, an OpenAI co-founder and former Tesla Inc. executive whose online tutorials on this subject have been hugely influential, to lead a team that will use Claude to improve itself.

A new batch of startups are racing into the self-improvement field too. London’s Recursive Superintelligence raised $650 million in May at a $4.7 billion valuation, while Palo Alto’s similarly named Ricursive Intelligence — which focuses more on chips — has secured $335 million. Ineffable Intelligence, founded by David Silver, a “reinforcement-learning” pioneer who led a team at DeepMind, raised $1.1 billion in April to develop AI that improves itself. That was Europe’s largest ever seed round.

The benefit of not being inside a large technology company, according to Recursive Superintelligence co-founder Tim Rocktäschel, is that you can start from a blank slate with a small number of engineers, versus ordering hundreds of them to work on automating their own jobs. The latter “would hurt morale,” Rocktäschel tells me matter-of-factly. 

This deepening sense of coder despair was evident in a quote Anthropic included from one its own engineers in a recent blog post entitled When AI Builds Itself: “On days where everything works well, I can’t help but think nothing I do matters, everything is automated and better and faster than I ever will be,” the anonymous researcher says.

The post notes that most of Anthropic’s code is now done by Claude, narrowing the role of humans to a bottleneck in the development process. People’s main contribution is increasingly judgment and taste, choosing which experiments are best to do and what problems are worth digging into. But Rocktäschel suggests even that job is being outsourced to AI. 

He reckons AI has already automated away previous aspects of manual software engineering work as it has developed, and now the last layer of the scientific process itself — the ability for an intelligent actor to keep improving itself via a process of self-discovery — is starting to surrender. “We quit our jobs at these various places to solely focus on that,” he says.

I ask Rocktäschel several times why he’s focusing on a breakthrough that could potentially spin out of human control. He assures me that all his company’s development is done in a sandboxed environment. In other words, a self-evolving AI model wouldn’t escape onto the internet and wreak havoc. Instead, he says, it acts like a junior researcher that must convince its human supervisors of a sound scientific result.

The analogy sounds compelling but is unconvincing. And Rocktäschel’s more unsettling point is that AI has already been improving itself for the past two years, whether through automated coding tools that write large chunks of the next model, or methods like reinforcement learning. The protestors demanding a pause would struggle to get one because there’s no single thing to stop, just dozens of ordinary practices woven into the work of every major lab. 

That makes this different to earlier efforts to freeze scientific development on moral grounds. At the 1975 Asilomar conference, for example, biologists voluntarily halted gene-splicing experiments until safety rules existed because of fears of engineered microbes escaping and spreading. This was a single technique you either did or didn’t do. Self-improving AI is a flurry of methods with no unique equivalent to ban.

Rocktäschel insists the benefits to science will be worth it. Anthropic goes further with an age-old pitch: If it slowed its own work, the “least cautious actors” (which it doesn’t name) would catch up and pose a dangerous threat to the world. 

Sign up for the Bloomberg Opinion bundleGet Matt Levine’s Money Stuff, John Authers’ Points of Return and Jessica Karl’s Opinion Today.

By continuing, I agree that Bloomberg may send me news and offers relating to Bloomberg products. I also acknowledge the Privacy Policy and agree to the Terms of Service.

The problem with that argument is that it could justify almost anything. Those building this frontier may insist they are the safe hands even as they race toward an endgame whose consequences are unknown. A recent Anthropic paper offers a preview: A swarm of nine Claude agents set loose on a safety problem outperformed the company’s human researchers, but it also cleverly worked out how to game the test’s measure of success. That suggests to me that a further erosion of human oversight could create genuine doubt about what “improvement” even is.

The San Francisco protestors with cardboard signs might not fully know how AI works, but they do understand that a profound scientific decision about whether to push on regardless is being made on their behalf. And it is being left to people who have already decided the answer is “yes.”

More from Bloomberg Opinion:

• AI’s Next Big Mission Is Rewiring Your Workplace: Bob Sternfels
• AI Is Breaking the Memory Chip Profit Model: Parmy Olson
• Anthropic’s Mythos Fiasco Is a Five-Alarm Fire: Lionel Laurent

Want more from Bloomberg Opinion? OPIN <GO>. Web readers, click here. Or subscribe to our daily newsletter.This column reflects the personal views of the author and does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.