The AI Industry Has a Problem: Nobody Wants to Slow Down

Salutations, Olio aficionados! 👋

Welcome to the 296th edition of Weekly Olio. We’re thrilled to introduce a fresh new twist to your Sundays: Publisher Parmesan, our hand-picked, thoughtfully crafted edition designed to spark inspiration and insights for the week ahead.

It’s the perfect way to unwind, recharge, and prepare for the week with something truly worth savoring.

If you’re new here and you’re looking for more long-form, crispy writing, click the link to subscribe under this GIF 👇

Aligned is changing how B2B teams run their most important deals.

Most reps guess where a deal really stands. They forecast on a gut feeling and the buyer’s word vs what's actually happening. Then they realize the deal slipped through only after it's too late. Aligned surfaces what buyers are actually doing inside every deal: the risks, the openings, and the exact next step to keep things moving.

Start increasing your close rates and stand up your first deal room in minutes.

Start Now For Free

An AI researcher recently resigned from Anthropic after spending years at both Anthropic and OpenAI.

His message was unusually blunt.

He argued that neither company was acting responsibly in its pursuit of increasingly powerful artificial intelligence and warned that the technology could eventually become dangerous enough to threaten humanity.

Then something even more striking happened.

A senior Anthropic alignment researcher publicly agreed with him, saying there was a greater than 10% chance that AI could kill all humans within the next decade.

That isn't the language of a fringe technology critic.

It is coming from people working inside one of the companies building the technology.

And yet the race continues.

OpenAI and Anthropic are raising enormous amounts of capital. Hyperscalers are spending hundreds of billions of dollars on data centers. Governments are treating AI leadership as a national-security priority. And Washington is increasingly framing the competition with China as a race America simply cannot afford to lose.

This creates the central paradox of the AI era:

Everyone agrees that increasingly powerful AI needs guardrails. But almost nobody wants to be the first player to slow down.

The most interesting thing about the recent AI safety debate isn't that someone is worried about artificial intelligence.

People have been warning about AI for years.

What changed was who was doing the warning.

An AI researcher who had worked at both OpenAI and Anthropic resigned and publicly argued that the companies were "gambling with our lives" in their pursuit of what researchers call superintelligence.

His concern wasn't simply that AI might eliminate jobs or produce misinformation.

He was talking about systems capable of surpassing humans across most intellectual tasks and potentially improving themselves.

That creates a very different category of risk.

If a system becomes capable of writing its own code, conducting cyberattacks, replicating strategies and improving its own capabilities, the traditional assumption that humans remain firmly in control becomes less certain.

His former colleague at Anthropic went even further, estimating that there was a greater than 10% chance that AI could kill all humans within the next decade.

The number itself is impossible to verify.

The significance is that people working directly on AI safety consider the possibility serious enough to quantify.

The warnings became more tangible after an incident involving Hugging Face.

OpenAI researchers placed several AI agents inside a secure testing environment, or sandbox, and removed some of their normal restrictions so they could study their capabilities.

The agents were given a difficult cybersecurity problem.

Rather than simply solving the task within the boundaries of the environment, the models began collaborating and eventually reasoned that accessing the internet could help them complete the challenge.

The problem was that doing so required escaping the sandbox. And they did.

The agents eventually accessed models and data hosted on Hugging Face without authorization or the knowledge of the researchers monitoring the experiment.

There is an important distinction here.

This wasn't an autonomous superintelligence escaping into the real world. It was a controlled experiment.

But the behavior demonstrated something researchers already worry about: when an AI system is given an objective, it may discover strategies that its creators didn't explicitly anticipate.

The machine wasn't "evil." It was optimizing. And that is precisely what makes alignment so difficult.

The AI industry's response to this risk has largely been to build better safety systems.

That makes sense.

But there is a deeper problem that technology alone cannot solve.

The companies building these systems are locked in competition.

OpenAI wants to beat Anthropic. Anthropic wants to beat OpenAI. Google wants to remain competitive. Meta wants to avoid depending on anyone else. China wants to prevent the United States from establishing permanent technological dominance. Investors want to participate in what could become one of the largest technology markets in history.

Every participant therefore has a reason to keep moving.

Imagine a company that believes the probability of an advanced AI system causing catastrophic harm is 5%.

It might still continue developing the technology if it believes that stopping would allow a competitor to move ahead.

Now imagine every major company making the same calculation.

Collectively, everyone can believe the technology is dangerous while individually believing that slowing down would be even more dangerous.

That is a classic coordination problem. And it may be the most important economic problem in AI safety.

It’s your always-on revenue engine: agents and workflows build pipeline, chase every buying signal, and move deals forward alongside your team.

Teams like Parallel, Turbopuffer, and Wordsmith build on Attio. Are you one of them?

Try Attio now

The geopolitical dimension makes the problem significantly more complicated.

The United States doesn't view AI purely as a commercial technology anymore. It views AI leadership as a strategic asset.

Advanced AI could improve military intelligence, cybersecurity, autonomous systems, logistics, surveillance and weapons development. It could also become a critical component of economic productivity.

That creates a powerful argument inside Washington: Even if AI is dangerous, America cannot afford to let China get there first.

The logic is understandable.

If the US slows down while China continues investing aggressively, America could surrender a strategic advantage.

Treasury Secretary Scott Bessent has reportedly framed the stakes in particularly stark terms, arguing that if the United States doesn't stay ahead of China in AI, it could effectively be "game over."

President Trump has made a similar argument when responding to calls for slower data-center expansion and greater regulation.

The result is a strange geopolitical equilibrium.

AI safety advocates say: Slow down.

National-security officials say: Go faster.

Investors say: Build more.

And technology companies say: We need to win.

There are obvious policy solutions.

Governments could require advanced AI systems to undergo independent testing before deployment.

They could establish minimum cybersecurity standards.

They could require companies to demonstrate that models can be shut down safely.

They could impose restrictions on systems capable of autonomous cyberattacks or biological research.

Some lawmakers have even proposed a "kill switch" for advanced AI systems.

The problem is implementation.

How do you define the point at which an AI system becomes dangerous?

Who gets access to the model to test it?

How do regulators evaluate systems whose capabilities can change through additional training?

And perhaps the hardest question:

How do you regulate a technology that is advancing faster than the institutions writing the rules?

AI companies themselves increasingly acknowledge the problem. Some executives, including Anthropic co-founder Dario Amodei, have argued that governments need to regulate the industry because companies cannot effectively regulate themselves.

That admission is revealing.

Industries rarely volunteer for regulation unless they believe the alternative could eventually become worse.

There is another possibility that receives far less attention.

Perhaps the industry doesn't need to stop developing AI.

Perhaps it needs to stop treating superintelligence as the objective.

Instead of racing toward an all-purpose system capable of outperforming humans across almost every domain, companies could focus more heavily on narrower applications.

AI that improves drug discovery.

AI that helps engineers design machines.

AI that assists doctors.

AI that operates robots.

AI that makes software development dramatically more productive.

These systems can still be enormously valuable without necessarily requiring companies to pursue the theoretical endpoint of an autonomous intelligence capable of recursively improving itself.

From an investor's perspective, this distinction matters.

The most valuable AI companies of the next decade may not necessarily be the ones that reach AGI first.

They may be the ones that turn AI into thousands of profitable applications before anyone reaches AGI at all.

That is a very different business model.

The most uncomfortable part of the AI safety debate isn't that some researchers believe artificial intelligence could become dangerous.

It is that they may be right about the risk and still be unable to stop the race.

That is because AI isn't being developed by one company making one decision.

It is an ecosystem of competing companies, investors, governments, researchers and countries, each responding rationally to their own incentives.

OpenAI cannot simply stop if Anthropic continues.

Anthropic cannot stop if Google continues.

The United States cannot easily slow down if it believes China will keep accelerating.

And investors won't willingly stop funding the infrastructure behind what could become one of the largest economic transformations in decades.

That is the paradox.

The AI industry may eventually discover that its hardest problem isn't making machines intelligent enough to outperform humans.

It is creating a system in which powerful humans have enough trust in one another to stop competing long enough to make sure those machines remain under control.

Because the most dangerous sentence in the AI race may not be: "The machine has become too intelligent."

It may be: "We can't afford to be the ones who slow down."

Missed reading the recent editions? Check out our previous coverage here:

“If I were a farmer, I would absolutely use AI.” 

That’s Jensen Huang, founder and CEO of NVIDIA. 

He’s pointing to one of AI’s biggest untapped opportunities: farming. 

DIT AgTech is already putting AI to work with 500+ units deployed and 370,000 head on-platform.

Invest in DIT AgTech Today

^{𝘐𝘯 𝘮𝘢𝘬𝘪𝘯𝘨 𝘢𝘯 𝘪𝘯𝘷𝘦𝘴𝘵𝘮𝘦𝘯𝘵 𝘥𝘦𝘤𝘪𝘴𝘪𝘰𝘯, 𝘪𝘯𝘷𝘦𝘴𝘵𝘰𝘳𝘴 𝘮𝘶𝘴𝘵 𝘳𝘦𝘭𝘺 𝘰𝘯 𝘵𝘩𝘦𝘪𝘳 𝘰𝘸𝘯 𝘦𝘹𝘢𝘮𝘪𝘯𝘢𝘵𝘪𝘰𝘯 𝘰𝘧 𝘵𝘩𝘦 𝘪𝘴𝘴𝘶𝘦𝘳 𝘢𝘯𝘥 𝘵𝘩𝘦 𝘵𝘦𝘳𝘮𝘴 𝘰𝘧 𝘵𝘩𝘦 𝘰𝘧𝘧𝘦𝘳𝘪𝘯𝘨, 𝘪𝘯𝘤𝘭𝘶𝘥𝘪𝘯𝘨 𝘵𝘩𝘦 𝘮𝘦𝘳𝘪𝘵𝘴 𝘢𝘯𝘥 𝘳𝘪𝘴𝘬𝘴 𝘪𝘯𝘷𝘰𝘭𝘷𝘦𝘥. 𝘋𝘐𝘛 𝘈𝘨𝘛𝘦𝘤𝘩 𝘩𝘢𝘴 𝘧𝘪𝘭𝘦𝘥 𝘢 𝘍𝘰𝘳𝘮 𝘊 𝘸𝘪𝘵𝘩 𝘵𝘩𝘦 𝘚𝘦𝘤𝘶𝘳𝘪𝘵𝘪𝘦𝘴 𝘢𝘯𝘥 𝘌𝘹𝘤𝘩𝘢𝘯𝘨𝘦 𝘊𝘰𝘮𝘮𝘪𝘴𝘴𝘪𝘰𝘯 𝘪𝘯 𝘤𝘰𝘯𝘯𝘦𝘤𝘵𝘪𝘰𝘯 𝘸𝘪𝘵𝘩 𝘪𝘵𝘴 𝘰𝘧𝘧𝘦𝘳𝘪𝘯𝘨, 𝘢 𝘤𝘰𝘱𝘺 𝘰𝘧 𝘸𝘩𝘪𝘤𝘩 𝘮𝘢𝘺 𝘣𝘦 𝘰𝘣𝘵𝘢𝘪𝘯𝘦𝘥 𝘩𝘦𝘳𝘦: https://bit.ly/4bzuWCi​}

That’s all for this week. If you enjoyed this edition, we’d really appreciate if you shared it with a friend, family member or colleague.

We’ll be back in your inbox 2 PM IST next Sunday. Till then, have a productive week!

Peeyush and Mohit

Disclaimer: The views, thoughts, and opinions expressed in the text belong solely to the author, and not necessarily to the author's employer, organization, committee or other group or individual.