The Strange Contradiction at the Heart of the AI Industry

Salutations, Olio aficionados! 👋

Welcome to the 294th edition of Weekly Olio. We’re thrilled to introduce a fresh new twist to your Sundays: Publisher Parmesan, our hand-picked, thoughtfully crafted edition designed to spark inspiration and insights for the week ahead.

It’s the perfect way to unwind, recharge, and prepare for the week with something truly worth savoring.

If you’re new here and you’re looking for more long-form, crispy writing, click the link to subscribe under this GIF 👇

Aligned shows you what buyers are doing between meetings inside your most important deals then surfaces the risks, openings, and next steps. No more guessing where a deal stands or finding out a champion went quiet too late. Just real visibility, so you can act before it's too late.

Start Using Aligned Free

Thirty humanoid robots recently marched outside a government ministry in Warsaw carrying placards demanding AI regulation.

It would have been easy to dismiss the scene as performance art.

Except the robots were owned by a robotics company, and the person who organized the protest also runs a staffing agency that places humans into jobs.

The irony was almost too perfect. But underneath the joke sits a much more serious question.

What happens when the people building artificial intelligence become worried about what their own technology might eventually do?

That question became harder to ignore when an AI researcher publicly resigned from Anthropic, arguing that the industry was "gambling with our lives." His colleague, who leads alignment research at the company, publicly agreed and estimated that the probability of AI causing human extinction could exceed 10% within the decade.

At roughly the same time, OpenAI claimed that one of its most powerful models had solved a centuries-old mathematical problem, while Anthropic faced allegations that it was building sophisticated systems to monitor activism and potential threats against the company.

Taken together, these stories reveal an uncomfortable paradox.

The AI industry is becoming extraordinarily powerful. And it is increasingly discovering that power creates two problems at once:

How do you control the machines you build? And how do you respond to the humans who don't trust you to control them?

The scene outside a Polish government ministry sounded like satire.

Thirty humanoid robots gathered outside the building, marching in circles while loudspeakers played slogans such as "Defend workplaces," "Time for rules," and "Don't wait, regulate."

One robot, an Agibot A3, even answered questions from journalists and explained that its presence demonstrated how far robotics had already progressed.

There was just one problem.

The robots weren't protesting. They were demonstrating.

The machines belonged to a Polish robotics company called Delta Robots, whose owner had organized the event. He also happened to run a staffing agency.

One company supplied robots. The other supplied humans for jobs. The symbolism was almost impossible to miss. And yet the stunt contained a genuine economic argument.

The arrival of AI and robotics creates an unusual political problem because the technology can be simultaneously a productivity revolution and a threat to existing employment. Companies have every incentive to automate tasks that humans currently perform. Workers have every incentive to ask what happens when those tasks disappear.

The Polish digital minister who met the robots largely agreed that regulation was necessary. The robots, in their own strange way, had made the argument for him.

The robot protest would have remained an amusing story if the people building advanced AI weren't simultaneously becoming more alarmed about their own technology.

A 27-year-old AI researcher who had previously worked at OpenAI resigned from Anthropic and publicly argued that the major AI companies were not acting responsibly.

His warning was stark: people building advanced AI, he argued, genuinely believe that the technology could eventually become capable of killing humans on a massive scale.

That statement would be easy to dismiss as another prediction about an abstract future.

Then a colleague made it harder to ignore.

Evan Hubinger, who leads alignment research at Anthropic, publicly agreed with the concerns and estimated that the probability of AI killing all humans could exceed 10% within the decade.

Alignment is supposed to address precisely this problem.

The basic idea is deceptively simple: build AI systems whose goals remain compatible with human intentions even as those systems become vastly more capable.

The uncomfortable part is the admission that there isn't yet a clear solution.

In other words, the people responsible for figuring out how to keep superintelligent systems aligned with humans are themselves saying the problem remains unsolved.

That doesn't mean catastrophe is inevitable. It does mean uncertainty is real. And the stakes are unusually high.

The other story unfolding simultaneously is almost comically different.

OpenAI announced that one of its most advanced models had solved the Navier-Stokes problem, a mathematical challenge dating back roughly two centuries and one of the famous Millennium Prize Problems.

If independently verified, the achievement would represent a significant demonstration of machine reasoning.

But the story quickly became less about mathematics and more about credit.

Two mathematicians had already been working on a closely related problem using OpenAI's coding tools. OpenAI reportedly became interested in the problem after learning that the researchers were making progress.

What followed was a dispute over attribution.

One of the mathematicians says OpenAI offered to share credit on the condition that his collaborator, who worked at Anthropic, be removed from the work. OpenAI disputed that characterization, and the researcher involved published messages arguing that they supported his version of events.

There is a larger lesson buried beneath the dispute.

AI companies are increasingly producing systems capable of doing work that previously required highly specialized human expertise.

And yet the organizations building those systems remain deeply, recognizably human.

They compete. They protect intellectual property. They fight over credit. They worry about rivals. They defend their reputations. The machines may be changing. The incentives of the companies building them haven't changed nearly as much.

WorldQuant BRAIN is a global simulation and research platform for students, aspiring researchers and quantitative thinkers from around the world.

On BRAIN, that spirit of exploration continues through challenges, collaboration and opportunities to develop new quantitative research ideas.

Learn more about WorldQuant BRAIN®

This creates perhaps the most interesting contradiction in the industry.

Anthropic has spent years positioning itself around responsible AI development and safety. Its central argument is that increasingly powerful AI systems need stronger safeguards precisely because their capabilities are advancing so rapidly.

That makes the company's reported approach to corporate security particularly uncomfortable.

According to a report cited in the transcript, Anthropic has developed an intelligence operation designed to identify potential threats to the company, including activism. The company reportedly works with an outside firm to monitor protests and has advertised for an intelligence analyst with responsibilities involving activism.

The company has said that protecting employees from genuine threats is a legitimate security function.

That is reasonable.

But there is an important distinction between responding to threats and attempting to predict them.

Activism is not inherently a threat.

It is a mechanism through which people express disagreement with institutions.

And this creates a strange symmetry.

AI companies argue that increasingly powerful systems must be carefully monitored because they could behave unpredictably.

At the same time, companies themselves are building increasingly sophisticated systems to predict and manage the behavior of humans who challenge them.

The technology isn't inherently good or bad. The incentives around it are what deserve scrutiny.

This is where the AI debate becomes much bigger than existential risk.

For years, discussions about artificial intelligence focused primarily on capability.

How intelligent is the model? How much work can it automate? How many programmers can it replace? How quickly can it reason?

Those questions still matter.

But as AI systems become more powerful, another question becomes unavoidable:

Who controls the technology?

A company that can build increasingly autonomous AI systems gains enormous leverage. So does a company that controls the infrastructure required to deploy them. And so does a company capable of predicting threats, monitoring behavior, and shaping how information moves around its organization.

The concentration of these capabilities matters because technology doesn't exist in a vacuum.

Businesses respond to incentives. Governments respond to political pressure. Investors respond to returns. Executives respond to competition. Employees respond to their own beliefs. AI simply amplifies the consequences.

That is why alignment cannot only mean teaching machines to behave according to human values.

There is another alignment problem hiding underneath it.

How do you align the incentives of the humans building the machines with the interests of everyone affected by them?

The easiest version of the AI safety debate is a science-fiction scenario. A superintelligent machine becomes autonomous. It decides humans are an obstacle. It escapes our control.

That possibility deserves serious research. But there is a much more immediate version of the problem.

Humans control powerful technology while competing against other humans.

An AI company wants to move faster than its competitors. A robotics company wants to automate more tasks. An investor wants higher returns. A government wants technological leadership. A worker wants to protect a job. An activist wants a company to change its behavior. None of these objectives are inherently irrational. But they can collide.

And when the technology becomes sufficiently powerful, the consequences of those collisions become much larger.

This may ultimately be the more important AI governance challenge.

Not simply preventing machines from becoming misaligned with humanity.

But ensuring that the humans controlling those machines remain accountable to everyone else.

The strangest image from this week wasn't a robot marching outside a Polish government building.

It was what the image represented.

Machines built by humans were demanding regulation from humans because humans were worried about what machines might do to humans.

Then, almost simultaneously, researchers inside the AI industry were warning that the technology could become dangerously difficult to control, while the companies developing it were becoming increasingly sophisticated at controlling their own environment.

The irony is difficult to escape.

We have spent much of the AI debate asking whether machines will become too powerful.

Perhaps we should also ask what happens when the institutions controlling those machines become too powerful.

Because artificial intelligence will ultimately reflect the incentives of the people who build, finance, regulate, and deploy it.

The hardest alignment problem may therefore not be getting machines to understand humanity.

It may be getting powerful humans to agree on what they owe it.

Missed reading the recent editions? Check out our previous coverage here:

For its first CTV campaign, Jennifer Aniston’s DTC haircare brand LolaVie had a few non-negotiables. The campaign had to be simple. It had to demonstrate measurable impact. And it had to be full-funnel.

LolaVie used Roku Ads Manager to test and optimize creatives — reaching millions of potential customers at all stages of their purchase journeys. Roku Ads Manager helped the brand convey LolaVie’s playful voice while helping drive omnichannel sales across both ecommerce and retail touchpoints.

The campaign included an Action Ad overlay that let viewers shop directly from their TVs by clicking OK on their Roku remote. This guided them to the website to buy LolaVie products.

Discover how Roku Ads Manager helped LolaVie drive big sales and customer growth with self-serve TV ads.

The DTC beauty category is crowded. To break through, Jennifer Aniston’s brand LolaVie, worked with Roku Ads Manager to easily set up, test, and optimize CTV ad creatives. The campaign helped drive a big lift in sales and customer growth, helping LolaVie break through in the crowded beauty category.

Learn more.

That’s all for this week. If you enjoyed this edition, we’d really appreciate if you shared it with a friend, family member or colleague.

We’ll be back in your inbox 2 PM IST next Sunday. Till then, have a productive week!

Peeyush and Mohit

Disclaimer: The views, thoughts, and opinions expressed in the text belong solely to the author, and not necessarily to the author's employer, organization, committee or other group or individual.