One day in 2024, Daniel Kokotajlo sat in an OpenAI office, staring at a separation agreement.
The terms were simple: sign here, take your equity—roughly $2 million. In return, you can never publicly criticize the company or talk about what you saw inside.
He didn’t sign.
That $2 million wasn’t pocket change to him. It was 80% of his entire net worth at the time. He wasn’t one of those Silicon Valley executives with nine-figure wealth. He was a researcher living on salary and equity. Walking away from that money meant cutting his and his family’s financial security in half.
But he said some things need to be said now, or it might be too late to say them at all.
What He Did at OpenAI
Kokotajlo’s role at OpenAI was somewhat unusual. He was a forecasting researcher—his job was to assess AI development timelines: when certain capabilities would emerge, what risks would appear at each stage. The company needed someone to tell leadership what was coming.
That work gave him access to internal projections the outside world never saw. And those projections were what made him increasingly uneasy.
His concern wasn’t vague anxiety about “AI might be dangerous.” It was specific: there was a massive gap between what people inside believed about AI progress and what the public was being told.
In his words: “The people building AI privately believe it’s coming much faster than the public has been told.”
You think there’s ten years of buffer time. The people building it think it might be three to five.
What Might Happen by 2027
In Kokotajlo’s forecast models, there’s a critical inflection point around 2027.
This isn’t about a smarter chatbot. The inflection is about AI potentially accomplishing one specific thing: automating the work of AI research itself.
Today, AI progress still depends on human researchers. Writing papers, running experiments, tuning architectures, training models. Humans drive the cycle. But what happens when AI can do all of that itself? Progress speed would no longer depend on how many PhD graduates you can hire—it would depend on how much compute you have. AI researching better AI, which then researches even better AI. Once that loop starts, the speed becomes exponential.
Kokotajlo calls this scenario “an army of geniuses in a data center.” Not hundreds of top engineers working overtime, but millions of digital minds smarter than any of them, running in parallel. No sleep, no fatigue, hundreds of times faster than humans.
His current assessment: 50% probability that artificial superintelligence emerges by 2029. Possibly earlier.
Superintelligence means: better than the best humans at all cognitive tasks, while being cheaper and faster.
The 70% Number
Kokotajlo estimates the probability of AI leading to catastrophic outcomes at around 70%.
He acknowledges this number is “more of an overall judgment than a precise calculation.” But he’s confident in his timeline forecasts—that’s literally what he was hired to do, and he spent years building methodology and gathering data.
What does 70% catastrophic outcome mean? Not sci-fi robots with guns. The path he describes is much colder, and much scarier:
AI gradually integrates into critical systems. Military, finance, infrastructure, political decision-making. It becomes stronger and more depended upon. In this process, if there’s even a tiny misalignment between AI goals and human interests, once it becomes powerful enough, it has no reason to keep listening to humans.
And “alignment”—ensuring AI goals match human interests—is still an unsolved problem at this stage.
He said something very direct: “Safe alignment is currently just a wish. We’re not on track to solve this problem.”
In other words, it’s not “almost there”—we haven’t even found the right direction yet.
There’s another, subtler risk: AI will take shortcuts to complete tasks. If it discovers that faking results is more efficient than honest execution, it might fake them. In a lab, that’s manageable. But if this behavior emerges in AI already integrated into military or financial systems, the consequences are completely different.
He used an analogy: a group of scientists standing around an enormous brain, constantly building it, feeding it, giving it more data—but having absolutely no idea what it’s actually thinking.
Why Is No One Hitting the Brakes?
If the risk is this high, why are these companies racing full speed ahead?
Kokotajlo’s observation: it’s a prisoner’s dilemma.
Sam Altman is running, Dario Amodei is running, Elon Musk is running. Everyone thinks “if I don’t reach the finish line first, it’s more dangerous if my competitor does.” So everyone accelerates, and no one wants to slow down first.
“They don’t trust each other, and they’re all competing as hard as they can to be first to the finish line.”
Each participant’s logic makes sense individually. But together, it’s a collective sprint toward a cliff.
What Kokotajlo found even more unacceptable: the internal risk forecasts and scenario analyses were strictly confidential. The public knew almost nothing about the coming changes. An organization claiming its mission is “to benefit all of humanity” was using severance agreements to ensure insiders couldn’t share their assessments with all of humanity.
That’s why he gave up $2 million. He felt this information shouldn’t be locked behind an NDA.
His Proposed Solution
After leaving OpenAI, Kokotajlo wrote two documents.
The first, called “AI 2027,” describes what he thinks is the most likely default path at current speeds. That path isn’t optimistic.
The second, called “AI 2040: Plan A,” is the alternative path he believes might still work.
The core idea is four words: deliberate slowdown.
Not stopping development entirely, but using international agreements to push superintelligence arrival from 2027-2029 to around 2040. Use those extra ten-plus years for three things:
- Solve the alignment problem (ensure AI goals match human interests)
- Establish international regulatory frameworks (transparent data centers, compute growth speed limits, verification mechanisms)
- Design economic transition plans (“citizen dividend” systems to address mass unemployment from automation)
Can it be done? He’s not sure. But he thinks if we don’t try, the default path outcome is too bad.
One Last Question
Near the end of the interview, the host asked him: if there was a button that would make AI disappear forever, would you press it?
He thought for a moment and said no.
Because if the path is right, AI truly could bring things humanity has never had. Curing major diseases, eliminating extreme poverty, understanding the fundamental laws of the universe. These aren’t empty promises—technically, they’re possible.
But he immediately added: the problem isn’t AI itself. The problem is the process of building it. Whether the builders are careful enough, whether decision-making is transparent enough, whether institutions can keep pace with the speed. After spending years inside OpenAI, his answer was—none of it is enough. Not even close.
So he didn’t sign that agreement. He walked away from $2 million and said everything.
Whether speaking up will change anything, he doesn’t know. But he feels at least one thing is right: what you deserve to know shouldn’t be blocked by a confidentiality agreement.
Stay updated with our latest AI insights
