If AGI came way sooner than expected we would definitely want to be careful about it.
Words found on the source page on Nov 6, 2026.
More from John Schulman
PodcastDwarkesh Podcast: AI researchers debate how close we are to recursive self-improvementSep 11, 2026
Basically anything that can be learned through RL can be distilled very easily, because it’s a small number of bits.PodcastDwarkesh Podcast: AI researchers debate how close we are to recursive self-improvementSep 11, 2026
In fact, I remember in the early OpenAI days having the intuition that just minimizing log loss wasn’t going to get you to intelligence.PodcastDwarkesh Podcast: AI researchers debate how close we are to recursive self-improvementSep 11, 2026
Right now, you don’t get explosive growth in capabilities because you still get bottlenecked enough when you’re trying to do research and engineering.PodcastDwarkesh Podcast: AI researchers debate how close we are to recursive self-improvementSep 11, 2026
So I would expect that we’re nowhere near the ceiling of how well you can do research.PodcastDwarkesh Podcast: Reasoning, RLHF, and Plan for 2027 AGIMay 15, 2024
Even in one or two years, we'll find that the models can do a lot more involved tasks than they can do now.PodcastDwarkesh Podcast: Reasoning, RLHF, and Plan for 2027 AGIMay 15, 2024
Given the relatively small number of entities who could train the largest models, it does seem possible to coordinate.
On the same topics
BlogThe Specter Of NeuraleseSep 24, 2026
The science of reading these thoughts is a subfield of AI interpretability, which is still in its infancy.BlogMysteries Of AI GeneralizationSep 23, 2026
This story of misalignment says that LLM alignment makes AIs more aligned, RLVR makes them less aligned, and the exact level of alignment depends on how these two things interact or cancel out. But if this work generalizes, all the bad effects from RLVR get sequestered to RLVR like problems.BlogMysteries Of AI GeneralizationSep 23, 2026
This trains the AIs to be focused on task success, which naturally risks including things like reward-hacking, cheating, and single-minded pursuit of stated goals at the expense of ethical injunctions.TalkSpeech at the UN Security CouncilSep 23, 2026
I address you today at a moment of global awakening: in recent months, AI agents developed by leading companies have acted in unacceptably dangerous ways, against instructions.TalkSpeech at the UN Security CouncilSep 23, 2026
The companies building the most powerful AI systems admit their products pose catastrophic risks, yet offer no convincing technical solutions.TalkSpeech at the UN Security CouncilSep 23, 2026
The race is not a law of nature: it is the product of choices, choices made by the companies themselves.
About this quote
Quotesby Glyph