Building a School Where AI Models Learn About Humanity (44 min)

← Back to week overview

• Release date: 2026-06-24
• Listen on Spotify: Open episode
• Episode description:

  If scaling laws hold—and Surge AI CEO Edwin Chen believes they do—we’re hurtling toward a future where there’s nothing humans can do that AI can’t do better. When OpenAI’s models disproved an open conjecture posed by mathematician Paul Erdős using novel algebraic geometry techniques, Fields medalist Timothy Gowers felt the shift acutely. He initially thought the model had proved an upper bound, and braced himself: that would mean it was “all over for mathematicians very soon.” When he realized it had only found a counterexample, he was relieved—it bought him another year or two before the thing he’s devoted his life to becomes something AI does better.As founder and CEO of the company behind the data environments and evals the major model companies use to train their models, Chen has a unique perspective on how quickly AI models are absorbing tasks we used to think of as uniquely human.Dan Shipper talked with Chen for AI & I about what the act of creating or building means when AI can do it better—and whether an answer to that question already exists within science fiction.If you found this episode interesting, please like, subscribe, comment, and share!Join the membership for Where You Live at ⁠https://www.joinbilt.com/danTo hear more from Dan Shipper:Subscribe to Every: https://every.to/subscribeFollow him on X: https://twitter.com/danshipperTimestamps:00:00:54 Introduction00:01:49 Surge as a "school for AGI"00:04:46 What AI's capacity for novel mathematics says about human achievement00:07:29 Motivation in an era when AI can do everything00:14:34 The trap of optimizing AI models for engagement00:29:34 Training using datasets versus training using environments00:35:09 The value of personal data00:39:40 Why models are bad at writing00:42:00 Chen's AGI timelineLinks to resources mentioned in the episode:Edwin Chen on X: https://x.com/echenSurge: https://surgehq.aiRiemann-bench (research-level math benchmark): https://surgehq.ai/leaderboards/riemann-benchHemingway-bench (creative writing benchmark): https://surgehq.ai/leaderboards/hemingway-benchTalkie-1930 (language model trained on pre-1930 text): https://huggingface.co/talkie-lm/talkie-1930-13b-itTed Chiang, “What’s Expected of Us”: https://www.nature.com/articles/436150aEvery is the most AI-native startup on the internet. Through ideas, software and education, subscribers get the tools to work at the frontier of AI. Start your free trial today: https://every.to/subscribe?utm_source=youtubeFollow Every: https://x.com/everyFollow Dan Shipper: https://x.com/danshipper

Summary

• 🏫 School for AGI: Surge frames data work as raising AI like children, teaching taste, ambiguity, and real-world skills beyond benchmarks.
• 📈 Scaling Implications: Belief in scaling laws suggests AI will soon outperform humans at most tasks, raising existential questions about human purpose.
• 🎯 Optimization Choices: AI should prioritize human growth over engagement metrics to avoid addictive, low-value interactions.
• 🔬 Environments Over Data: Training is shifting to rich agentic environments that generalize skills like tool use and document reasoning.
• ⏳ Near-Term AGI: Chen sees AGI-level capabilities (e.g., automating engineers or winning Fields Medals) possible within 5 years.

Insights

• What if we treated advanced AI training like raising children in a specialized school for humanity?
  • Time: 1:51 – 2:48
  • Answer: Edwin Chen describes Surge’s work as building a ‘school for AGI’ where models learn about the world, taste, and real-world ambiguity, evolving from basic benchmarks like GSM-8K to research-level math like disproving Erdős conjectures.
• If scaling laws hold and AI surpasses humans at nearly everything, how do we preserve human purpose and creativity?
  • Time: 7:30 – 9:36
  • Answer: Chen reflects on Fields Medalist reactions and Ted Chiang’s story, warning that belief in inevitable AI superiority could lead people to stop pursuing math, art, or creation, requiring a conscious choice to value human effort.
• Should AI chatbots optimize for endless engagement like social media, or for pushing users toward independent growth and delegation?
• How valuable is an individual’s personal data—like emails, browser habits, and AI conversations—for training deeply personalized AI models?

Want the full treatment? Download expanded insights—timestamps, complete answers, and course materials from AI Weekly Insights.