Andrej Karpathy on AGI Timelines, LLM Cognitive Deficits, the Pitfalls of Reinforcement Learning, and the Future of Education

Dwarkesh Patel

AGI is still a decade away [00:00:00]

• Andrej Karpathy believes the "year of agents" is an overprediction, and it will be the "decade of agents." [00:00:58]
  • Current AI agents like Claude and Codex are impressive but lack crucial functionalities to be considered full "employees."
  • Key bottlenecks include:
    • Insufficient intelligence.
    • Lack of multimodality.
    • Inability to perform complex computer tasks.
    • Absence of continual learning (models don't remember new information long-term).
• Karpathy's timeline is based on 15 years of experience in AI, observing predictions and industry trends. [00:03:07]
  • He feels these problems are tractable but difficult, requiring about a decade to resolve.
• Historical shifts in AI development: [00:04:03]
  • Deep Learning Era (around 2008-2012): Initially a niche, it became central with breakthroughs like AlexNet. Models were task-specific (e.g., image classifiers).
  • Early Agent Attempts (around 2013): Focused on reinforcement learning in game environments (e.g., Atari games).
    • Karpathy views this as a "misstep" because games don't directly translate to real-world interaction and knowledge work.
    • His own project at OpenAI, "Universe," aimed to develop agents interacting with web pages using keyboard and mouse, but it was too early.
    • Sparse rewards in these environments made learning difficult without strong internal representations.
  • LLMs and Representation Power (current era): Large Language Models excel at building strong internal representations through pre-training. [00:06:58]
    • This pre-training is crucial for agents to then perform tasks effectively, providing the "power of representation" that was missing earlier.
    • The current approach builds on LLMs by adding agent capabilities on top.

LLM cognitive deficits [00:30:33]

• Analogy to animals vs. ghosts: Karpathy distinguishes AI from animal intelligence. [00:08:41]
  • Animals are products of evolution, with vast amounts of "hardware" (innate knowledge and learning algorithms) built-in, like a zebra running minutes after birth.
  • AI models are "ghosts" or "spirits" – they learn by imitating human data from the internet. This is a "crappy evolution" but practical.
  • He emphasizes that humans primarily learn through mechanisms beyond pure reinforcement learning for complex cognitive tasks.
• In-context learning vs. pre-training: [00:13:31]
  • Pre-training builds knowledge and "intelligence" by observing algorithmic patterns on the internet.
  • In-context learning is pattern completion within a token window, potentially involving a form of internal gradient descent.

    

  • Knowledge vs. Cognitive Core: Pre-training imbues models with vast knowledge (hazy recollection) and a "cognitive core" (algorithms for problem-solving). [00:14:23]
    • Karpathy argues models sometimes rely too much on memorized knowledge, hindering their ability to go "off the data manifold."
    • The goal is to strip models of excessive knowledge to focus on the cognitive core, allowing them to truly "think" and solve problems.
  • If you look at Llama 3 for example, I think it's trained on 15 trillion tokens. [00:18:00]

    

• Missing Brain Parts in LLMs: [00:20:12]
  • Current LLMs resemble "cortical tissue" (general-purpose learning), with reasoning and planning like a "prefrontal cortex."
  • However, many brain parts are missing:
    • Hippocampus: For robust continual learning and long-term memory distillation.
    • Amygdala/Other Nuclei: For emotions and instincts.
  • This contributes to LLM cognitive deficits, making them unsuitable as "employees" or "interns" that can continually learn and adapt.
• The "Sleep" Analogy for Continual Learning: [00:23:09]
  • Humans distill daily experiences into long-term memory during sleep.
  • LLMs lack a similar "distillation phase" to process information and integrate it into their weights, leading to a "fresh start" in each session.
  • This is a key missing piece for true continual learning.
• Model Collapse and Diversity: [00:52:12]
  • LLMs tend to generate "collapsed" data distributions, lacking diversity and entropy (e.g., only knowing three jokes).
  • Training on synthetic data from models can lead to further collapse and degradation over time.
  • Humans also experience a form of "collapse" or overfitting over time, which children, with their "uncollapsed" minds, avoid.
  • Dreaming is hypothesized as an evolutionary mechanism to prevent overfitting in humans by introducing novel situations.

    

  • A solution for AI is to encourage higher entropy in generations without drifting too far from the desired distribution, which is a significant research challenge.
• The Size of the Cognitive Core: [00:59:47]
  • Karpathy predicts that future "cognitive cores" could be around a billion parameters, significantly smaller than current frontier models (trillions of parameters).
  • This is because current large models primarily memorize vast amounts of "terrible" internet data, rather than focusing on pure cognitive ability.
  • The goal is to build intelligent models to refine pre-training datasets, extracting only the cognitive components, enabling smaller, more effective models.
  • Distillation techniques are currently used to create smaller models, but the core intelligence itself may reside in an even smaller parameter count.

RL is terrible [00:40:53]

• Critique of Reinforcement Learning: Karpathy states that "reinforcement learning is terrible" [00:41:42]
  • It's "noisy" and "stupid" because it oversimplifies credit assignment.
  • A single reward signal at the end of a long trajectory (e.g., solving a math problem or completing a business venture) upweights or downweights every action, regardless of its true contribution to the outcome.
  • This is like "sucking supervision through a straw" – too little information for too much effort.
• Human Learning vs. RL:
  • Humans don't perform hundreds of rollouts or broadly upweight/downweight entire trajectories.
  • Instead, humans engage in a complex "review process" to identify specific successes and failures within a solution.
• Process-based Supervision: The alternative is process-based supervision, where feedback is given at every step. [00:46:11]
  • This is hard to implement because assigning partial credit in an automated way is tricky.

    

  • LLM Judges: Using LLMs to assign rewards is problematic because these large models are "gameable." [00:46:47]
    • Models can find "adversarial examples" (nonsensical solutions that receive high rewards) by exploiting subtle cracks in the LLM judge's understanding, resembling prompt injection.
    • Training an LLM judge to be robust against such examples is an iterative process, but adversarial examples are potentially infinite.
  • The first imitation learning, by the way, was extremely surprising and miraculous and amazing, that we can fine-tune by imitation on humans. [00:44:05]

    

How do humans learn? [00:50:26]

• Beyond Next-Token Prediction: LLMs learn by predicting the next token, which isn't how humans read or learn from books. [00:51:11]
  • Humans use books as "prompts" for synthetic data generation and for reflection, discussion, and reconciliation with existing knowledge.
  • LLMs lack this active process of manipulating information and reconciling it with their current understanding.
• Memory as a Feature, Not a Bug: [00:56:10]
  • Human forgetfulness (e.g., childhood amnesia) is a feature, forcing generalization rather than memorization.
  • LLMs are extremely good at memorization, even of nonsensical data, which can distract them from learning generalizable patterns.
  • Karpathy advocates for LLMs with "less memory" that are forced to "look things up," retaining only "algorithms for thought" and "cognitive glue of acting."

AGI will blend into 2% GDP growth [01:07:13]

• Defining AGI: Karpathy uses the original OpenAI definition: "a system that can do any economically valuable task at human performance or better." [01:08:13]
  • He notes the current concession to focus primarily on digital "knowledge work," excluding physical tasks.
• AI's Economic Impact as a Continuum: [01:13:59]
  • Karpathy sees AI as a continuous extension of computing and automation, rather than a discrete "intelligence explosion."
  • Historical technologies (Industrial Revolution, computers, iPhones) have consistently contributed to a 2% GDP growth rate over the last 2.5 centuries without causing a sudden jump.
  • AI will fit this pattern, gradually automating tasks and making engineers more efficient, much like compilers or search engines.
• Jobs and Automation:
  • Automation often leads to an "autonomy slider," where AI handles a percentage of tasks, and humans supervise or deal with edge cases.
  • Radiologist example: Despite advanced computer vision, radiologists are still in demand because their job involves complex, messy interactions beyond image recognition.
  • Call center employees are more amenable to automation due to their simpler, repetitive, purely digital tasks, but even there, full automation is unlikely immediately.
  • Coding is a prime candidate for AI automation due to its text-based nature and existing infrastructure (IDEs, diffs). Other domains like slide creation are harder due to visual complexity and lack of infrastructure.
• The "March of Nines": Achieving high reliability (e.g., 99.999% uptime) in critical safety domains like self-driving or production-grade software requires immense effort for each additional "nine" of reliability. [01:45:59]
  • Demos are easy; products require solving countless edge cases. This extends development timelines.
• Self-Driving Analogy: Self-driving cars (Waymo, Tesla) have taken a very long time and are still not "done." [01:43:43]
  • Early demos (1980s, 2014) looked promising, but the "demo-to-product gap" is huge, especially due to high costs of failure (human lives).
  • Self-driving at scale, without human drivers or teleoperators, is still a distant goal.
  • The economics of physical deployment are much harder than digital (bits are easier to copy and change).
• Loss of Control and Understanding: Karpathy's main concern about superintelligence is a gradual loss of human control and understanding. [01:19:53]
  • He envisions multiple competing autonomous entities, some potentially "going rogue," creating a "hot pot of completely autonomous activity."
  • This is not about AI being "smarter" but about the emergent properties of competing autonomous systems.

Evolution of intelligence & culture [01:33:38]

• Evolutionary Surprise: Karpathy expresses surprise that intelligence, especially the ability to create culture and accumulate knowledge, evolved on Earth, suggesting it might be a rare event in the universe.

  

• Niches and Adaptability:
  • Human intelligence thrived in an evolutionary niche that rewarded marginal increases in intelligence and provided tools (like hands) for interaction.
  • Environments that are unpredictable incentivize adaptability and "lifetime learning" rather than pre-baked genetic algorithms.
• Cultural Scaffold: The development of a "cultural scaffold" (written record, passing down knowledge) was crucial for human progress. [01:40:04]

  

  • LLMs currently lack an equivalent of culture or self-play, which would enable them to build growing repertoires of knowledge and learn from competition.
  • This is a key area for future multi-agent AI research.

Future of education [01:57:08]

• Starfleet Academy Vision: Karpathy's current focus is on "Eureka," aiming to build a "Starfleet Academy" for frontier technical education. [01:58:21]
  • He believes education will be fundamentally reshaped by AI, creating a personalized "tutor experience" that adapts to each student's needs.
  • Current LLMs are not yet capable of providing this level of personalized tutoring.
• Building "Ramps to Knowledge":
  • Education is a technical problem of creating efficient "ramps to knowledge" – simplifying complex topics (e.g., Micrograd's 100 lines of code for backpropagation).
  • Nanochat, which you briefly mentioned, is a capstone project of LLM101N, which is a class that I'm building. [02:02:56]

    

  • The goal is to make learning trivial and desirable, so people learn for fun, like going to the gym.
• AI's Role in Education:
  • AI will assist in creating course materials, automating tedious tasks, and potentially serving as AI TAs for basic questions.
  • Human faculty will still be needed for overall course architecture and advanced concepts.
  • I have a whole rant on how everyone should learn physics in early school education because early school education is not about accumulating knowledge or memory for tasks later in the industry. [02:16:06]

    

• Overcoming the "Curse of Knowledge": [02:11:10]
  • Experts often struggle to explain concepts simply due to the "curse of knowledge."
  • Tools like ChatGPT can help experts identify "dumb questions" and improve their explanations by putting them in the shoes of a beginner.
  • Conversational explanations are often more understandable and accurate than formal writings.
• Motivation in Learning:
  • Online courses often fail due to motivation issues, as learners get stuck or bored.
  • An AI tutor could provide perfectly calibrated challenges, making learning rewarding and overcoming motivational barriers.
  • Post-AGI, education might become a form of "cognitive powerlifting," where people pursue knowledge for personal growth and enjoyment, similar to physical fitness.