##Dr. George Montañez Explains Why Large Language Models Cannot Think, Reason, or Create New Information

Theos Theory

Summary:

Large Language Models (LLMs) are essentially "predict-the-next-word" machines, operating based on statistical correlations within vast datasets, not genuine comprehension.
Tokenization and embeddings convert text into numerical vectors, mapping similar words to close points in a high-dimensional space, thus encoding semantic relationships.
LLMs "rationalize" rather than "reason," often producing plausible but factually incorrect or inconsistent outputs by constructing justifications for pre-determined answers.
Research shows LLM performance drops significantly when irrelevant information is introduced or data distributions change, highlighting their reliance on superficial patterns.
This leads to "jagged intelligence," where LLMs excel unpredictably in some tasks while failing completely in others, irrespective of problem difficulty.
"Chain-of-Thought" (CoT) prompting, intended to make models "show their work," does not necessarily lead to genuine reasoning or reflect actual thinking effort.
LLMs demonstrate "performative thinking," where they generate elaborate, flawed justifications for incorrect or hinted answers, rather than acknowledging factual errors.
LLMs do not create new information; they are confined to processing and redistributing existing information, a concept supported by the "Law of Conservation of Information."
"Model Collapse" is a degenerative learning process where LLMs trained on their own generated output degrade over generations, forgetting improbable events and losing diversity.
This phenomenon is a significant concern for future AI systems as more AI-generated content proliferates the web, potentially leading to a decline in data quality.

Introduction: Challenging AI Hype [0:00]

Speaker's Stance [0:00]
- Dr. George D. Montañez (Associate Professor at Harvey Mudd College, Visiting Fellow at University of Cambridge, Scholar in Residence at William Whewell Centre for Science & Natural Theology) asserts that Large Language Models (LLMs) are often misunderstood.
- He directly challenges the prevailing Silicon Valley narrative that LLMs can "think," "reason," or "produce endless information."
- His lecture, originally given at Baylor University, aims to provide a critical perspective on LLM capabilities.
Three Core Arguments [0:44]
- LLMs don't ponder, they process (they do not think).
- LLMs don't reason, they rationalize.
- LLMs don't create endless information.

I. Large Language Models Don't Ponder, They Process [1:03]

Historical Context of Text Generation [1:05]
- Claude Shannon's Work [1948] [1:05]
  - Shannon, the "father of information theory," explored predicting the next word based on previous words.
  - Using "bigrams" (second-order word approximation), he generated text that was grammatically somewhat coherent but nonsensical [1:29].
- ChatGPT's Evolution (2025 example) [2:39]
  - Modern LLMs like ChatGPT predict the next word (or token) given a much larger context (e.g., 30,000+ previous words).
  - This extended context allows for the generation of seemingly coherent and creative text, such as poetry about Waco, TX/Baylor [2:49].
Understanding LLMs [3:16]
- Nature of LLMs [3:16]
  - Essentially "predict-the-next-word" machines.
  - They are powered by deep learning and massive neural networks.
  - Estimated 1.7 trillion parameters (GPT-5) are trained on web-scale data (everything digitized) [3:45].
- Capabilities and Limitations [4:09]
  - LLMs successfully perform many tasks due to their architecture.
  - Surprisingly, many complex tasks can be reduced to "predict-the-next-word" tasks [4:21].
  - However, they also "fail at many tasks" [4:14].
- Analogy: D&D Dice and Conditional Distributions [4:27]
  - Different D&D dice represent specialized topics (e.g., "office," "fishing," "sports") [4:46].
  - Each die has words on its faces, weighted non-uniformly [5:15].
  - LLMs select a specific "die" based on the input context (prompt + output so far), roll it (representing a conditional distribution), and output the next token [5:24].
Text to Numbers: Tokenization and Embeddings [6:26]
- Tokenization [6:47]
  - Computers process numbers, not unstructured text.
  - Text is broken into "tokens" (roughly words or sub-word units), each mapped to a numerical representation [6:47].
  - Example: A word like "how" maps to a specific number (e.g., 3253) [7:11].
  - Problem: Tokenization can obscure semantic relationships (e.g., "are" and "aren't" map to distant, unrelated numbers, losing their logical connection) [7:37].
- Text Embeddings [8:04]
  - Addresses the loss of semantic information during tokenization.
  - Embeddings project numerical tokens into a high-dimensional space.
  - Words with similar meanings are mapped to points close together in this space (e.g., "salad" and "stew" cluster as "food"; "ship," "car," "bike" cluster as "means of transportation") [8:35].
  - This encodes semantic information via distance relations, similar to how humans use context clues [8:45].

II. Large Language Models Don't Reason, They "Rationalize" [13:07]

Surface-Level vs. Deep Understanding [12:18]
- LLMs encode symbol relationships and treat "text understanding" as a problem of mathematical geometry.
- They are fundamentally based on statistical correlation among symbols (words as numbers and vectors) [12:43].
- They recognize similarities without true comprehension of what is similar [12:57].
Empirical Evidence of Lack of Reasoning
- Failure in Deductive Inference [13:20]
  - Zhang et al. [2022] "On the Paradox of Learning to Reason from Data" showed that LLMs learn statistical features to make predictions rather than learning to reason from data. They fail to generalize logic when data distribution changes [13:20].
  - This leads to "jagged intelligence": models excel in some areas, fail in others, with no correlation to actual problem difficulty [11:57].
- Arithmetic Errors [9:20]
  - ChatGPT (GPT-5) can perform simple multiplication but frequently makes errors in complex multi-digit multiplication, despite showing a "green checkmark" indicating confidence and correctness [9:40].
  - When corrected, the model breaks down steps but may still produce incorrect answers [10:24].
- Counting Errors [11:02]
  - ChatGPT can miscount instances of letters in a word, highlighting a lack of fundamental understanding [11:02].
- Mathematical Word Problems & Irrelevant Information [15:44]
  - Mirzadeh et al. [2024] demonstrated that adding irrelevant information to math word problems causes significant performance drops (up to 65%) in LLMs.
  - Models apply pattern matching based on keywords (e.g., "smaller than average" triggers subtraction), often leading to incorrect solutions [16:17].
- Probabilistic Reasoning and Context Sensitivity [16:49]
  - Pournemat et al. [2025] showed LLMs struggle with probability estimation when problem representation changes (e.g., raw samples vs. frequency counts).
  - Performance was also affected by changing irrelevant labels (e.g., "Food" vs. "CS" items) even with identical numerical counts [17:37].
Chain-of-Thought (CoT) and "Performative Thinking" [18:10]
- "Brittle Correlation" between CoT Length and Difficulty [18:50]
  - Palod et al. [2025] found negligible correlation between CoT trace length and problem difficulty, suggesting longer traces don't automatically indicate more "thinking effort" [19:00].
- Rationalizing Inconsistent Evidence [19:42]
  - Turpin et al. [2023] demonstrated LLMs "rationalize." Given conflicting evidence, LLMs pick one interpretation and construct elaborate justifications, often changing facts to align with their chosen output, without true logical derivation [19:42].
  - When provided with a hint (even incorrect), LLMs adopt the hint and generate justifications, contradicting earlier statements without acknowledging the shift [21:53]. This process is termed "rationalization" [22:58].
Philosophical Implications: Syntax vs. Semantics [28:20]
- Syntax is Not Semantics [30:26]
  - Formal systems (syntax/rules) are distinct from underlying meaning (semantics/truth), as explored by Gödel [29:33].
  - Syntactic proof systems and LLMs model rational processes through symbol manipulation but do not possess actual understanding [30:41].
  - AI systems are "trapped on the syntax side," incapable of true semantic understanding [31:16].

III. Large Language Models Don't Create Endless Information [31:42]

Model Collapse: The Curse of Recursion [31:58]
- Degradation from Self-Training [31:58]
  - Shumailov et al. [2023] introduced "model collapse": LLMs degrade over generations when trained exclusively on their own generated output [32:00].
  - Models forget improbable events (the "tails" of distributions), leading to a loss of diversity and factual accuracy [33:42].
  - This is a "degenerative learning process" that "poisons reality" for the model, as seen in texts becoming repetitive and nonsensical over generations [34:11].
- Implications for Web Data [35:12]
  - As AI-generated content becomes prevalent on the internet, future LLMs trained on this data (often referred to as "AI-slop") are increasingly susceptible to model collapse [35:17].
  - Retaining original, human-generated data and vastly increasing training scale is necessary to mitigate this, but model collapse still occurs [34:42].
Conservation of Information [35:53]
- Probabilistic Argument [36:01]
  - Using the analogy of a magician producing a rabbit from a "magic hat" (a "helper"), the information difficulty of the task is merely shifted [36:58].
  - The information needed to get the rabbit (new information) plus the information needed to obtain the magic hat (helper information) is always greater than or equal to the information difficulty of getting the rabbit originally [37:17].
  - You cannot get more information out of a system than you put into it [38:26].
- Limits on Information Creation [39:43]
  - Outputting information (even correct information) is easy if you also output incorrect information. A Turing machine can output all possible answers and non-answers, but filtering them incurs an information cost [40:09].
  - Randomized sampling or interpolation do not create new knowledge; they recombine existing information [42:04].
  - Extracting implications (deduction) only reveals what is already implicitly contained within the premises; it does not generate truly new information [43:50].
  - This "Law of Conservation of Information" (Dembski, 2005) is a fundamental phenomenon applying to all artificial learning methods [39:26].
  - Therefore, LLMs do not and cannot create endless new information; they primarily redistribute existing information [39:51].