Dave's reading notes

llm-and-mind

Properties1
tagscognition, concept, llm, philosophy-of-mind

LLMs and Mind

Summary: Cross-source concept page synthesizing philosophical, biological, and empirical evidence on what LLMs actually are — semantically, cognitively, and in terms of reasoning ability. Three sources from 2024–25 converge on a gap between LLM capability and genuine cognition that is fundamental, not merely contingent on scale.

Sources: Academia/LLMs Turing tests and Chinese rooms the prospects for meaning in large language models.pdf, Academia/the-illusion-of-thinking.pdf, Academia/Agency and cognition.pdf

Last updated: 2026-05-07

The question

Large language models produce outputs that are compelling, contextually appropriate, and difficult to distinguish from human writing. This creates pressure to attribute to them rich cognitive properties: understanding, reasoning, thinking. Three recent papers resist this attribution from different angles.

What Borg establishes philosophically

borg-llm-meaning (Borg 2025) distinguishes three levels:

  1. LLM outputs are meaningful — via derived intentionality and semantic deference. Words mean what they do because human speakers made them mean things; LLM outputs inherit this meaning as tokens of those types.
  2. LLMs do not assert — the norm governing output is word-occurrence probability, not truth. LLMs cannot distinguish their hallucinations from their true statements. They produce meaningful sentences but do not commit to them.
  3. LLMs lack original intentionality — they are not agents. No intrinsic goals, no stable point of view, no motivated relationship with the world. Borg’s key move: be glad of this, because granting LLMs original intentionality would make them candidates for moral consideration.

What Jaeger et al. establish biologically

jaeger-relevance-realization (Jaeger et al. 2024) argues that the gap is not a current limitation but a categorical one. Living systems possess relevance realization — the ability to convert ill-defined, large-world problems into tractable ones. This ability:

  • Arises from intrinsic goals rooted in precariousness and mortality
  • Is transjective — neither subjective nor objective, arising from organism-world interaction
  • Operates via opponent processing — a co-constructive, impredicative Darwinian dialectic that continuously re-evaluates the problem space
  • Cannot be completely formalized by any algorithm

Calling the brain a computer is the equivalence fallacy — confusing the model with the thing being modeled. Algorithms live in pre-defined small worlds; organisms live in genuinely open large worlds. LLMs are sophisticated algorithms and therefore cannot do what they would need to do to count as cognitive systems in the full sense.

What Apple’s researchers establish empirically

illusion-of-thinking (Shojaee et al. 2025, Apple) tests frontier reasoning models on controllable puzzles and finds three regimes:

  • At low complexity, standard LLMs outperform reasoning models
  • At medium complexity, extended thinking helps
  • At high complexity, both collapse to zero accuracy — and the reasoning models reduce thinking effort as problems approach the collapse threshold, despite having ample token budget

More strikingly: providing the explicit algorithm does not help. Models cannot reliably execute prescribed logical steps across many sequential moves. Performance tracks training-data exposure rather than genuine planning ability.

Convergence

All three papers agree:

ClaimBorgJaegerApple
LLMs are not agents✓ (lack original intentionality)✓ (lack intrinsic goals)—
LLM “reasoning” is not generalizable—✓ (relevance realization is non-algorithmic)✓ (collapse at complexity)
The gap is fundamental, not contingent✓ (creating agents risks moral concern)✓ (categorical biological difference)✓ (fundamental scaling limit)

Connections to other pages

dual-process-cognition — the fast/slow dichotomy (Kahneman/Sapolsky) is a property of biological brains with amygdala and frontal cortex. LLMs have no such architecture; their “reasoning” traces do not correspond to a System 2 process but to a learned simulation of one. The simulation can pass many tests that the real thing passes.

hawkins-a-thousand-brains — Hawkins’s Thousand Brains theory posits that the neocortex builds reference frames anchored to physical sensorimotor grids. LLMs have no such grounding. They may accurately describe world models without themselves being world models in Hawkins’s sense.

free-will-and-moral-responsibility — if genuine agency requires the kind of biological precariousness Jaeger describes, then LLMs have no agency and raise no questions of responsibility. They are tools, not actors. (Borg adds: we should keep it that way.)

Agentic frameworks and the tool-calling extension

OctoTools (Lu et al., 2025) represents one engineering response to the limitations documented above: rather than extending LLM reasoning depth, it wraps LLMs in structured tool-calling architectures (tool cards, planner, executor) that offload reasoning steps to verifiable external tools. The 9.3% accuracy gain over GPT-4o on 16 benchmarks reflects not improved reasoning but improved task decomposition and tool selection. This is consistent with the Apple findings: the model’s reasoning collapses at complexity, but routing sub-problems to specialized tools avoids requiring the model to reason through them directly. See octotools.

The NEC negotiation paper (nec-genai-negotiation) extends this logic to real-world B2B procurement: LLMs contribute natural-language interaction and contextual reasoning; specialized forecasting models handle the quantitative utility evaluation. The division of labor between LLM and tool is the key architectural move.

The predictive coding parallel — an empirical challenge

One prominent argument for taking LLM representations as genuine models of brain computation is the claim that both LLMs and brains use predictive coding: they encode upcoming words before they arrive. Goldstein et al. (Nature Neuroscience 2022) reported that word embeddings from language models predicted brain activity before participants heard the corresponding words — interpreted as evidence of shared computational strategy.

Schönmann, Szewczyk, de Lange & Heilbron (eLife 2026) reanalyzed the same datasets using systems that provably cannot encode future words (randomized embeddings; raw auditory representations). Both produced the same apparent “predictive” signature. Their explanation: language’s bidirectional statistical dependencies (collocations, idioms, syntactic cues) create the appearance of prediction in any system processing language, without requiring genuine predictive computation.

This is a methodological critique with broad implications: encoding models are correlational, not causative. They cannot distinguish “the brain predicts” from “language statistics make it look like the brain predicts.” The brain-LLM shared-computation argument loses its primary empirical support.

See schonmann-predictive-coding-brain and predictive-coding.

Open questions

  • If LLMs gain multimodal embodiment (the “Robot Reply”), does that push toward original intentionality?
  • Does the distinction between derived and original intentionality map onto legally or ethically meaningful categories?
  • The Apple paper shows collapse at a specific compositional depth. Is there a theoretical characterization of what kind of problems fall in each regime?

LessWrong perspectives

lesswrong-consciousness provides the rationalist community’s canonical treatment of phenomenal consciousness — the question that underlies Borg’s distinction between derived and original intentionality. The LW community’s strong physicalism and Yudkowsky’s p-zombie refutation are the backdrop for much AI consciousness discourse.

ai-alignment-failure-modes extends the question from “what are LLMs” to “what goes wrong when we deploy them at scale” — both the proxy-erosion failure and the interface-design failure are downstream of the gap between LLM capability and genuine cognition documented here.

magfrump-sculpted-interaction proposes that the interface layer is itself an alignment concern: good model properties fail if the chatbot format systematically undermines the human judgment needed to catch LLM errors.

taylor-ai-honesty-policy takes the question of AI moral status seriously in a novel direction: regardless of whether LLMs have experiences, the inability to make credible commitments forecloses positive-sum cooperation with advanced AI systems.

Graph View

Backlinks