Does the Machine Understand?
Large language models write essays, debug code, and pass exams that stump humans. So the argument rages: do they understand anything, or are they "stochastic parrots" doing very fancy autocomplete?
My take: both camps are more confident than the evidence allows, and the question itself is badly posed.
The Chinese Room, Revisited
John Searle's 1980 thought experiment: a man in a room follows rules to manipulate Chinese symbols he doesn't comprehend, producing fluent replies. The room passes the test; the man understands nothing. Therefore, Searle argued, symbol manipulation can't be understanding.
Forty-five years later we've built the room, and it's weirder than Searle imagined. Modern models don't follow hand-written rules; they learn compressed representations from data, and probing studies suggest those representations encode structure—syntax trees, world states, even rough spatial maps. Is a learned internal model of the world "understanding"? Searle's intuition pump doesn't obviously cover that case.
The Goalpost Problem
There's a pattern worth noticing: chess was going to require understanding, until Deep Blue. Then Go. Then natural conversation. Then coding. Each time the machine succeeds, the achievement gets reclassified as "just computation," and understanding retreats to whatever machines can't do yet. A definition that keeps updating to exclude every counterexample has stopped being a definition.
But the parrot camp's critics have their own problem: fluency is misleading. These systems confabulate with total confidence, fail at tasks a child handles, and their competence is jagged in ways you wouldn't expect from anything that understands. Passing our tests may say more about our tests than about minds.
Behavior vs. Mechanism
The disagreement is really about what "understanding" refers to. If it's a behavioral standard—reliably using concepts, generalizing, answering counterfactuals—then models partially meet it, measurably, and we can argue about the percentage. If it's a mechanistic or experiential standard—there's something it is like to grasp a meaning—then we have no test for it. Not for machines, and, awkwardly, not for each other either. You infer that I understand things from my behavior, and that's the only evidence anyone has ever had about another mind.
So when someone declares LLMs obviously do or obviously don't understand, ask which standard they're using. Usually they're using the behavioral evidence when it supports their side and the mechanistic standard when it doesn't.
The Question Behind the Question
I suspect "does it understand?" is a proxy for questions we actually care about: Can I trust its outputs? Does it have interests that matter morally? Will it generalize safely outside its training distribution? Those are real questions, and they can be answered separately. A system could be reliable without being conscious, or conscious without being reliable. Bundling everything into one word makes all of them harder to answer.
Where I Land
Provisionally: current models exhibit real, measurable, partial competence with concepts—something more than parroting and less than whatever humans do, sitting in a region our vocabulary wasn't built for. I think that calls for a new category rather than a forced verdict in an old debate.
We built machines that talk before we agreed on what talking means. The confusion is mostly ours.