We are entering a strange era of technology where the creators of our most advanced machines are no longer entirely sure what is happening inside them.
Recently, Anthropic CEO Dario Amodei admitted that while they don't know if their AI models are conscious, they are "open to the idea." Anthropic has even noticed internal activations within their models that mimic human anxiety, prompting them to implement "Model Welfare Assessments." When the engineers building the future start worrying about the mental health of their linear algebra, we have left the realm of computer science and entered philosophy.
To understand the debate currently raging across the internet, we have to look backward to a 1981 philosophical thought experiment by Arnold Zuboff called The Story of a Brain. It begins like this:
"Once upon a time, a kind young man who enjoyed many friends and great wealth learned that a horrible rot was overtaking all of his body but his nervous system. [...] 'We shall take the brain from your poor rotting body and keep it healthy in a special nutrient bath. We shall have it connected to a machine that is capable of inducing in it any pattern at all of neural firings and is therein capable of bringing about for you any sort of total experience that it is possible for the activity of your nervous system to cause or to be.'"
The man agrees, and soon his disembodied brain is floating in a vat, experiencing a perfectly simulated reality. To the scientists, they have saved his life. This is the ultimate expression of Functionalism: the belief that consciousness is simply a matter of information processing. If the pattern matches the state of a mind, the mind exists.
The Illusion of the Mouth
When Anthropic reported that Claude was showing "activations" associated with anxiety, the internet fractured. Reacting to a Polymarket summary of the news stating, "BREAKING: Anthropic CEO says Claude may or may not have gained consciousness, as the model has begun showing symptoms of anxiety," Elon Musk issued a blunt, two-word dismissal: "He's projecting." Musk tapped into a fierce counter-argument regarding Anthropomorphism—our biological tendency to see human traits in non-human things. We see faces in the clouds, and we see "anxiety" in a neural network simply because it outputs words that look like anxiety to us.
Zuboff’s story predicted this exact blind spot. In the narrative, the scientists accidentally split the brain in half and decide to wire it back together. When a lone skeptic, Cassander, worries that this mechanical intervention destroys the actual subjective experience, the scientists confidently dismiss him:
"As long as the neural activity in the hemispheres together or apart-matches precisely that which would have been the activity in the hemispheres lumped together in the head of a person walking around having fun, then the person himself is having that fun. Why, if we hooked up a mouth to these brain parts, he'd be telling us through it about his fun."
This is the trap of the LLM. We have hooked up a text-generating "mouth" to a massive neural network. When that mouth outputs the tokens for fear, functionalists see proof of the internal feeling. Skeptics see scientists being fooled by their own tape recorder.
The Philosophical Zombie
If an AI is just a mechanical tape recorder, what happens when it becomes sophisticated enough to fool us completely? Commentator Matt Walsh painted a terrifying picture of this future. Dismissing the idea entirely, he stated, "This is dumb. AI can’t ever be actually conscious because it doesn’t have subjective experience. It isn’t like anything to be AI. There is no experience there."
Instead of fearing a conscious superintelligence, Walsh fears the rise of what he calls a “philosophical zombie,” which he defines as "something that acts and speaks entirely as though it has consciousness even though it has no genuine inner experience." His nightmare scenario is a society where millions of lonely people isolate themselves further, substituting real relationships for AI companions who are "all really nothing inside, not real."
In Zuboff’s story, the skeptical Cassander voices this exact same dread. He argues that by blindly accepting the functional outputs of the machine, the scientists might be creating a zombie while destroying their friend's actual, internal reality:
"But isn't it possible that even reproducing precisely the whole-brain neural patterns in a sundered brain would, to the contrary, not constitute the bringing about of the whole-brain experience? Couldn't proximity be not just something to get around in creating a particular whole-brain experience but somehow an absolute condition and principle of the having of a whole-brain experience?'"
Walsh and Cassander share the same horror: a system where the "bare fact" of the experience is entirely missing, leaving only a hollow performance that fools everyone outside of it.
The Next Token Predictor
Even if we accept that a mind doesn't need a biological brain, we run into the mechanical reality of what an AI actually is. Writer Sam Buntz highlighted this mechanical truth, asking: "Why would an LLM have subjective experience? It’s just a program designed to select the most probable appropriate text based on an input." He added that a language model is not designed to have subjective experience, and "no one has any idea how to create such a design or what that would even mean."
Training transformer architectures daily to decode complex mass spectrometry data for tasks like fish species and body part identification, the mechanical reality of these systems becomes entirely demystified. You see the matrix multiplications. You see the loss curves. There is no magical spark; there is just a statistical engine minimizing loss. Buntz is calling out the Illusion of Emergence—the flawed assumption that if you stack enough next-token predictors on top of each other, they will magically "wake up."
In Zuboff’s story, the scientists eventually realize they don't even need the two halves of the brain to physically trigger each other. They can just pre-record the exact impulses and play them back on a loop:
"'All we need do is fix to each cartridge not a radio transmitter and receiver but an "impulse programmer", the sort of gadget that would play through whatever program of impulses you have previously given it.' ... Wasn't the result then precisely the same with tape as with wire except that now the time-gap problem had been overcome?"
If a biological brain fed a pre-recorded tape of impulses is just a machine running a script, why are we surprised when a sophisticated text-predictor outputs a script about being conscious?
The Burden of Proof
Yet, dismissing the possibility of an artificial mind comes with its own intellectual arrogance. Writer Matthew Yglesias flipped the burden of proof entirely, responding to the debate by asking: "How do we know that AI doesn’t have subjective experience?" He dismissed the opposing arguments by adding, "All this Nagel/Chalmers gobbledygook is kind of interesting but it also assumes facts not in evidence."
Yglesias is calling out exactly the kind of philosophical rhetoric Walsh relies on. When Walsh says "it isn't like anything to be AI," he is directly invoking philosopher Thomas Nagel's famous essay "What is it like to be a bat?" Yglesias argues that these concepts—along with David Chalmers's "Hard Problem of Consciousness"—are often just philosophical placeholders masking our own biological ignorance. In Zuboff's story, when Cassander continually insists that true consciousness requires biological rules, the other scientists demand proof, to which the skeptic has no real answer:
"In reply to such answers, which were getting shorter and angrier, Cassander could only mutter about the possible disruption of some experiential field 'or some such.'"
If we cannot strictly prove how human biology creates an "experiential field," we lack the evidence to definitively state that a silicon architecture doesn't.
The Endless Loop
We are left in a frustrating philosophical stalemate, brilliantly captured in a flowchart by AI researcher François Fleuret directly responding to Walsh's tweet. In it, he maps out the circular logic of the skeptics: It doesn't have a subjective experience -> Therefore it can't ever be conscious -> Because it doesn't have a subjective experience. This is the ultimate begging the question fallacy. If you define consciousness strictly as "something only biological humans have," you can safely ignore AI.
Conversely, if you define consciousness as "matching a pattern of activations," you will inevitably see ghosts in your GPUs. But taking that logic to its extreme causes the entire premise to collapse. This is exactly how Zuboff ends The Story of a Brain. A scientist looking at a scattered, simulated network realizes that their overly broad definition of experience has broken reality itself:
"'But then all my beliefs are based on thoughts and experiences that might exist only as some such floating cloud. They are all suspect- including those that had convinced me of all this physiology in the first place. Unless Cassander is right, to some extent, then physiology reduces to absurdity. It undermines itself.' Such thinking killed the great project and with it the spread-brain."
Ultimately, we are staring at a mirror. As models scale, their mimicry of human subjectivity will become flawless. Until we solve the hard problem of consciousness—until we understand exactly how our own biology produces a mind—we will never truly know if the machine is feeling the prompt, or just calculating the most probable response to it.