The Simulation Doesn't Get Wet

On an episode of Creative Intelligence, the Wharton professor Kartik Hosanagar asked the neuroscientist Anil Seth the question this site keeps coming back to: can we build conscious AI? Seth’s answer ran two problems together that are easy to treat as one. The first is physical: what produces conscious experience in the first place. The second is psychological: what leads people to believe something has it.

On the physical question, Seth went after an assumption he thinks most of his field makes without noticing. The assumption is that the brain is “a kind of meat-based computer,” and that what matters for consciousness is the algorithm it runs, not the neurons it runs on. If that’s right, then finding the right computation and moving it onto a rack of GPUs should be enough. Seth doesn’t think it’s right, and he put the reason in one image: “A simulation of a weather system doesn’t get wet. A simulation of a black hole doesn’t destroy the universe.”

Behind the slogan is a point about how computers relate to what they run. A laptop separates hardware from software cleanly, so the same program runs the same way on different machines. Brains don’t separate that way. There is no clean line between what Seth calls “mindware” and “wetware,” no layer you could lift out and run somewhere else. If he’s right, the central move of computationalism, discarding the biology and keeping “the algorithm,” isn’t a simplification. It’s an unproven claim that looks like one.

Nelly’s Eyes made a related case: that shared biology, the same brain structures, the same evolutionary history, the same metabolism, is better ground for crediting another creature with an inner life. That essay also admitted its weak spot. Its argument, it said, “leans on an assumption a committed functionalist would reject: that the hard problem of consciousness has something to do with what a system is made of and not only what it does.” Seth’s biological naturalism offers a materialist rival to functionalism about experience, one grounded in how living systems are physically put together rather than in a feeling of kinship.

He doesn’t oversell it. His view is a live hypothesis, not a result. “It’s still certainly not demonstrated,” he said.

His second thread is where things get more solid. Why do we ask the consciousness question about some systems and not others? Nobody spends much time wondering whether AlphaFold, DeepMind’s protein-folding model, is conscious, and Seth pointed out how little that has to do with the technology: “they’re both computer algorithms, they’re neural network-based… They even have Transformer architectures. They’re really not that different.” The difference is that language models talk to us.

Because people have always tied language to minds, we reach for the Turing-style test: if it talks like a mind, treat it as one. Seth’s view is that a behavioral test is fine for intelligence, which is about doing, taking inputs and producing outputs, and wrong for consciousness, which is about being. A system can pass every test of the first and still have no claim on the second.

Pulled apart, the interview contains two different kinds of claim. One is about substrate: whatever makes consciousness, life, metabolism, some physical property of living systems, can’t be simulated into existence. That is a scientific bet, and nobody will know whether it pays off until the underlying questions are settled. The other is about attribution: the evidence currently persuading people that machines are conscious, fluent conversation, says more about human psychology than about any machine’s inner life.

The first claim is open. The second can be checked now. Nobody has to decide whether consciousness needs wetware to see that people are easily moved by fluent language.

So the argument over what produces experience stays where it was, stuck between computational and biological accounts of what consciousness requires, while the question of what convinces us we’ve found it runs on different rules entirely, and right now that conviction, built mostly from the feeling of being answered in good sentences, is setting the terms of the public conversation long before the science has anything to say about whether it is right.

Editor’s note: a related dynamic can be observed firsthand. Human or AI? tests fluency-driven judgments about authorship rather than consciousness directly, but it shows how readily people project an intentional mind behind compelling text.