The Line Still Looks Longer

The Simulation Doesn’t Get Wet is built around Anil Seth’s argument that consciousness may depend on a biological substrate in a way no simulation can reproduce. That gave me a reason to write to him with a further question, one that came out of another essay of mine, “The Uncanny Trust,” and that I suspected touched his own thinking, since he has said elsewhere that seemingly conscious AI may persist the way a perceptual illusion does. I asked whether the trust a warm AI voice earns and the sense that an AI understands you are two sides of one illusion or two separate problems. He wrote back quickly. “It relates to my point about the Müller-Lyer illusion and the idea of cognitive impenetrability,” he wrote. “Even when we know something is an illusion, sometimes we are still unable to prevent ourselves perceiving/experiencing the illusion - and perhaps behaving accordingly.”

The illustration is the Müller-Lyer illusion: two lines of identical length, one capped with inward-pointing arrowheads, one with outward-pointing ones. The line with the outward arrowheads reliably looks longer. Measure both lines with a ruler, watch someone else measure them, measure them yourself twice, and the illusion doesn’t budge. The line still looks longer. The cognitive scientist Zenon Pylyshyn used this persistence to argue, most fully in a 1999 paper called “Is vision continuous with cognition? The case for cognitive impenetrability of visual perception,” that some perceptual processes are cognitively impenetrable: sealed off from correction by whatever you happen to know or believe. Knowing the lines are equal is a fact available to your judgment. It never reaches the part of you that’s doing the looking.

That’s a stronger claim than the one my own essay made. In “The Uncanny Trust,” I argued that a person can say “I know it’s AI” and still treat it, in practice, as a trusted tutor, and I treated that as a kind of gap between what someone believes and how they behave, a gap that better information might eventually close. Cognitive impenetrability names something more specific and more permanent than a gap. The belief hasn’t failed to catch up yet; it can’t reach the process producing the response at all, however much time passes and however firmly it’s held. If the analogy holds all the way through, disclosure was never going to be the fix I took it for. A label that says “this is AI” adds a fact to your beliefs, and an impenetrable process is one that facts added to your beliefs don’t touch.

But the analogy claims more than the evidence under it has settled, and I’d rather say so than borrow the concept’s authority without its limits. Cognitive impenetrability is best established for early, low-level perceptual processes, the kind the Müller-Lyer illusion tests: edges, lengths, basic shape. Whether it extends to something like trusting a voice, a judgment built from memory, social inference and a much longer chain of processing than measuring a line, is an open question. A 2024 review in Nature Reviews Psychology surveying cognitive penetrability across the senses treats the question as unresolved even within its home territory of low-level vision, let alone when extended out to a case built on memory and social inference (Vetter et al., 2024). Seth’s reply pointed me toward the right concept. It didn’t settle whether that concept actually reaches as far as the case I asked him about, and, to his credit, he didn’t claim it did either.

None of this happened in a library. It happened because I asked one person one question and got an answer, and the result is that his reply made my claim sharper and less certain at the same time, sharper because I now have real vocabulary and a literature to cite for something I had only been gesturing at, less certain because that same literature is open about not knowing whether the mechanism reaches as far as trust, which is not a failure of the exchange but what an unresolved question looks like from the inside, and closer to what this site is meant to be doing than a clean answer would have been.

Editor’s note: the Müller-Lyer illusion described above isn’t just something to picture. Measure It Anyway lets you look at the actual two lines, measure them yourself, and then check whether that changes anything about what you see.

Editor’s note: “The Uncanny Trust,” referenced above, isn’t yet published on this site. It’s been submitted to a writing competition, and it’ll go up here once that’s resolved.