The Black Box Inside the Black Box: What Yan-Xiang-Yi Can (and Can't) Explain About AI

In the third century, the philosopher Wang Bi (王弼), writing on the Book of Changes, set out how meaning passes from mind to word. In the chapter of his Zhouyi Lüeli (周易略例) on images, he wrote: 言者所以明象，得象而忘言；象者所以存意，得意而忘象. Words exist to make the image clear, and once the image is grasped the words can be forgotten; images exist to hold the meaning, and once the meaning is grasped the image can be forgotten.

The three terms in that passage, yan (言), xiang (象) and yi (意), make a ladder. Yi is the meaning or intent itself: continuous, elusive, never fully captured. Xiang is the image, the intermediate figure, in the Changes the trigrams and hexagrams, which gives yi a shape without forcing it into the straight line of a sentence. Yan is language, the final, linear, spoken or written output. Each rung is a vehicle for the one above it, and each is meant to be left behind once it has done its work.

It is easy to see why this ladder appeals to researchers working on explainable AI. A neural network’s internal activations, millions of continuous numbers, resemble yi: they carry whatever the model “means,” and no person can read them directly. Interpretability methods try to insert something like xiang between those activations and the model’s output, a discrete, inspectable layer, such as a set of concept scores, an attention map, or a symbolic graph, through which the computation can be traced before it is rendered as an explanation in words, yan. The hope is that forcing the reasoning through a layer people can read will open the black box.

The assumption behind that hope is that the black box is a translation problem. The model knows why it decided as it did; it simply knows it in a language of vectors. Add a structured symbolic layer that shows which concepts were active and which rules fired, and the knowledge has been translated. The story is tidy. It does not hold.

The first difficulty is that adding a xiang layer may only move the black box. Popular after-the-fact techniques, such as Grad-CAM heat maps that highlight the pixels an image classifier relied on, or TCAV, which tests how strongly a learned concept direction influences a prediction, build the intermediate layer after the model has been trained. But the mapping from raw activations to that layer, whether it comes from a probe classifier, an attention head or a traced circuit, is itself a learned, high-dimensional function. The step from yi to xiang is as opaque as the computation it claims to translate.

When a heat map shows why a model labeled a bird a kingfisher, or a probe reports that a concept like “fraud” was active when a language model flagged a transaction, the question is who checked that the image matches what caused the decision. Many after-the-fact explanations are tuned to look plausible to people, not to track the causes. They resemble human reasoning, while the transformation that produced them remains inside the matrix.

It helps here to separate two senses of “explain.” The first is causal fidelity, an audit trail: does the intermediate layer reflect what actually drove the output? The second is understanding: does the layer show that the system grasps the concepts it manipulates? After-the-fact explanations feel like tricks because they fail the first test. They offer no assurance of fidelity at all.

Machine learning researchers recognized this, and one response was the concept bottleneck model (Koh et al., 2020). In such a model the final prediction is computed only from a layer of human-named concepts, so the concepts are not a story added afterward but the channel the decision must pass through. That goes a long way toward the first sense of explanation, though not all the way. Later work found that when concept scores are soft numbers rather than plain yes-or-no values, they can carry information beyond the concepts they are named for, so the bottleneck leaks (Mahinpei et al., 2021). Fidelity can be improved by design, but it has to be tested, not assumed.

Suppose, though, that it were achieved completely. The second question would remain: does a perfectly faithful audit trail mean the system understands what it is doing?

John Searle’s Chinese Room argument answers no. A person in a locked room follows a rulebook for manipulating Chinese characters, and to an observer outside the room seems fluent in Chinese. Grant the observer the rulebook as well, every table and every substitution open to inspection. The system is now fully transparent at the level of its rules. A concept bottleneck model with no leakage would be the best possible version of that rulebook. Yet, as Searle argued, transparency at the level of rules adds no understanding. The person in the room knows no Chinese, and the rulebook knows none either; it only maps shapes to shapes.

The two senses of explanation, then, lie on different axes, not at two stages of one road. Interpretability in the first sense is a property of the audit: whether an observer can trace and predict the system’s transformations. Understanding in the second sense is a property of the system: whether its symbols are grounded in any grasp of the world. A causally faithful xiang layer settles the first and says nothing about the second. A legible bottleneck buys an audit trail. It does not buy a mind.

What does this leave of yan-xiang-yi in the age of AI? It weakens the claims of research that invokes symbolic bridging as though it settled interpretability. Wang Bi was describing the limits of representation, how every image falls short of the meaning it carries and is meant to be discarded once that meaning is reached. He was not offering a method for opening black boxes.

As a design principle for auditability, the ladder is useful. Built into architectures such as concept bottleneck models, and tested for leakage, a discrete xiang layer between activations and language gives engineers a better instrument for filtering unsafe behavior, tracing errors and checking alignment. As a route to a model’s understanding, it offers nothing. An intermediate symbolic layer neither reveals a model’s intent nor shows that the model understands the world it models. Wang Bi treated the image as scaffolding, to be forgotten once the meaning was grasped. An interpretability researcher might keep the same modesty and treat these layers as scaffolding built for human readers, a way of making the mathematics look familiar.

A fully transparent Chinese Room is still a room full of paper. A well-structured, faithful xiang layer is still a map, drawn for human eyes, of a landscape the machine does not see. Systems built around such layers deserve to be valued for what they provide, a clearer and more checkable view of behavior, so long as no one mistakes a clearer window for a mind behind the glass.

Editor’s note: an open-rulebook chatbot can be tried directly. Talk to the Room lets you chat with one, then read the exact rules behind every reply it gave.