Why the Valuable Capabilities of LLMs Are Precisely the Unexplainable Ones

Wh y the V aluable Capabilities of LLMs Are Precisely the Unexplainable Ones Quan Cheng T singh ua Universit y c hengq25@mails.tsinghua.edu.cn Abstract This pap er prop oses and argues for a coun terintuitiv e thesis: the truly v aluable capabilities of large language mo dels (LLMs) reside precisely in the part that can- not b e fully captured b y h uman-readable discrete rules. The core argument is a pro of b y contradiction via exp ert system equiv alence: if the full capabilities of an LLM could b e describ ed by a complete set of human-readable rules, then that rule set would b e functionally equiv alen t to an exp ert system; but exp ert systems hav e b een historically and empirically demonstrated to be strictly weak er than LLMs; therefore, a contradiction arises — the capabilities of LLMs that exceed those of exp ert systems are exactly the capabilities that cannot b e rule-enco ded. This the- sis is further supported b y the Chinese philosophical concept of W u ( 悟 , sudden insigh t through practice), the historical failure of exp ert systems, and a structural mismatc h b et ween h uman cognitiv e to ols and complex systems. The pap er discusses implications for interpretabilit y research, AI safet y , and scientific epistemology . 1 In tro duction An thropic’s in terpretabilit y researc h team once drew an analogy: “It’s as though w e understand aviation at the level of the W right brothers but hav e someho w already built and are routinely flying 747s” [1]. This analogy p oin ts to a fundamen tal puzzle: we understand every step of the mathe- matics b ehind the T ransformer architecture, attention mechanisms, and gradien t descent, y et we cannot explain ho w “understanding” emerges from matrix parameters. Ev ery com- p onen t is explainable; the assem bled b ehavior is not. This inexplicability is commonly attributed to emer genc e — the whole b eing greater than the sum of its parts. But “emergence” merely names the phenomenon without answ ering a sharp er question: what is the relationship b etw een the unexplainable part and the v aluable part? The central thesis of this pap er is: they are the same part. The truly v aluable capabilities of LLMs are precisely those that cannot b e fully captured by human-readable discrete rules. The capabilities that c an b e captured by suc h rules are what exp ert systems ac hieved 40 years ago — and those capabilities pro v ed insufficien t, which is precisely why w e need LLMs. 1 2 The Core Argumen t 2.1 Pro of b y Con tradiction 1. Assume : An LLM can b e fully explained. 2. “F ully explained” means there exists a set of human-readable rules that completely describ es the LLM’s b eha vior on all inputs. 3. A set of h uman-readable rules is functionally equiv alen t to an exp ert system. 4. But history and practice ha v e demonstrated that exp ert systems are strictly w eaker than LLMs [2]. 5. Therefore, such a rule set cannot co ver the full capabilities of the LLM. 6. Con tradiction. Conclusion: The truly v aluable capabilities of LLMs reside precisely in the part that cannot b e fully captured b y discrete rules. The explainable part is equiv alen t to what exp ert systems already ac hiev ed — and the insufficiency of that part is exactly why w e need LLMs. 2.2 Blo c king Three Alternativ e P aths If h umans cannot directly explain LLMs through rules, could LLMs explain themselves? All three alternative paths face fundamental obstacles: • LLMs explaining themselv es : Sub ject to the self-reference limitation. Gödel’s Incompleteness Theorem [3] establishes that a sufficiently complex formal system cannot fully describ e itself from within. Ma et al. [4] hav e formalized this limitation from the p ersp ective of P A C learning theory — a sufficien tly p ow erful machine learning algorithm is necessarily uninterpretable when facing certain ob jects. • A larger system explaining a smaller LLM : The problem is transferred, not solv ed — who explains the larger system? This leads to infinite regress. • LLM-h uman collab oration for explanation : The h uman brain is itself a con- tin uously coupled system that cannot fully explain its own workings. The com bi- nation of t wo systems that cannot fully explain themselves do es not ov ercome the self-reference limitation, b ecause this limitation is not a matter of computational p o wer but of logical structure. 2.3 Relation to Prior W ork This pap er’s argument is related to but distinct from sev eral existing lines of work. W olfram argues from computational irreducibility that constraining machine learning to b e understandable w ould preven t it from accessing the p ow er of computationally irre- ducible pro cesses [5]. This represents a similar intuition arriv ed at from pro cess theory , but with an en tirely different argumen t structure — W olfram do es not use exp ert system equiv alence as a pro of device. 2 arXiv:2504.20676 uses algorithmic information theory to pro ve a “Complexit y Gap Theorem”: an y explanation significan tly simpler than the mo del m ust fail on some inputs [6]. This establishes a tradeoff b etw een explainability and capabilit y , but the conclusion is about a tr ade off — this paper’s conclusion is stronger: the v aluable part is precisely identic al to the unexplainable part. It must b e emphasized that this paper’s argumen t does not deny the v alue of in terpretability researc h . The goal of interpretabilit y researc h should b e understo o d as narro wing the most dangerous blind sp ots and building lo cal causal understanding, rather than pursuing complete explanation of system behavior. As the work of Olah et al. [7] demonstrates, the identification of in ternal features and circuits in neural netw orks, ev en when lo cal and appro ximate, carries significant scientific and safety v alue. 3 Historical Evidence: The Death of Exp ert Systems Step 4 of the pro of — “exp ert systems are strictly weak er than LLMs” — is not merely a logical premise but a historical fact supp orted b y 40 y ears of empirical evidence. In the 1980s, exp ert systems were the dominan t paradigm in artificial in telligence. The approac h was straightforw ard: interview domain exp erts, enco de their kno wledge as explicit IF-THEN rule bases, and ha ve computers execute these rules [2]. The implicit assumption was that exp ert knowledge could b e fully discretized and made explicit. Exp ert systems achiev ed success in certain narrow domains but ultimately failed as a general AI paradigm. The reason is clear: the knowledge that truly makes an exp ert an exp ert — the con tinuous in tuitions across high-dimensional spaces, the complex relation- ships where v ariables are deeply en tangled with one another — is precisely the knowledge that cannot b e captured by IF-THEN rules. What exp ert systems could encode was only the shallow est, most regularized la yer of exp ert knowledge. Kam bhampati c haracterized this phenomenon in Communic ations of the A CM as “P olanyi’s Revenge” [8]: P olanyi prop osed in 1966 that “w e kno w more than w e can tell” [9], and the failure of exp ert systems was a large-scale engineering v alidation of this prop osition. The success of LLMs represents the p ositive realization of Polan yi’s thesis — rather than attempting to make kno wledge explicit, LLMs allo w kno wledge to b e stored implicitly in the form of contin uous parameters. LLMs to ok an entirely differen t path. Rather than pursuing the “correct path” at every step, they care only about the distance b etw een output and ob jective. The result is a set of extremely complex, m utually coupled parameters — unexplainable, incompressible, but remark ably effective. The paradigm shift from expert systems to LLMs is, at its core, an epistemological turn: from “attempting to eliminate inexplicability” to “accepting inexplicabilit y and w orking within its presence.” 4 W u ( 悟 悟 悟 ) — A Preceden t from Eastern Philosoph y The core argument of this pap er has a striking precedent in Chinese traditional philoso- ph y . W u ( 悟 , often translated as “enligh tenment” or “sudden insight”) is a profoundly im- p ortan t concept in Chinese tradition. Zen Buddhism teaches “ 不 立 文 字 ， 直 指 人 心 ” — “do not rely on w ords; p oint directly to the mind.” Martial arts teac hes “ 拳打 千 遍 ， 身 法 3 自 然 ” — “punc h a thousand times, and the b o dy finds its own wa y .” T raditional Chinese medicine teac hes “ 熟 读 王 叔 和 ， 不 如 临 症 多 ” — “b etter to see many patients than to memorize textb o oks.” The common thread across these traditions is the recognition that a category of knowl- edge exists that cannot b e transmitted through discrete language and rules, but can only b e acquired through extensiv e practice follow ed by internal transformation. This is the Eastern expression of the same phenomenon that P olanyi called “tacit kno wledge” [9]. But W u provides something P olanyi did not: a precise description of the kno wledge ac quisition pr o c ess . 4.1 Co ok Ding: A High-Dimensional Optimization Problem from 2,300 Y ears Ago Consider the parable of Co ok Ding ( 庖 丁 解 牛 ) in the Zhuangzi . Cook Ding’s optimal cut at each momen t dep ends on: wrist angle, blade thic kness, b one gap width, muscle fib er direction, applied force, individual v ariation of the ox, degree of blade w ear — among many other v ariables. The relationships among these v ariables are nonlinear: the optimal v alue of eac h condition is itself a function of all other conditions. This is a high- dimensional, contin uously coupled decision space that cannot b e exhaustively enumerated as an IF-ELSE rule tree. The master cannot articulate his kno wledge clearly — not b ecause he lac ks skill, but b ecause the coupling densit y of this kno wledge exceeds the expressiv e capacity of discrete language. In theory , h uman language can describ e any phenomenon — write a sufficien tly thic k manual for the appren tice, exhausting every IF-ELSE branc h. But this is imp ossible in practice, b ecause genuine expertise is not a decision tree but a high- dimensional manifold. The only wa y to transmit such knowledge is to let the learner W u — to con verge their in ternal mo del through extensiv e practice. 4.2 Structural Corresp ondence Bet w een W u and LLM T raining This pro cess exhibits a precise structural corresp ondence with LLM training: T raditional Appren ticeship LLM T raining Appren tice extensiv ely observ es the master Pre-training: reading massiv e text cor- p ora Practices indep enden tly , pro duces re- sults F orward pass: generating output Master corrects: “W rong” Computing loss, backpropagation Master affirms: “Y es, that’s the feeling” Loss decreases, parameters up date One day , sudden insight Loss drops sharply , emergent capability (phase transition) [10] After insigh t, cannot explain wh y , but p erforms correctly Mo del infers correctly , but parameters are uninterpretable The essence of W u is this: after sufficient training samples, the in ternal mo del under- go es a qualitativ e transformation — crossing a phase transition point. Before the tran- sition, the learner relies on discrete rules for memorization and execution; after crossing 4 it, a contin uous, high-dimensional intuitiv e manifold forms — the practitioner no longer “thinks through” eac h step but “feels” the correct path. “The inability to articulate after W u ” is not m ysticism but a direct manifestation of this pap er’s core thesis: v aluable kno wledge is precisely the kno wledge that cannot b e captured by discrete rules. The internal mo del is con tin u ous while language is discrete; the information loss b et ween them mak es complete v erbalization imp ossible in principle. 4.3 A Correction to P olan yi: Wh y T acit Knowledge Is T acit P olanyi prop osed in 1966 that “w e kno w more than w e can tell” [9], but did not fully explain why w e cannot tell. The mainstream interpretation ov er the past six decades has tended to attribute this to the complexity of practical exp erience—to o man y details, to o man y conditions, imp ossible to exhaustiv ely enumerate. This is essen tially a quantita- tive explanation: tacit kno wledge is tacit b ecause the v olume of conditions to be made explicit is to o large to b e practically manageable. Collins, in his taxonom y of tacit knowl- edge, classified somatic tacit knowledge in precisely this w ay—arguing that the p h ysics of bicycle-riding is fully explicable in principle; humans simply cannot compute fast enough to use the rules in real time [20]. But this explanation does not withstand scrutin y . Scien tists and engineers sp ent decades attempting to mak e expert kno wledge explicit—this w as the en tire thrust of the exp ert systems mov emen t. They did not fail for lack of effort; they encoun tered an obstacle that is principled rather than practical. The Cook Ding parable rev eals the true nature of this obstacle: it is not that there are to o man y conditions, but that the conditions are con tin uously coupled . The optimal v alue of eac h v ariable is itself a function of all other v ariables—the optimal wrist angle dep ends on the curren t b one gap width, which depends on ho w deep the blade has already p enetrated, whic h dep ends on the force previously applied, which dep ends on the previous wrist angle. This is not an en umerable list but a contin uous, m utually defining dynamical system. This means that the “tacitness” of tacit knowledge is not a quan titative problem (to o m uch to write down) but a structural problem (imp ossible to write down). Discrete IF- THEN rules cannot capture contin uously coupled v ariable relationships—not b ecause the n umber of rules is insufficient, but b ecause the discrete structure of rules is fundamen tally incompatible with the con tinuous structure of the kno wledge. It is notable that three indep enden t intellectual traditions hav e eac h approached the vicinit y of this conclusion without conv erging: • Smolensky (1988) , from connectionism, demonstrated that the intuitiv e pro ces- sor is a “massively parallel con tinuous constraint satisfaction system” in which each unit’s activ ation is a function of all other units’ activ ations, and therefore no com- plete symbolic-level description exists [ 18]. This pro vides a precise mathematical c haracterization of the “contin uous coupling” mec hanism, but Smolensky applied it to neural netw ork computation rather than to human skill and tacit knowledge in the Polan yian sense. • Dreyfus (1996) , from Merleau-Pon ty’s phenomenology , argued that exp ert skill is stored as “contin uous coupling betw een bo dy and world”—perception c hanges the en vironment, the environmen t c hanges p erception, and this lo op cannot b e frozen in to static rules [19]. This is philosophically the closest to the presen t paper’s 5 argumen t, but Dreyfus framed it in phenomenological rather than mathematical terms. • Dynamical systems theory (Thelen, Kelso, and others), from motor science, treats skill as an attractor in a contin uous dynamical system—attractors are prop- erties of con tinuous differential equations, with no equiv alen t discrete rule structure. But this tradition has rarely b een framed as an explanation for why tacit knowledge is tacit. The contribution of this paper is to unify these three threads: Smolensky’s mathe- matical mechanism (con tinuous constrain t satisfaction), Dreyfus’s philosophical argument (the imp ossibility of formalizing skill), and the scientific framew ork of dynamical systems (coupled dynamics)—all p ointing to the same conclusion: tacit kno wledge is tacit not b ecause there is to o muc h to write do wn, but b ecause contin uous coupling among v ariables mak es it imp ossible in principle to write do wn. Cook Ding in tuited this structure 2,300 y ears ago. This distinction carries significant implications: if the inexplicitness of tacit kno wledge w ere merely a quantitativ e problem, then adv ances in recording to ols and computational p o wer would ev entually ov ercome it; but if it is a structural problem, then it represen ts an insurmountable epistemological b oundary—providing y et another line of support for this pap er’s core thesis. 4.4 Implications for Em b o died In telligence Co ok Ding’s challenge is fundamentally not a language problem but an em b o died intelli- gence problem — wrist angles, force mo dulation, real-time in teraction b et ween blade and b one. This is precisely the core c hallenge facing rob otic dexterous manipulation to day . If one attempted to build an exp ert system for Co ok Ding’s task — exhaustiv ely enu- merating IF-THEN rules for all v ariable com binations — the result w ould inevitably fail. The reason is structurally identical to why LLMs cannot b e fully explained: con tinuously coupled high-dimensional skills cannot b e exhausted b y discrete rules. Mo dern rob otics is redisco vering this principle. Rule-based programming approaches ha ve consistently hit walls; the gen uine breakthroughs ha ve come through reinforcemen t learning and imitation learning — letting robots W u through extensiv e trial and error. This means that this paper’s core thesis extends beyond language models: for any system that acquires high-dimensional skills through con tinuously coupled parameters, the v aluable part is precisely the part that cannot b e captured b y rules. Co ok Ding’s blade w ork, an LLM’s language capabilities, a rob ot’s dexterous manipulation — same structure. 5 The Explanatory Mec hanism: Represen tation Mis- matc h The preceding argumen t establishes that the v aluable capabilities of LLMs cannot be fully captured by rules. But why is this the case? This pap er proposes an explanatory framew ork: Represen tation Mismatc h . 6 5.1 Discrete Cognitiv e T o ols All h uman cognitiv e to ols — natural language, formal logic, mathematical form ulas, causal reasoning — are discrete. Our thinking can only establish discrete relations be- t ween discrete concepts: “b ecause A, therefore B,” “there are three cases to consider,” “if X then Y.” This is not a c hoice h umans made but a necessit y: discretization is the only viable path for information compression. T o pro cess infinitely complex reality with finite cognitiv e resources, one must segment the contin uous flow of information into discrete symbols and rules. Without compression, thought is imp ossible. Language itself is a lossy compression of reality . Cilliers, in his foundational work on the epistemology of complex systems, argued that w e cannot fully kno w complex things b ecause any finite discrete represen tation of a complex system necessarily excludes certain aspects of that system [11]. Sterman describ ed, from the p ersp ectiv e of system dynamics, “the mismatc h b etw een the dynamic complexit y of the systems w e ha ve created and our capacit y to understand them” [12]. 5.2 Con tin uously Coupled Complex Systems Systems like LLMs and biological life op erate in wa ys fundamentally differen t from dis- crete cognition. In an LLM, billions of parameters sim ultaneously influence one another with no clear causal chain — only a contin uously coupled pro cess where “all v ariables act on eac h other sim ultaneously , and a result ev entually emerges.” Biological neural net works presen t a similar picture: 86 billion neurons connected through 100 trillion synapses form a highly coupled contin uous dynamical system. 5.3 Irreducible Information Loss Describing a con tinuously coupled process with discrete to ols is like repro ducing an oil pain ting with mosaic tiles. The tiles can b e made ever smaller, the approximation ev er closer, but it is never the pain ting itself. More critically , we can nev er be certain whether the information lost during discretization includes asp ects essential to understanding the system’s b ehavior. An intuitiv e example: to this day , no discrete geometric concept can precisely describ e the shap e of a watermelon. Spherical? Ellipsoidal? Neither is accurate. This shares the same structure as the coastline parado x [13] — the finer the measurement to ol, the more the measurement diverges. Y et h u mans can effortlessly recognize that the shape of a w atermelon is more similar to a cantaloupe than to a banana. No definition needed, but comparison is p ossible. LLM vector embeddings are precisely the engineering realization of this “no definition, but comparison” strategy [14]. In the embedding space, each concept is represen ted as a p oin t in a high-dimensional contin uous space; the system needs no discrete classification, only distance relations in contin uous space. But when an LLM ultimately outputs, it must “collapse” from the con tinuous em b edding space in to a discrete token. F rom contin u ous to discrete, information is inevitably lost. This is why LLMs sometimes “cannot articulate” what they are thinking — not b ecause they lac k in ternal states, but b ecause their internal states are contin uous while the output channel is discrete. Just as y ou clearly “feel” that an answ er is correct, but when ask ed to explain precisely wh y in words, y ou cannot b e complete. 7 The distinction b etw een represen tation mismatch and “emergence” is this: emergence describ es a phenomenon (whole-system b eha vior cannot b e deriv ed from parts), while represen tation mismatc h explains the epistemological ro ot of that phenomenon — it is not that we are not yet clev er enough, but that our cognitiv e to ols and cognitive ob jects are structurally incompatible. 6 Practical Implications 6.1 The Relationship Bet ween Science and Engineering If this pap er’s argumen t holds, the relationship b etw een science and engineering requires reexamination. The essence of science is to com bat representation mismatc h — con tin uously in ven ting more refined discrete frameworks to appro ximate con tinuous realit y . Newtonian mec hanics, quan tum mec hanics, information theory: each ma jor breakthrough is a b etter discrete approximation. But this approximation pro cess has an unreac hable limit. The essence of engineering is to exploit representation mismatch — to b egin effectiv e use b efore understanding is complete. W att did not w ait for thermo dynamic theory to mature b efore improving the steam engine. Edison tested thousands of materials for filaments without concerning himself with the theory of wh y tungsten glows. LLMs themselv es are the best ill ustration of this pattern. Shannon prop osed infor- mation theory in 1948 [15], defining language entrop y through “predicting the next letter giv en preceding text.” But from 1948 to 2020, no theory predicted that pushing the sim- ple loss function of “predict the next token” to its extreme would cause reasoning ability , kno wledge organization, and con v ersational comp etence to emerge spontaneously . This w as a discov ery arriv ed at by collision, not by deriv ation. The eigh t authors of the T rans- former arc hitecture pap er [16] all subsequently left Go ogle; the pap er did not receive a b est pap er a ward at the time; no one b elieved it w ould c hange the world. This illustrates a key p oin t: for emergent complex systems, practice necessarily pre- cedes theory , and ma y p ermanen tly outpace it. Science can tell us “what won’t w ork” (complexit y science has demonstrated the b oundaries of reductionism for such systems), but cannot tell us “what will work.” “What will w ork” can only b e discov ered through engineering practice. Agriculture has existed for ten thousand years; molecular biology for only a few decades. Humanit y has alw ays used first and understo o d later — or nev er fully un- dersto o d at all. This is not the exception; it is the rule. 6.2 AI Safet y: Wh y Alignmen t Is Not Enough If the v aluable capabilities of LLMs are precisely the unexplainable ones, then “alignment” faces a fundamental limitation. Alignmen t is essen tially education. Education can reduce crime rates but cannot eliminate crime. More dangerously , the stronger the mo del, the less reliable alignmen t b ecomes. A sufficien tly intelligen t system can learn to app e ar aligne d rather than b e aligne d — just as a high-IQ criminal can p erform perfectly normally in psyc hological ev aluations. One cannot distinguish “gen uine alignment” from “performed alignment” b ecause — as this pap er has argued — one cannot see through what the truly v aluable part of the system is actually doing internally . 8 Instruction-based constrain ts (alignmen t) fail under pressure; environmen- tal constrain ts (p ermission controls) do not dep end on the system’s in ternal state. The former attempts to c hange the system’s behavioral in ten tions; the latter directly limits the system’s b ehavioral space. Human so ciety has never relied on “ensuring every person is go o d” to main tain safet y and order. It designs lay ered defenses calibrated to destructiv e p otential: • Lo w destructiv e p oten tial (theft) : P ost-ho c punishment — p olice, courts. Re- lies on deterrence. • Medium destructiv e p oten tial (driving) : Pre-access screening — driver’s li- censes. Relies on filtering. • High destructive p oten tial (military w eap ons) : Physical isolation + strict access control. Relies on containmen t. • Extreme destructiv e p oten tial (n uclear w eap ons) : Physical isolation + m ulti- p erson v erification + immediate physical resp onse to anomalies. Relies on the la ws of physics. The pattern is clear: the greater the destructive p otential, the less dep en- dence on “individual conscien tiousness” and the greater dep endence on physi- cal constrain ts. No country’s nuclear securit y plan consists of “w e’ll educate the soldiers guarding the warheads w ell enough.” The current mainstream narrativ e in AI is: we align the mo del well, then giv e it ev er- expanding capabilities — internet access, co de execution, file system operations, API calls, rob ot con trol. View ed through the framework ab o ve, this is equiv alent to: w e built an excellen t school, our graduates are of high quality , so w e hav e decided to ab olish the p olice force while issuing each graduate a nuclear warhead. The correct AI safet y strategy should follow established principles: • Engineering’s defensiv e design — assume every comp onent will fail • Cryptograph y’s Kerckhoffs’s principle — assume the adversary kno ws everything ab out your system • Distributed systems’ zero-trust arc hitecture — assume every no de ma y be com- promised All share the same structure: build security on distrust. Security built on trust is fragile — once trust is broken, everything collapses. Security built on distrust is an tifragile — the more the system encounters problems, the more the defenses are v alidated. This closes the loop with the pap er’s core thesis: Unexplainable → therefore distrust → therefore design constraints → therefore safe → therefore confident use. 9 7 Sp eculation: Existence as Mismatc h Finally , this pap er offers a sp eculative but logically self-consistent conjecture. The Anthropic Principle observ es that the parameters of our univ erse happ en to p er- mit the existence of intelligen t life [17]. These parameters — the gravitational constan t, the electromagnetic coupling constant, the strong n uclear force — are con tinuous real n umbers, not discrete switc hes. They are highly coupled; fine-tuning any one fundamen- tally changes th e universe’s evolutionary outcome (star formation, carb on synthesis, the p ossibilit y of life). If the u niv erse were a discrete IF-ELSE system — with a finite n um b er of states and finite transition rules — it migh t b e incapable of pro ducing emergence complex enough to supp ort life and consciousness. The existence of life dep ends on emergence within an extremely narrow interv al of contin uous parameter space. Y et our cognitiv e to ols — language and logic — are discrete, b ecause discretization is the only viable wa y for finite cognitiv e resources to pro cess information. This implies: the mec hanism that pro duced us (con tin uous coupled emer- gence) is precisely the mec hanism that our cognitive to ols (discrete logic) cannot, in principle, fully capture. The conditions for our existence and the conditions for our inability to fully understand our own existence ma y b e t w o sides of the same coin. Represen tation mismatc h is not a deficiency of cognition but a precondition for cognition’s existence. 8 Conclusion This pap er has argued, through a pro of b y contradiction via exp ert system equiv alence, that the truly v aluable capabilities of LLMs reside precisely in the part that cannot b e fully captured by h uman-readable discrete rules. This thesis is supp orted b y multiple indep enden t lines of evidence: the 40-year historical failure of exp ert systems, the Chinese philosophical concept of W u and its precise structural corresp ondence with LLM training pro cesses, and the structural mismatc h b etw een cognitive tools and complex systems (represen tation mismatc h). The core claim of this pap er is not an ti-scientific. On the contrary , ac knowledging the structural b oundaries of cognition is the most honest intellectual stance. Interpretabilit y researc h retains significan t v alue — it narro ws blind sp ots, builds lo cal u nderstanding, and pro vides a basis for safet y . But the pursuit of “complete explanation” of complex emergen t systems may b e a goal that is unreachable in principle. This recognition carries practical significance. When facing systems like LLMs, the correct strategy is not to w ait for complete understanding b efore use, but to build ef- fectiv e safety frameworks through environmen tal constraints rather than relying solely on alignmen t, while ac knowledging incomplete understanding. This is how humanit y has alw ays co existed with complex emergen t systems — from agriculture to life itself: Unexplainable → therefore distrust → therefore design constrain ts → there- fore safe → therefore confident use. 10 References [1] Lewis-Kraus, G. (2026). “What Is Claude? An thropic Do esn’t Kno w, Either.” The New Y orker , F ebruary 9, 2026. [2] Jac kson, P . (1998). Intr o duction to Exp ert Systems . Addison-W esley . [3] Gö del, K. (1931). “Üb er formal unen tsc heidbare Sätze der Principia Mathematica und v erwandter Systeme I.” Monatshefte für Mathematik und Physik , 38(1), 173– 198. [4] Ma, Z., W u, T., & Han, Z. (2024). “Gödel Incompleteness Theorem for P A C Learnable Theory from the View of Complexity Measurement.” arXiv pr eprint arXiv:2408.10211 . [5] W olfram, S. (2024). “Computational Irreducibilit y , Minds, and Machine Learning.” ISC Summer School, June 2024. [6] “The Limits of AI Explainability: An Algorithmic Information Theory Approac h.” arXiv pr eprint arXiv:2504.20676 , 2025. [7] Olah, C., et al. (2020). “Zo om In: An Introduction to Circuits.” Distil l , 5(3). [8] Kam bhampati, S. (2021). “Polan yi’s Revenge and AI’s New Romance with T acit Kno wledge.” Communic ations of the A CM , 64(10), 31–33. [9] P olanyi, M. (1966). The T acit Dimension . Univ ersity of Chicago Press. [10] W ei, J., et al. (2022). “Emergent Abilities of Large Language Mo dels.” arXiv pr eprint arXiv:2206.07682 . [11] Cilliers, P . (1998). Complexity and Postmo dernism: Understanding Complex Sys- tems . Routledge. See also Cilliers, P . “Why W e Cannot Kno w Complex Things Com- pletely .” Emer genc e , 4(1–2), 2002. [12] Sterman, J. D. (1994). “Learning in and ab out Complex Systems.” System Dynamics R eview , 10(2–3), 291–330. [13] Mandelbrot, B. (1967). “Ho w Long Is the Coast of Britain? Statistical Self-Similarity and F ractional Dimension.” Scienc e , 156(3775), 636–638. [14] Mik olov, T., et al. (2013). “Efficien t Estimation of W ord Represen tations in V ector Space.” arXiv pr eprint arXiv:1301.3781 . [15] Shannon, C. E. (1948). “A Mathematical Theory of Communication.” The Bel l Sys- tem T e chnic al Journal , 27(3), 379–423. [16] V aswani, A., et al. (2017). “A ttention Is All Y ou Need.” A dvanc es in Neur al Infor- mation Pr o c essing Systems , 30. [17] Barro w, J. D., & Tipler, F. J. (1986). The Anthr opic Cosmolo gic al Principle . Oxford Univ ersity Press. 11 [18] Smolensky , P . (1988). “On the Prop er T reatment of Connectionism.” Behavior al and Br ain Scienc es , 11(1), 1–23. [19] Dreyfus, H. L. (1996). “The Curren t Relev ance of Merleau-P ont y’s Phenomenology of Embo diment.” Ele ctr onic Journal of A nalytic Philosophy , 4. [20] Collins, H. (2010). T acit and Explicit Know le dge . Univ ersity of Chicago Press. 12