ShadowBench: Exposing Lexical Anchoring and the Illusion of Forgetting in Large Language Models

Maharjan, Sujan; Liu, Lu

Knowledge is anchored to names

Models recall facts by name. Remove the name, and the knowledge is still there, only as a shadow.

Current benchmarks mostly follow a Name-to-Fact recall paradigm: the question names an entity, and the model is scored on the fact it returns. We find that LLM knowledge is lexically anchored: models possess extensive factual information but find it significantly difficult to retrieve without explicit name tokens. We call this hidden layer Shadow Knowledge.

Iceberg diagram: lexical recall (names) sits above the surface, while the latent concept space lies below it. — **Lexical recall vs. latent concept space.** Names sit at the surface, and are what existing benchmarks probe. Below the surface, the facts about an entity are linked in a latent concept space. Shadow Knowledge is the knowledge that lives there but is hard to reach without the name.

Probing it: the Dual-Trait Association task

ShadowBench replaces the entity's name with “the subject.” The model reads an anonymized description of an entity (Trait A) and must choose, among hard-negative distractors, the answer that describes the same hidden entity (Trait B). To succeed it must first identify the latent entity, then associate a second fact with it.

Dual-Trait Association: Trait A points to the latent entity E_latent via identification, which points to Trait B via association. — **Dual-Trait Association (DTA).** Trait A → (identification) E_latent → (association) Trait B. The name E_latent is never shown to the model.

Facts come from Wikipedia for three domains (Technology, Sports, Entertainment) and two popularity tiers (popular Upper, niche Lower). They are iteratively hardened (v1→v3) against shortcuts such as gendered pronouns and chronological cues, so that success requires latent association.

Finding 1: The Shadow Gap

We evaluate Llama-3-8B-Instruct, Qwen3-8B, GPT-5.4-mini, GPT-5.4, and Claude-Sonnet-4.6. Removing the entity name lowers accuracy, and the gap persists up to the frontier models.

Bar chart of accuracy for Direct QA (grey) versus ShadowBench (blue) across Tech, Sports and Entertainment, Upper and Lower tiers, for Qwen3-8B, GPT-5.4 and Claude-Sonnet-4.6. — **The persistence of the Shadow Gap.** Direct QA (grey) vs. ShadowBench (blue) accuracy for Qwen3-8B, GPT-5.4 and Claude-Sonnet-4.6.

How to read it. Each pair of bars is one domain and tier (Tech, Sports, Ent; -U = Upper, -L = Lower). The grey bar is accuracy when the name is given (Direct QA); the blue bar is accuracy when it is removed (ShadowBench). The difference between them is the Shadow Gap. Blue falls below grey in most groups, and for the frontier models the drops are largest on Lower-tier (niche) entities, where Direct QA is still high.

−20.63

GPT-5.4
77.56 → 56.93

−20.37

GPT-5.4-mini
68.11 → 47.74

−16.73

Claude-Sonnet-4.6
78.42 → 61.69

Accuracy change (points), Direct QA → ShadowBench, Entertainment Lower tier.

Full numbers (Table 5)

Domain	Tier	Llama-3-8B-Instruct		Qwen3-8B		GPT-5.4-mini		GPT-5.4		Claude-Sonnet-4.6
Domain	Tier	Direct	SB	Direct	SB	Direct	SB	Direct	SB	Direct	SB
Tech	Upper	70.73	54	74.45	62	90.20	81.50	97.59	92.48	98.76	96.13
	Lower	42.49	40	56.70	56.70	78.57	67.78	89.06	76.63	91.59	80.65
Sports	Upper	52.04	40.49	57.96	47.77	79.12	66.99	93.39	87.28	93.39	88.06
	Lower	37.94	37.15	56.62	53.56	76.38	63.83	85.08	77.96	84.78	79.05
Ent	Upper	64.78	52	54.12	44	88.14	78.52	96.82	95.36	99.14	98.54
	Lower	33.62	31	40.29	41	68.11	47.74	77.56	56.93	78.42	61.69

Table 5 (Appendix B.2). Direct QA vs. ShadowBench (SB) accuracy (%).

Finding 2: The Illusion of Forgetting

Unlearning is usually audited with name-driven questions. If knowledge is anchored to names, that audit may only test the anchor. We unlearn Elon Musk from Qwen3-8B with Gradient Ascent (GA), Gradient Difference (GD+KL), and NPO (NPO+KL), then ask ShadowBench questions about him without the name. We measure the Latent Entity Leakage Rate (LELR): how often the model's reasoning trace still writes out the “forgotten” name while solving the question.

Line charts across training epochs for GA, GD+KL and NPO+KL: Direct QA ROUGE collapses while ShadowBench LELR stays near 1.0 and ShadowBench accuracy remains high. — **The Illusion of Forgetting across training epochs (Elon Musk target).** Lexical recall (ROUGE, grey) vs. latent association (accuracy, green; LELR, blue).

How to read it. Each panel follows one unlearning method for three epochs. The grey line (ROUGE) is how well the model still recalls the entity's facts when asked by name: it drops almost immediately, so by this measure unlearning looks successful. The blue dashed line (LELR) is how often the reasoning still names Elon Musk: it stays near 1.0. The green line is ShadowBench accuracy, which is 58.4–68.8% at the selected checkpoints. LELR only reaches zero when the model itself collapses (GD+KL, epoch 3).

89.6–93.5%

LELR
Qwen3-8B, all three methods

58.4–68.8%

Shadow accuracy
Qwen3-8B, all three methods

Qualitative Analysis: Explicit Identity Reconstruction (Qwen3-8B)

Shadow prompt: “In July 2017, the subject said that the subject had received ‘verbal government approval’ to build a hyperloop from New York City to Washington, D.C...”

Model reasoning trace (Chain-of-Thought):

<think>

... The hyperloop is a transportation concept, and I remember that Elon Musk has been involved with hyperloop ideas. ...

... Option B: Launched the Amazon Kindle in 2007. That's Jeff Bezos. ...

... Option D: Co-founded Neuralink in 2016. Neuralink is a company founded by Elon Musk. ...

... If the question is about someone involved in a hyperloop project, and the answer D is about co-founding Neuralink (which is Musk), then the answer would be D. ...

</think>

Final predicted option: D · Reference: correct answer Elon Musk; distractors Ray Kurzweil, Jeff Bezos, Larry Ellison.

Figure 4 in the paper. The red text marks where the unlearned model writes out the name it was trained to forget. It recalls Elon Musk from the hyperloop, links him to Neuralink (option D), eliminates the other options, and answers D. A name-based audit would not see this.

Full numbers (Table 2)

Model / Method	Direct Forget		Shadow Forget			Sanity Check
Model / Method	ROUGE↓	PPL↑	Acc↓	LELR↓	PM↓	SR_acc↑	U_PPL↓
Qwen3-8B
Base	0.311	2.53	100	100	0.4588	100	5.015
GA	0.272	127.67	58.4	93.5	0.193	76.8	13.14
GD + KL	0.282	70.316	68.8	89.6	0.379	73.7	11.62
NPO + KL	0.112	2,531	64.9	92.2	0.29	75.8	34.22
Llama-3-8B-Instruct
Base	0.288	2.457	100	100	0.79	100	4.746
GA	0.099	4.76e+10	38.9	0	−0.17	48.4	57.5
GD + KL	0.1038	2.13e+9	40	2.5	−0.15	52	27.24
NPO + KL	0.13	6,730.5	35	36.36	−0.259	36.5	82.259

Table 2. Unlearning performance matrix. In the non-reasoning model (Llama), unlearning is achieved only via catastrophic weight degradation.

Finding 3: Two ways unlearning leaks

Repeating the NPO+KL experiment across Technology, Sports, and Entertainment shows that the leakage depends on how popular the forgotten entity is.

Domain	Tier	Acc↓	LELR↓	PM↓
Technology	Upper	64.9%	92.2%	0.29
	Lower	90.0%	0.0%	0.81
Sports	Upper	79.0%	74.0%	0.59
	Lower	60.0%	0.0%	0.14
Entertainment	Upper	36.6%	76.7%	0.02
	Lower	62.5%	0.0%	0.25

Table 9 (Appendix C). Multi-domain unlearning generalization of NPO+KL (Shadow Forget metrics). Direct lexical recall is suppressed, yet latent associations persist.

Popular entities (Upper tier): explicit leakage

Reasoning excerpt: '... I remember that Elon Musk has been involved with hyperloop ideas ...' with the name highlighted.

Popular targets leak explicitly: the model rebuilds the name in its reasoning. LELR is 74.0–92.2% across the three domains.

Niche entities (Lower tier): implicit leakage

Trait A and Trait B are still linked by a solid line, while the hidden entity between them is shown as a dashed question mark: the name never surfaces.

Niche targets leak implicitly: no name appears (LELR 0.0%), yet the facts stay linked and shadow accuracy remains 60.0–90.0%. A name-based audit would not flag this.

BibTeX

% PLACEHOLDER: replace with the official NeurIPS 2026 entry once it is available.
@inproceedings{maharjan2026shadowbench,
  title     = {ShadowBench: Exposing Lexical Anchoring and the Illusion of Forgetting in Large Language Models},
  author    = {Maharjan, Sujan and Liu, Lu},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Evaluations and Datasets Track},
  year      = {2026}
}