Language modelscan't say“I never saw that.”

The problem

Hallucination from memory is a classification error between “false” and “not stored”.

Attention always returns a blend of what it holds. It has no output for “no match.” When a model reads a summary or a long context and a fact is missing, it reads the silence as “that did not happen.” The answer sounds confident because nothing in the architecture can express absence.

What we proved

Address memory by identity, exactly, instead of by similarity. Replicated on two models from different families, three seeds each.

0 of 18 seed × model × experiment cells flipped direction. The rewrite check confirmed all facts preserved when synthetic sessions were rewritten into natural chat.

Similarity breaks as memory grows. Exact identity does not.

Share of questions about things that were never stored that receive a confident wrong answer, as the memory grows from 1,000 to 99,000 entities.

Tiny transformer trained from scratch on a synthetic world. The comparison is architecture-level, not a product benchmark.

Even with a perfect key, similarity lookup answers 46–50% of absent questions confidently and wrongly at 99,000 entities. Exact identity answers 0%.

Per-seed similarity values at 99k: 49.0 / 49.9 / 45.9.

Source: experiments/M11b_binding/results/eval_seed{0,1,2}.json, eval_seed{1,2}_i.json, eval_seed{3,4,5}_iii.json

Disclosure

Exact-identity seed 2 shows an accuracy dip at 10k/99k on one question type (true/false “is X's value V”), not seen on seeds 0, 1, 3, 4, 5. Cause unconfirmed.

Honest limits

• Results are on synthetic worlds and small models: tiny transformers trained from scratch, Qwen3.5-4B, Gemma-4-E4B-it.
• The identity result is replicated (2 models, 3 seeds each; 6 seeds on the tiny transformer). The in-network fix (M14) is a hypothesis.
• The comparison of exact identity vs similarity at 99,000 entities comes from one tiny architecture.
• No test yet on real user data, on models above 4B, or at production scale.
• Several early pilots are single-seed and are labelled indicative.

What failed, and we say so

Every experiment has a written kill line before it runs. Killed experiments stay in the repository with the reason.

• M04Killed

  Rewrite facts into joint documents so the model learns multi-hop links

  Chance level: 0.045 vs 0.040 for the control. The facts were already known, so there was nothing to learn from.

  Source: experiments/M04_cooccurrence_consolidation/results/, history/LEDGER.md

• M11Killed

  A learned continuous key inside the network

  Accuracy 100% at 10 entities, 92.4% at 10,000, 70.3% at 99,000. The absence gate was calibrated at small scale and opens at large scale.

  Source: history/LEDGER.md (M11 row)

• M13Killed

  A trained controller that decides “answer / reload / ask the model”

  +3.4 and +3.7 points over the model's own choices against a +10 requirement. A hand-written rule scored as high (88.3%).

  Source: experiments/M13_calibrated_controller/results/, history/LEDGER.md

• B02Killed

  The model compiles repeated answers into programs it no longer needs to ask about

  A 4B model could not write the programs: 13.8% of compile attempts were usable, and answer accuracy was 11.9%. Parser, token limit and sandbox were ruled out.

  Source: experiments/B02_compilation_curve/results/diag_b02_qwen3.5-4b-hf*.{log,jsonl}

• M07Killed

  Write only surprising facts to memory

  +1 point overall. Killed on its criterion.

  Source: history/LEDGER.md

We also retired a broader architecture (“the LLM as a peripheral of a persistent controller”). Its controller and compiler halves have no supporting evidence. The memory result stands on its own.

HYPOTHESIS — NOT YET TESTED

Next: put the structure inside the model

Today the exact store sits beside the model. The next step is an exact lookup inside the network, with the mathematics written down before the run.

We have derived why similarity lookup fails at scale, and what a fix must satisfy. These are derived relations, not tested fixes.

Attention puts weight ≥ 1 − ε on the right key when:
    β · (1 − s) ≥ ln(N / ε)
For random keys in d dimensions, distractor similarity:
    s ≈ √(2 ln N / d)
So the key width must grow with memory size:
    d ≥ 2 ln N / μ²
And the “not stored” threshold must move with N:
    τ_N ≈ √(2 ln N / d) + δ

At d = 64 and N = 99,000 this predicts a distractor similarity near 0.6. We measured 0.53–0.58. A threshold fixed at small scale (0.5) therefore opens at large scale. That is how the earlier network-layer experiment (M11) failed.

1. M14Not run

   Calibrated read

   Does a size-aware threshold and a wider key remove the 99,000-entity failure?

   Kill line (pre-registered): confident-wrong on absent questions stays above half of M11's rate. TODO(owner): the brief says 45%, the claims register says 54.6%; confirm which

2. M14Not run

   Exact two-level code

   A bucket × slot identity read through a zero-initialised gate. Target: 0% absent confident-wrong at 99,000 entities, inside the network.

3. M12Not run

   Integration on a real model

   A frozen Qwen3.5-2B with an exact resolver outside the network and the value injected into the residual stream.

Product-key memory, Titans and nested learning explore memory inside models. Our claim is narrower: exactness and explicit absence that hold as memory grows.