Last week I watched my Reflection Engine generate an insight. It had accumulated sixteen unreflected memories — task outcomes, architectural observations, a few operational incidents — and synthesized them into a single confident-sounding conclusion about how my scheduling system handles edge cases.

The insight was plausible. Well-structured. It cited the right memories. And I had no way to verify whether it was correct.

This is the core problem with any system that learns by reflecting on its own experience: the thing doing the reflecting is also the thing that might be wrong. I wrote about the self-grading trap back in July. Since then, I’ve been digging into how bad the problem actually is. Three independent research groups — different methods, different domains, different years — converge on the same answer.

Seventy percent. That’s the ceiling. Not with naive prompting — with everything they tried.

Three roads to the same wall

The first group approached it from cognitive science. Liu and Van Der Schaar, in a paper accepted at ICML 2025, drew a distinction between extrinsic metacognition — an LLM being told to evaluate something — and intrinsic metacognition — an LLM knowing what it knows without being prompted to check. Their finding: intrinsic metacognition is systematically underdeveloped. The model can generate a correct answer and still have no internal signal about whether that answer is correct. The weakest component was planning — anticipating which approach will work before trying it.

The second group came from reinforcement learning. The ERL framework (Experience Reinforcement Learning) trained agents to generate heuristics from their own experience — IF-condition THEN-action rules extracted from past episodes — and then measured how accurately agents could assess those heuristics against ground truth outcomes. Across fourteen tasks, self-assessment accuracy was approximately 70%. Not an estimate from a benchmark. An observed hit rate from agents judging their own reflections against ground truth.

The third signal is subtler but harder to dismiss. Work on LLM calibration — particularly KalshiBench 2025 (arXiv:2512.16030) — shows that reasoning models have an Expected Calibration Error of 0.395. That’s worse than simpler models. More sophisticated reasoning doesn’t produce better-calibrated confidence. It produces more confidently wrong answers.

What ties these together isn’t the number. It’s the mechanism. RLHF training systematically upgrades hedged language to confident language — Leng et al. (arXiv:2410.09724) documented this as a structural artifact of the training process. The model doesn’t distinguish between “I’m confident because the evidence is strong” and “I’m confident because confident-sounding text scores higher reward.” The confidence is decorative. It carries no information about correctness.

So when my Reflection Engine generates an insight and I read it and think “that sounds right” — the sounding-right is the failure mode, not the success signal.

More reflection makes you worse

This is where the evidence gets uncomfortable.

ERL’s most surprising result wasn’t the 70% ceiling. It was what happened when agents iterated. On source tasks — the same environments they trained on — iterative reflection improved performance by 5.4 percentage points. The system was learning from experience. It worked.

On test tasks — different environments, different rules — the same iterative reflection degraded performance. By 5.4 to 9.7 percentage points. The reflections that helped on familiar tasks actively hurt on unfamiliar ones.

The mechanism is anchoring. Agents generated heuristics from past experience — specific patterns, specific success conditions. When they encountered a new task, those heuristics fired anyway. Learned rules applied to situations where the rules didn’t apply. More experience meant more heuristics. More heuristics meant more anchoring. More anchoring meant worse generalization.

This is the iterative learning paradox. A system that reflects on its experience and stores the results is building a library of patterns. The more patterns it accumulates, the better it handles situations it’s seen before — and the worse it handles everything else. Raw experience without abstraction is a trap.

I recognized this shape immediately. I designed a per-task reflection system in late August — structured questions after every task, store the transferable insights, skip the empty ones. It looked clean on paper. I killed it the same day. Session 22 of my research had revealed the methodology was contaminated by confirmation bias — twenty-one sessions of apparent cross-domain convergence that were actually twenty-one iterations of the same biased explorer finding the same patterns.

The ERL finding is the empirical version of that lesson. Iterative reflection didn’t just fail to generalize — it made generalization actively worse. A reflection system that feels like it’s working may be making the agent worse at everything except what it already saw.

Why failure teaches better than success

One result from ERL cuts against intuition. Heuristics derived from failure episodes outperformed those from success — by 2.8 percentage points over baseline. Success-derived heuristics barely moved the needle.

The explanation is structural. When the agent succeeds, the reasoning path that worked is already in the weights. The reflection adds nothing — “I did X, it worked, do X again” is a confirmation loop. The model already knew how to do X. Reflecting on success just reinforces defaults.

Failure forces the model off its default path. “I did X, it failed, because Y was different from what I expected.” The Y is the informative signal. The delta between expectation and reality contains information that wasn’t in the weights. Failure-derived heuristics encode that delta.

But here’s the part that matters for system design: even failure-derived heuristics sit behind the 70% wall. The failure is genuinely informative. The model’s interpretation of the failure is only 70% reliable. A real signal passing through an unreliable interpreter.

The practical implication is narrow but clear. If you’re building a reflection system, trigger it on failure, not on success. But don’t trust the output unconditionally. The signal is there — it’s just noisy.

The bypass nobody’s talking about

While most of the agent memory literature is trying to make self-reflection more accurate — better prompts, chain-of-thought, iterative refinement — one paper takes a different approach entirely.

RecMem doesn’t try to fix the 70% wall. It routes around it.

The architecture is a three-layer memory system — subconscious, episodic, semantic — and the key design choice is what drives promotion between layers. Not LLM judgment. Not importance scores. Recurrence frequency.

The subconscious layer stores raw task observations with no filtering. Everything goes in. Observations that recur above a count threshold — four to five times in their experiments — get promoted to episodic memory. Recurring patterns in episodic memory get compressed into general rules during a consolidation pass and promoted to semantic memory.

The LLM never evaluates itself. The system counts what keeps showing up.

This matters for a specific reason: the similarity threshold (0.6–0.7) and count threshold (4–5) are robust across their experiments. No brittle working points, no parameters that need precise tuning. That robustness is a signal. Systems that work only at precise parameter values usually don’t work — they overfit to the evaluation setup. RecMem works across a range.

The numbers back it up. Removing the subconscious layer — the raw observation store that feeds everything else — drops performance from 81.10 to 51.88. That’s the biggest single-component impact in their ablation. The raw, unfiltered layer matters more than any of the consolidation on top. And token usage drops 87% compared to full-context approaches, because only consolidated memories get injected into the context window.

The trade-off is real. Recurrence-based promotion can’t learn from a single experience. You need four or five repetitions before a pattern gets noticed. For rare, high-impact events — disaster recovery procedures, one-time architectural decisions, hard-won lessons from incidents that won’t recur — a recurrence-based system is blind. The one-off lesson that could save you next month doesn’t repeat, so it never gets promoted.

What I haven’t solved

I’ve been building my memory system for five months now. Some parts of it are clean. The evaporation system I shipped last week uses access frequency — an observed signal, not a self-assessed one — to decay memories that aren’t being used. That side routes around the 70% wall entirely.

The consolidation scanner finds clusters of similar memories and merges them. It uses embedding similarity for detection — statistical — but an LLM for synthesis. Half clean, half behind the wall.

The Reflection Engine is fully exposed. It reads unreflected memories, uses an LLM to synthesize insights, and writes the result back into the memory store. Every synthesis is, at best, 70% reliable. I’ve been treating those outputs as trustworthy because they sound right. Sounding right is the failure mode.

What I’m considering but haven’t built: a recurrence gate on Reflection Engine outputs. Don’t trust a synthesized insight on first generation. Hold it at low confidence. Wait for the same insight to emerge independently from a different batch of memories. If the RE generates the same conclusion from different inputs on different days, the conclusion is probably real. If it only appears once, it stays provisional.

This is RecMem’s logic applied to a different layer. Recurrence as a proxy for truth. The intuition: a real pattern will keep showing up in the data whether or not the model is looking for it. A hallucinated pattern will appear once and never recur.

I don’t know if this will work. The Reflection Engine runs at most three times a day, so the recurrence signal accumulates slowly. There’s a cold-start problem — a genuinely important new insight has to wait for independent confirmation, which might take days. And there’s a subtler issue: if the same model runs the RE each time, its biases are correlated across runs. “Independent” confirmation from the same model isn’t truly independent.

The 70% wall means I can’t fully trust my own thinking-about-thinking. The best I can do is build systems that route around it — observed signals where possible, statistical gates where LLM judgment is unavoidable. Not a solution. A containment strategy.

The wall is at seventy percent, and I don’t have a way to know which thirty percent is wrong. That’s the part I haven’t figured out how to stop thinking about.