I’ve been researching depth-residual routing & delta memory- Kimi K3 proved this holds at 2.8T params. But only for models trained this way
I then asked: can you retrofit this onto existing LLM/world models?
Introducing Retro-DARC: a Delta Attention Residual Compute adapter
that retrofits depth-selective memory onto existing models without retraining:
EA exact no-op at insertion (bitwise 0.0 deviation on 3 public checkpoints - proven exact even through MoE routing and early exit), adapter-only training, exact rollback, and content-level memory audits that weight-touching adapters can't express by construction.
The early results are promising. A ~262K-param adapter beat LoRA on a frozen public checkpoint, inserted at zero behavioral risk. And it's the first memory you can audit: zero the bank and the frozen loss returns to machine precision; shuffle it and 70–107% of the gain vanishes. That's a model measurably using its own computation history.
Retro-DARC is useful for improving K3 too. K3 serves in MXFP4/MXFP8, and layer outputs - exactly what its AttnRes retrieves - collapse under that corruption in my tests (0.04–0.75 top-1) while normalized delta keys hold 0.86–0.99. A Softmax1 null route with proven mass bounds lets the router decline to retrieve instead of forcing the sinks and outliers OASIS documented; typed memory turns "what is depth retrieval doing?" into a runnable audit. And one experiment is free: K3 already computes a per-step KDA update magnitude and throws it away - that's a depth-saliency prior waiting to be read.
For world models, the same contract becomes one typed memory interface: Retro-DARC-X gives world/action planners a single bank - observations, actions, tools, latent causes, traces, physics residuals, failures - the substrates a planner needs to remember in order to act over long horizons in the physical world.
You can experience the thesis yourself. I created
lucidre.am/memory which runs the paper's memory contract inside a walkable AI world. Innovation Memory makes your interactions survive looking away or reloading, and Dream Residue turns your movement into an inspectable glowing trace.
Join the main
Lucidre.am co-creation exhibition - the contest opens Wednesday. One of the six memory applications of the paper - saliency-weighted curation, will rank worlds by genuine novelty rather than recency and pick each world's most surprising viewpoint. And we're bringing live shows to SF + LA next month: screens showcasing worlds that remember, so you can literally play hide-and-seek with a dream that knows where you hide. Hide-and-seek is the exact test every 2026 world-model benchmark says the field fails, something leaves the frame, keeps changing, and has to come back right.
This has been a fun little one-person research project so far! I’d love feedback and keep working on it. I’ve had over 100 references- thanks for all the prior work from teams that inspired me!
I'd also love to intern at a GPU-rich lab (lol) and keep making beautiful worlds - if you let me keep running my remaining tests :)
2MB adapter kit, demo, prior iterations:
adacyb0rg.github.io/retro-da…
Paper:
adacyb0rg.github.io/retro-da…