NVIDIA · arXiv 2610.12582 · posted Thu 8 Oct 2026 · 26 pages
One model,
many depths.
One Model to Loop Them All: Elastic Latent Recurrence for Language Models. A looped language model that costs nothing extra at one loop and gets better with every loop you add. Then Dory1.0-4B, the model NVIDIA says was built on this recipe, re-measured.
→ / space / swipe to advance · o contents · f fullscreen. Paper numbers come from its tables; Dory numbers from NVIDIA's model card and your two Modal experiments (19–20 Oct). Hover or tap a mark for its value.
The idea
A loop adds depth without adding parameters.
Effective depth D(k) = Lin + k·Lrec + Lout. Koi-4B splits its 52 layers 14 / 24 / 14. Dory splits 72 layers 16 / 40 / 16, which is 8 / 20 / 8 attention-plus-MLP blocks. Memory is per-loop KV for attention layers inside the loop; with last-loop caching it stays at 1×.
The paper's word is elastic depth: one checkpoint, served at any loop count it was trained on. It works like a reasoning-effort setting, except the extra thinking happens in the hidden state rather than in more tokens.
The problem
Looped models have paid a one-loop tax.
If one loop is worse than a plain model of the same size, the latency tier goes to a separately trained plain model. The goal here is narrower than "is looping compute-efficient?": make the model elastic without the tax.
- A gated state update with a learned step size per token and channel.
- Loop counts mixed inside each optimizer step: a rotating slice of data-parallel ranks always runs one loop.
- Each pass told where it is: conditioning on loop index and budget.
Plus three ways to keep it cheap: switch to mixed depths late in pretraining, reuse the first loop's MoE routes, keep only the last loop's KV cache. Every ELR model trains with its baseline's optimizer settings unchanged.
Ingredient 1 · the state update
Take a gated step toward the block's proposal. Don't replace the state.
si = si−1 + βi ⊙ (ri − si−1)
βi = σ(Wβ s̃i + bβ) ⊙ softplus(γ)
| Rule (GPT-0.5B) | What happens |
|---|---|
| Replace: si = ri | Highest one-loop loss; gets worse from 4 to 5 loops |
| Residual: si−1 + ri | Stops improving after 3 loops |
| Delta: fixed per-channel α, init 0.05 | Second at every depth, 0.006–0.007 behind |
| Adaptive delta (β above) | Only rule matching the baseline at 1 loop; lowest loss at every depth |
Wβ, bβ start at zero and γ at 0, so every step starts at σ(0)·ln 2 ≈ 0.35. Cost: d² + 2d parameters, about 0.2% of a 4B model.
Ingredient 2 · loop sampling
Mix loop counts inside every update, not across updates.
Huginn draws one loop count per step for all ranks. Even with one loop drawn a third of the time, the one-loop model stays above baseline (1.6871). Forcing a rotating fraction f of ranks to run one loop puts one-loop and deep gradients in the same update, at no step-time cost.
p1 = 1/K + (1 − 1/K)·f and k̄ = f + (1 − f)(K + 1)/2 (Eq. 3). The paper picks f so that p1 ≈ ⅓: 15% at K = 5, 10% at K = 4.
Fixed depth vs mixed depth
A model trained at one depth breaks at every other depth.
- Specialists win at home (1.603 at 5 loops, against 1.627 mixed) and collapse away from it: the 1-loop model reads 4.242 at 5 loops.
- The mixed model improves monotonically, 1.683 → 1.627, and serves five depths with one checkpoint.
- A surprise: the model trained only at one loop beats the plain baseline, 1.660 against 1.685, with the same layers run once. At one loop the gate is a learned skip around the middle block, and "the gate appears to help even without recurrence."
The paper's open question: conditioning closes most of the gap to the specialists at intermediate depths but not at the endpoints.
Ingredient 3 · conditioning
Tell each pass which pass it is, and how many are coming.
x̃ = (1 + λ(ci)) ⊙ RMSNorm(x) + μ(ci)
y = x + (1 + ω(ci)) ⊙ f(x̃ + A(g(ci) ⊙ Bx̃))
AdaLN/FiLM-style scale and shift, a low-rank correction with a loop-dependent gate, and an output rescale, on every recurrent and output layer. Progress τ = i/k and budget κ = k/K follow LoopFormer.
Grafted at the switch, starting from zero. Every head starts at zero, so the conditioned model is exactly the one-loop checkpoint at every k, and the deeper passes learn to depart from it.
A trade-off the paper notes but doesn't test: with a budget input, loop i depends on k, so a request can't stop early mid-computation. They serve each request at a fixed k and leave adaptive depth to future work.
Keeping training cheap
Train at one loop for most of the run. Switch to mixed depths late.
- 4B and 30B-A3B: last 25% mixed → 1.16× the baseline's training compute, against 1.62× mixed throughout.
- Koi-4B: last 50% of 7T tokens mixed → 1.31×.
- GPT-0.5B: 1.79× from scratch; a 75% start costs 1.20× and keeps 84% of the five-loop gain, for +0.009 loss at five loops. One loop stays below baseline at every start point.
Results · same data, same layers
No tax at one loop; four loops add about five to seven points.
Both are GDP-Hybrids: 23 Gated DeltaProduct + 6 attention + 23 feed-forward layers, split 14 / 24 / 14. At one loop single tasks swing both ways (4B: GSM8K −5.3, MATH-500 +5.2), and every number is a single run, so a half-point gap is hard to call. Largest 1→4 gains: GSM8K +11.8, MATH-500 +11.3 (4B).
What a loop costs to serve
Loops cost latency; last-loop KV keeps memory flat.
Last-loop KV cache
One cache per attention layer, overwritten each loop; a loop-i query reads earlier tokens' final-loop keys. Koi at 4 loops: 3.0 GiB instead of 7.5 GiB per 128K-token request (BF16), for 0 to −3 points (AA-LCR −3.0, HLE −1.4, SciCode −1.4). No retraining.
Cross-loop route replay (MoE)
Route each token once, on loop 1, and reuse the experts later: router calls 53 → 23 per 4-loop pass, update time −11.7% (−19.6% at fixed k = 4), loss +0.0038. Inference gains only 1.2%. Tokens keep just 53% of their loop-1 experts by loop 4, yet replayed outputs keep cosine 0.77 with fresh ones.
The full recipe, post-trained
Koi-4B: 18 at one loop, 29 at four.
- The second loop is 8 of the 11 points. Two loops sit above Qwen3.5-9B and level with Gemma 4 26B-A4B (6.5× the parameters).
- Training: 7T tokens at 8K, one loop for the first half; 50B tokens of 256K context extension; 4 epochs of SFT; 200 RLVR steps at a 4 : 1 : 1 : 1 ratio of 1–4 loops, with a pass-rate curriculum that sends harder prompts to more loops.
- Per benchmark, 1 → 4 loops: GPQA 59.9 → 72.8, IFBench 61.2 → 71.8, Terminal-Bench 12.1 → 28.0, SciCode 17.7 → 33.4, AA-LCR 51.3 → 68.3.
Koi has no matched baseline. Its one-loop score is compared with other open models, so data and post-training differences are folded in.
The released model
Koi is not Dory. Dory's loop has no gate and no conditioning.
| Koi-4B · the paper | Dory1.0-4B · released Fri 16 Oct | |
|---|---|---|
| Backbone | GDP-Hybrid: 23 Gated DeltaProduct + 6 attention + 23 MLP layers, no position encoding | nGPT, the normalized transformer: 36 attention + MLP blocks, 27 with a 1,024-token sliding window, 9 with full attention, RoPE |
| Input / loop / output | 14 / 24 / 14 layers | 16 / 40 / 16 layers (8 / 20 / 8 blocks); 5 of the 9 full-attention layers sit in the loop |
| Loop update | Adaptive-delta gate | No gate: s ← norm(core(norm(s + hin))). nGPT's per-layer α steps act inside |
| Loop conditioning | Yes, +155.6M parameters | None. The 507 tensors hold only per-layer α vectors; loop i is identical at every k |
| Pretraining | 7T tokens, BF16 | 6T tokens, NVFP4 for the first ~4.5T, then BF16; 300B tokens of context extension |
| Default | — | 2 loops; 1–4 supported |
| GPQA · 1 / 2 / 4 loops | 59.9 / 69.9 / 72.8 | 69.19 / 72.41 / 72.16 |
| IFBench | 61.2 / 70.1 / 71.8 | 74.29 / 77.94 / 79.0 |
| AA-LCR | 51.3 / 65.7 / 68.3 | 63 / 71 / 73 |
| Terminal-Bench 2.1 | 12.1 / 22.9 / 28.0 | 12.5 / 18.1 / 22.2 |
Koi: the paper's Table 10 (its harness). Dory: the model card (NeMo Gym), config.json, and the safetensors header at revision b366b87f, read Tue 20 Oct. Your independent PyTorch forward pass, built from NVIDIA's vLLM PR and llama.cpp fork, matches vLLM to 0.0002 nats with no gate and no conditioning. NVIDIA (Shantanu Acharya, 20 Oct): Dory was "built on" this paper's methodology and training recipe; the Dory tech report is due by the end of the month. Dory is stronger at one loop and gains less from looping.
Your experiments 1 and 2 · Mon 19 – Tue 20 Oct · Modal
Dory reproduces its card, and only the second loop pays.
Inside Dory · one 16-loop pass, read out after every loop
Loop 1 rewrites the state; loop 2 polishes; then it drifts to a worse fixed point.
- Each loop count is its own distribution. Text Dory writes at k loops is predicted best at k loops; the 1 → 2 shift is about 6× the 2 → 4 shift (0.088 vs 0.015 nats per token).
- It sharpens; it doesn't unlock. The second loop adds 4–9 points of solve rate on questions one loop sometimes gets, and 0.7 on questions it never gets. Difficulty order doesn't change, so routing questions to loop counts loses to a fixed 2.
Dory against other small models · same harness
Dory's edge is instruction following, and it's there at one loop.
| Model | GPQA | IFBench | IFNovel (unpublished) | AA-LCR |
|---|---|---|---|---|
| Dory, 1 loop | 69.2 | 74.2 | 87.3 | 62.7 |
| Dory, 2 loops | 74.1 | 78.6 | 89.1 | 71.3 |
| Qwen3.5-4B | 78.8 | 60.4 | 47.4 | 64.0 |
| gpt-oss-20b (high) | 64.9 | 68.6 | 77.4 | – |
| Nemotron-3-Nano-4B (NVIDIA, not looped) | 53.5 | 59.4 | 49.9 | 18.0 |
- The loop adds 2–5 points on top of a lead that is already whole at one loop. IFNovel's 22 constraint types exist in no public set, and the margin grows there in 9 of 9 comparisons.
- NVIDIA's own non-looped 4B is 15–37 points behind, so it isn't the shared recipe family. Post-training is the likeliest source; nothing here isolates it.
- GPQA isn't an edge: Qwen3.5-4B leads by 4.7–9.6 at every loop count. On plain text Dory is the weaker language model (bits per byte 0.606 at 4 loops against Qwen's 0.557).
- The card's Qwen IFBench (53.83) is low partly by grading. Qwen writes a blank line after
</think>; graded unstripped it loses about 6 points and Dory none.
Where this meets earlier Thursdays
Elastic depth from both ends: skip blocks, or repeat them.
Thu 10 Sep · Don't Drop Dropout (Cerebras)
Train with whole blocks randomly skipped, and one checkpoint runs shallower at inference. The review found the elasticity real but mostly "not collapsing", and every saving counted in FLOPs, not wall clock or energy. Those slides ↗
Today · Elastic Latent Recurrence (NVIDIA)
Train with the middle block repeated, and one checkpoint runs deeper. This time the one-loop end is protected by design, and the paper reports measured latency, not only FLOPs.
The energy question is where the bytes come from. A loop reuses the same weights, so it adds arithmetic, not parameters. On a GPU each loop still re-streams the core from HBM: Dory's 20-block core is about 1.8B parameters, roughly 3.7 GB in BF16, against tens of megabytes of on-chip cache.
Your GPU comparison (A100 to B200): at batch 256 a 4-loop token takes 2.4–3.1× the time of a 1-loop token, because decoding there is bound by memory traffic and per-loop KV multiplies the cache that has to be read. The B200 was the cheapest card per token, once its image had a CUDA compiler.
For the table
Questions to open with
- If the second loop does nearly all the work, what are loops three and four for? The authors suspect later loops are simply seen less in training (loop 4 is in 22.5% of examples). Would you reweight the loop distribution?
- Dory works with no gate and no conditioning. Is nGPT's per-layer step already doing the gate's job?
- At the same inference cost, would you rather have a 4B model looped twice, or a deeper model run once? The paper sets this aside on purpose.
- Where would a depth dial help outside language models? Vision, control, anything with a latency budget per input.
- Dory's lead is there at one loop. What in post-training could give a 4B model that kind of instruction following?