What Are Looped Transformers? A Technical Deep Dive

A looped transformer applies the same block of layers to a token more than once. Instead of stacking L distinct layers and running each exactly once, you keep a small stack of shared layers and run it in a loop: four real layers applied thirty-two times give you 132 layers of computation out of the parameters of eight. The idea goes back to the Universal Transformer in 2018, and the modern name for it is recurrent depth.

What makes it interesting is that the loop count is not baked into the checkpoint. In an ordinary transformer, depth is a property of the weight file — you ship eighty layers, and every prompt gets eighty layers. In a looped model the number of passes, r, is an argument you pass at call time, so the same weights can be told to think harder about a hard question than an easy one. And the extra thinking happens inside the hidden state, between emitted tokens, rather than in text anyone can read.

Which is why, on 2 September 2026, this became an argument about safety. The Information reported that OpenAI's Astra uses recurrent depth — the same weights applied several times over — and that this makes some of its internal reasoning harder to read. Astra shipped as a preview the next day, went generally available on the fourth, and by then the argument had sorted itself into two camps that were not really talking to each other.

One camp read it as a redline crossed. Ryan Greenblatt: Astra "fundamentally doesn't have a fully monitorable chain of thought in the same way as past models. Very very bad news." Steven Adler, until recently on OpenAI's safety staff, called it a violated redline. The other camp read it as nothing at all. @nrehiew_: a looped transformer has "zero impact, because it is simply spending more compute per token." Sebastian Raschka called it a tiny architectural tweak and noted, correctly, that reusing layers does not by itself suppress a visible chain of thought.

Then Jakub Pachocki posted, which is unusual for a chief scientist mid-news-cycle, and said something better than either camp: the depth of Astra's computation graph is within a factor of two of GPT-4 — and, separately, that chain-of-thought monitoring "is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes."

The architecture claim rests on one anonymous source. No technical report, no code, no patent, nobody outside OpenAI in a position to check it. So I stopped reading the argument and read the architecture, which — unlike Astra — is documented in detail, mostly by people who published long before any of this was a story. Thirteen papers later, what a loop actually buys turns out to be narrower than the alarmed reading requires and stranger than the deflationary one admits.

This post is the mechanical version: what the forward pass does line by line, how the test-time dial is manufactured during training, what the loop costs in FLOPs and KV cache with the arithmetic shown, what the literature establishes, and whether this is the thing that pushes models further. Every mechanism is drawn out as a diagram.

TL;DR

• Section 1 is the whole idea with no notation at all — four animated pictures showing what changes, why it makes answers better, and what it costs. Everything after that is the engineering. Symbols are collected in an appendix at the end.
• Depth is the thing being bought. Constant-depth transformers are confined to $\mathrm{AC}^0$. Chain-of-thought buys serial depth in token space; recurrent depth buys the same thing in latent space. Two implementations of one complexity-theoretic fix.
• Three parts, not one. A prelude P, a recurrent core R applied r times, a coda C. Huginn is (2, 4, 2) — eight real layers that unfold to 132 at r = 32.
• Two details every summary drops: the embedded input is re-injected on every iteration, and the initial state is random, not the embedding. Together they produce path independence, which is the entire licence to change r at inference.
• The dial is made during training, not discovered at inference: r is sampled per step from a heavy-tailed log-normal Poisson with mean 32, and the backward pass is truncated to the last k = 8 iterations, so memory depends on k and not on r.
• Nothing about the loop is free. Storage is saved; FLOPs, wall-clock and KV cache are not. At r = 32 the KV cache is 10.6 GiB for one 4096-token sequence, against 7 GB of weights.
• The exchange rate is measured and sublinear: $\varphi = 0.46$. One recurrence is worth $r^{0.46}$ in equivalent unique parameters.
• The strange result: looping buys reasoning, not knowledge — 282% of the gap recovered on math word problems against 37% on closed-book QA, confirmed independently by three groups.
• The frontier is a router, not a loop count. Mixture-of-Recursions beats a 315M vanilla model with 167M parameters at equal FLOPs by giving each token its own depth.
• Verdict: a genuine third scaling axis, and a component rather than a paradigm. On monitorability, the deflationary camp is right about the mechanism and the alarmed camp is right about the economics.

1. What is actually different, in four pictures

No notation in this section. Just what changes, why it helps, and what it costs. Every symbol in the post is collected in an appendix at the end.

The difference: the same layers, used again

A model is a stack of layers. Your question goes in the bottom, passes up through each layer in turn, and an answer comes out the top. Each layer is a separate pile of numbers — the "weights" — sitting in a file on disk.

An ordinary model has, say, eight layers and uses each one exactly once. A looped model has four layers and runs through those same four repeatedly. Watch which one finishes first.

That is the trick. Storage and thinking used to be the same number. Looping separates them into two dials.

Two honest notes before going on. The looped stack takes longer, not less time — you can see it still working after the other has stopped, and that is the correct picture: four passes over four layers is sixteen layers of work, and you wait for all sixteen. And this is not a free win. Four layers run four times are not as good as sixteen real layers — shared layers are worth measurably less than purpose-built ones, and section 6 puts a number on exactly how much less. What you get is most of the benefit from a quarter of the file.

(Real looped models keep a layer or two outside the loop at each end. Section 3 has the exact shape.)

What actually loops

The obvious objection: if you run the same four layers on the same question again, why doesn't it just produce the same answer again?

Because the second pass does not start over. It starts from where the first pass got to. The model carries a scratchpad of partial work — that chip in the diagram above — and each pass reads the previous pass's working, plus your original question, and improves on it. It is closer to re-reading a difficult paragraph than to asking the same question twice.

Why this is a cheaper way to scale

Until now there were two ways to make a model smarter, and both are expensive.

Make it bigger. More layers, more weights, a bigger file. That costs memory on every machine that ever runs it, forever.

Make it write out its reasoning. This is what "thinking" models do — they work through the problem in words before answering. That is not a trick; it is the model buying itself more steps, out loud. But the words fill up the space available for your actual document, and you pay for every one of them.

Looping buys the same extra steps silently. It adds no weights and writes nothing down.

Watch the flat green line: the file never gets bigger. That is the claim in one image — you buy more thinking without buying a bigger model. And that is worth real money: a small file that reasons like a big one fits on cheaper hardware, or fits more copies on the same machine. You convert a cost you pay always into a cost you pay only on the questions that need it.

Three things that image does not show, and you should not leave without them.

It is not free memory. The weights stay the same size, but while the model is answering it keeps a working scratchpad of everything each pass computed, and that scratchpad grows with every pass. At the highest settings it is larger than the model itself. Section 6 has the arithmetic.

The dial has to be built in. You cannot take an ordinary model and ask it to run its layers twice. It has to be trained from the start to accept any number of passes, and section 5 is about how that is done.

The returns flatten sooner than you would like. Watch the amber line go flat. For the best-documented model, more looping stops helping somewhere between eight and thirty-two passes depending on the question — and that is a lightly trained model. The one team that trained a looped model really hard found the useful range collapsed to two passes. The better the model is to begin with, the fewer extra passes it has any use for. That is the strongest argument against the whole idea and section 9 takes it seriously.

The catch: it thinks better, it does not know more

This is the part that surprised researchers, and three separate teams have now found it independently.

Extra passes help enormously with problems you have to work through — arithmetic, multi-step logic, following a chain of reasoning. They help far less with problems that just require knowing a fact. Some, but nothing like as much.

Extra passes let a model think harder about what it already knows. They do not teach it anything new. For that you still need a bigger model.

So the practical summary, before any machinery: a looped model gets you much of the reasoning of a far larger model out of a far smaller file, by spending time rather than by getting bigger — and it is worth doing exactly when your problem needs working out rather than looking up.

A looped model is not a smaller model. It is a model that thinks longer instead of being bigger.

Everything below is how that is built and what it costs. If you only wanted the idea, you have it.

2. A fixed-depth transformer has a shape of thought it cannot have

Start with why anyone wants depth, because that is what the whole architecture is answering.

A transformer with a fixed number of layers is a parallel object. Every token's computation runs through the same bounded chain of operations, and that chain does not get longer when the problem gets harder. Li, Liu, Zhou and Ma made this precise at ICLR 2024: constant-depth, constant-precision transformers are confined to $\mathrm{AC}^0$, a proper subset of $\mathrm{TC}^0$. Both names denote what a circuit of fixed depth can compute: $\mathrm{AC}^0$ allows unlimited AND and OR gates, $\mathrm{TC}^0$ adds majority gates and is strictly larger. That is a small class. It does not contain the kinds of problems that are inherently serial — composing a permutation group step by step, iterated squaring, evaluating a circuit whose gates depend on earlier gates.

The same paper gives the escape hatch, and it is why chain-of-thought works at all. Let the model emit T intermediate tokens and read them back, and with O(log n) embedding width it can solve anything computable by a boolean circuit of size T. Chain-of-thought is not a prompting trick that happens to help. It is a depth mechanism — the context window used as a tape, every emitted token one more step of serial computation. Merrill and Sabharwal supply the other half: transformers of depth Θ(log n) express regular-language recognition (state tracking) and graph connectivity (multi-step reasoning), neither of which fixed-depth transformers can express under standard complexity conjectures.

So there are exactly two ways to buy serial depth. Spend tokens, and the computation happens in the context window where you can read it. Spend iterations, and it happens in the hidden state where you cannot.

Recurrent depth takes the second route, and everything downstream — the parameter savings, the KV cache problem, the monitorability argument — falls out of that one choice.

Chain-of-thought buys serial depth in token space. Recurrent depth buys it in latent space. They are two implementations of the same complexity-theoretic fix.

One caveat to carry through the rest of this. The complexity results above are asymptotic — they need depth that grows with the input. A looped model run at a fixed loop count r is still a constant-depth transformer, exactly as expressive as the unrolled l_P + r·l_R + l_C-layer network it is equivalent to. Looping does not change the complexity class unless you let r scale with problem size. What it changes is the price of depth, and — at inference — who gets to choose it.

3. The forward pass, precisely

Almost every explainer stops at "it runs the same layers twice." That is true and it is not enough to implement anything.

The most detailed public architecture is Huginn — a 3.5B recurrent-depth model trained on 795B tokens on Oak Ridge's Frontier — and it has three parts, not one. A prelude P embeds tokens into latent space. A recurrent core R is applied r times, reusing one weight set. A coda C decodes the final latent state into logits. The paper writes the shape as a triplet (l_P, l_R, l_C); Huginn is (2, 4, 2). Eight real layers.

# Standard decoder-only transformer — L weight sets, one position
x = Embed(t)
for l in range(1, L + 1):
    x = Layer_l(x)          # each layer used exactly once
p = Unembed(Norm(x))
# depth L is a property of the checkpoint. Nothing picks it at call time.

# Recurrent depth — 3 weight sets, one position
e = P(t)                    # prelude, l_P layers
s = trunc_normal(0, 2/5)    # s is RANDOM, not e
for i in range(1, r + 1):   # r is an ARGUMENT
    s = R(e, s)             # core, l_R layers, ONE weight set
p = C(s)                    # coda, l_C layers
# depth = l_P + r*l_R + l_C

Two details in that second block are where the architecture actually lives, and both are routinely dropped in summaries.

The embedded input is re-injected at every iteration. The core does not take s alone; it takes R(e, s_{i−1}), and internally an adapter matrix A: ℝ^2h → ℝ^h projects the concatenation [s_{i−1} ; e] back down to the hidden dimension before the four transformer layers run. Why that is load-bearing rather than incidental is checked numerically below.

The initial state is random. s_0 is drawn from a truncated normal with variance σ_s² = 2/5, cut at 3σ — fresh, per forward pass. Not from the embedded input, which is the obvious choice and the wrong one. Random initialisation together with input injection is what produces path independence (Anil et al., 2022): different initialisations converge to the same trajectory. That property is not cosmetic. It is the thing that makes it legitimate to change r at inference at all — without it, a loop count the model was never trained at is just an out-of-distribution query.

The bus on the left is the detail most summaries lose: the embedded input e enters the core on every pass, not once at the bottom. The prelude and coda are ordinary transformer layers with their own weights, used exactly once — only the core is shared, and only the core's cost scales with r.

Inside the core block:

# adapter: concatenate the state with the input, project back down to h
A : R^(2h) -> R^h
R(e, s) = Block^(l_R) ( A([ s ; e ]) )

# each layer uses "sandwich" RMSNorm (n1..n4) — load-bearing at scale
x_hat = n2( x     + Attn( n1(x)     ) )
x     = n4( x_hat + MLP ( n3(x_hat) ) )

Attention is standard causal self-attention with RoPE (base 50000) and a gated SiLU MLP; RMSNorm ε = 1e−6; learnable biases on queries and keys and nowhere else. Ordinary, in other words.

Following one 10-token prompt all the way through makes the shapes concrete. Nothing here is batched, h = 5280, and r = 32:

The only tensor that grows is the concatenation, and the adapter immediately undoes it. Everything else in the loop is shape-preserving, which is the structural reason the block can be reapplied at all.

The third detail is a negative result worth more than most positive ones: the core block takes no step index. The authors tried the obvious variant — a step-conditioned core s_i = R_i(e, s_{i−1}) with a diffusion-style timestep embedding, so that iteration seven knows it is iteration seven — and report it "interacts badly with path independence, leading to models that cannot extrapolate." The block has to be blind to which iteration it is on. Whatever it has learned is therefore not a 32-step program; it is one step of something applicable an unspecified number of times.

Checking the two details numerically

Both claims above — that re-injecting e matters, and that random s_0 is safe — are testable in about twenty lines, with no model. Take a random contraction in place of the four learned layers and iterate it:

import numpy as np
np.random.seed(0)
h = 8
A = np.random.randn(h, 2 * h) / np.sqrt(2 * h)   # adapter, 2h -> h
W = np.random.randn(h, h) / np.sqrt(h)           # stands in for the core's 4 layers
e = np.random.randn(h)                            # the embedded input, held fixed

def core(s, e):                                   # R(e, s), with input injection
    return np.tanh(W @ (A @ np.concatenate([s, e])))

states = [np.random.randn(h) * 0.632 for _ in range(4)]   # four different s_0
for i in range(1, 25):
    states = [core(s, e) for s in states]
    if i in (1, 4, 8, 16, 24):
        spread = max(np.linalg.norm(a - b) for a in states for b in states)
        print(f"after {i:2d} iters, max distance between trajectories = {spread:.2e}")

after  1 iters, max distance between trajectories = 1.05e+00
after  4 iters, max distance between trajectories = 8.90e-02
after  8 iters, max distance between trajectories = 1.85e-03
after 16 iters, max distance between trajectories = 1.18e-06
after 24 iters, max distance between trajectories = 8.23e-10

Four different random starting states, four trajectories, one destination. That is path independence, and it is why s_0 can be noise: after enough iterations the state depends on the input and the weights, not on where it started. It is also why changing r at inference is legal rather than an out-of-distribution query — the model is not executing a 32-step program that breaks when you ask for 40.

The second half of the experiment is the more interesting one. Run the same core on two different inputs e and e2, and the two fixed points sit a distance of 1.116 apart — the input determines where the iteration lands, which is the entire point of running it. Now delete the injection, so that e enters only through s_0, and repeat:

def core_noinj(s):                                # e enters only via s_0
    return np.tanh(W @ (A @ np.concatenate([s, np.zeros(h)])))

distance between the fixed point reached from e and from e2 = 1.80e-04

Four orders of magnitude smaller. The same property that made the loop safe — forgetting where it started — has now made it useless, because the input is where it started. The deep state converges to a place determined by the weights alone, and the prompt has been washed out.

The paper's own justification is an analogy to gradient descent — you cannot optimise a data-dependent objective by looking at the data once — and the experiment above is that sentence made checkable. Formally, if e entered only through s_0, the core could not be a monotone operator on data-dependent functions: its fixed point would be a property of the weights alone rather than of the input.

4. What actually changes, line by line

"It loops" is not a difference an engineer can plan around. These are.

A looped model is not a smaller model. It is a model whose depth has been moved out of the weight file and into the call signature.

5. You do not train it at a depth. You train it over a distribution of depths.

This is the least-covered part of the architecture and the most interesting, because it is where the test-time dial is actually manufactured.

If you trained at a fixed r, you would get a model that works at that r and degrades away from it — which is just a deep model with tied weights. Instead the loss is an expectation over sampled depths:

Every optimiser step draws a new r from a log-normal Poisson distribution Λ with mean 32 and σ = 1/2:

The distribution is deliberately heavy-tailed: most steps run shallower than 32, and occasionally one runs far deeper. Which raises the obvious objection — if some steps unroll to 60 or 80 iterations, the activation memory should be catastrophic. It is not, because the backward pass is truncated: gradients flow through the last k = 8 iterations only. Memory and backward compute are therefore functions of k, not of r, and the heavy tail becomes affordable. Textbook truncated backpropagation through time, applied to depth instead of to time.

Depth extrapolation is manufactured here, not discovered at inference. Because the model has never been told which iteration it is on, and because it has seen many different r values with gradient only ever arriving through the last eight, the core learns an operator that is safe to apply an unspecified number of times.

Three consequences follow, and the third is the interesting one.

Stability is not automatic. The paper is unusually honest about this: two large runs failed before the third worked. A version with parameter-free RMSNorm, a parameter-free additive adapter A(s, e) = s + e and a peak learning rate of 4e−4 stalled with hidden-state collapse; a second collapsed the recurrence itself, with token correlation driving toward 1.0 — the model learning to ignore its own loop. The fix was the sandwich normalisation, a learned adapter, and dropping the peak learning rate by a factor of ten to 4e−5. At small scale every normalisation scheme worked. At scale, only one did.

Backprop is shorter than the forward pass, and that is fine. A model can be trained through 8 iterations and run at 32 or 64. The gradient never saw the deep regime. That should be alarming and mostly is not, because of the step-blindness above: what the core is being taught is not a 32-step program but a single step of a convergent operator.

Depth extrapolation at test time is the surprising part. The model was trained with a single r per sequence, and yet at inference different tokens converge at different rates — the authors show latent states settling quickly on filler and slowly on the semantically loaded words of a question. Nobody trained that. Nor is the convergence uniform in kind: PCA of the latent trajectories shows fixed points on most tokens, multi-dimensional orbits on arithmetic, and steady directional drift — "sliders" — that would let the model count how many iterations it has taken.

How far the extrapolation actually goes, stated honestly. The paper evaluates 1 to 64 iterations and plots convergence out to 128. It does not demonstrate unbounded depth. Saturation is task- and context-dependent: roughly 8–12 iterations zero-shot, about 20 with one in-context example, about 32 with 25–50 — the model uses more depth when there is more context to reason over. "Improves with more compute" is supported; "improves without limit" is not claimed.

6. Looping trades storage for time. It does not create compute.

The single most common error in the coverage of this architecture was treating the loop as free because it adds no weights. Here is the arithmetic for Huginn at each loop count, from its published parameter split.

Assumptions, so you can check this. Forward FLOPs taken as 2·(N_once + r·N_rec) with the paper's split N_once = 1.5B and N_rec = 1.5B. The factor of 2 is the standard accounting rule: every weight participates in one multiply and one add per token, so a forward pass over N parameters costs about 2N floating-point operations — and here the core's parameters are traversed r times rather than once. KV cache taken as (l_P + r·l_R + l_C) × 2 × h × 2 bytes in bf16 at h = 5280 with full multi-head attention — Huginn has no grouped-query attention. The "cache budget 4" column applies the paper's own zero-shot sharing scheme, which writes to cache slot i mod k. The r = 32 row lands on 49.5B, which is the derivation behind the paper's own statement that the model reaches "FLOP budgets equivalent to a standard 50B parameter fixed-depth transformer." Those parameter figures are compute equivalences, not claims of parity with a 50B model — and the two get conflated constantly.

Three things in that table deserve to be said out loud.

The KV cache is the sleeper problem. Every iteration runs real attention and writes its own keys and values, so the cache scales with effective depth, not with real layers. At r = 32 and a 4096-token context that is 10.6 GiB of cache for a single sequence — for a model whose weights are 7 GB in bf16. That is why the paper has a section on cache sharing at all, and the mitigation is not free either: Nanbeige tested sharing the KV cache across loop passes, found it "reduces the KV-cache by half" and that "its performance gains are consistently lower than those of the full, non-sharing loop configuration," and shipped the expensive version anyway.

In practice, what r should you set? The paper's own saturation points make a usable rule, and they are not constant — the model spends more depth productively when there is more context to reason over:

Nobody trained those thresholds. They are where a model trained across a distribution of depths happens to stop changing its mind.

Storage is saved; bandwidth is not. At batch-1 decode you still stream the core weights r times through the memory system.

elie bakouch, who builds these, put it in a line: "Why recurrent depth instead of just scaling depth? It's not faster at inference or training… since you still go through the full 'effective depth'. The advantage is storage."

The exchange rate is sublinear, and it has been measured. Schwethelm, Rueckert and Kaissis ran an iso-depth pretraining sweep over r ∈ {1, 2, 4, 8} across roughly 50× in training compute and fitted a joint scaling law in which recurrence enters as a power of r multiplying the recurrent parameter count:

One recurrence is worth $r^{0.46}$ in equivalent unique parameters — genuinely positive, and considerably less than one. That exponent is the honest centre of the subject. It is the reason a looped 3B behaves like something bigger, and the reason it does not behave like something much bigger.

Two practitioner quotes worth keeping alongside it. Sebastian Raschka, on a two-pass looped model: "almost 2× as expensive in terms of compute, because we run the embedded text through almost 2× as many layers." And the Nanbeige team, who actually shipped one, on what a second pass buys: "relative to a standard Transformer, it retains approximately 75% of the token efficiency and provides a significant capacity gain." A looped layer is not a real layer, and somebody measured by how much.

Set against the three ways a model can be given more compute, the trade becomes legible:

7. The interesting version is not a fixed loop count

A fixed r is a blunt instrument: the token the gets the same 32 passes as the operative word in a hard question. Every serious line of work in this area is converging on the same next move — let the model decide the depth per token — and there are now three distinct ways to do it.

Emergent, zero-shot. Huginn exits when the KL divergence between two successive latent states falls below 5e−4. Nothing was trained for this; the criterion is applied after the fact to a model that happened to converge. It works: on MTBench — a standard multi-turn chat benchmark graded out of 10 — the model scores 5.63 running every iteration and 5.56 exiting early, for a large saving in compute. And the exit distribution is semantically sensible — the model leaves early on high-school mathematics and takes about 3.5 steps longer on moral scenarios.

Learned halting. The Universal Transformer had this in 2018, with per-position Adaptive Computation Time. Ouro, ByteDance's 2025 looped family, trains an entropy-regularised objective for learned depth allocation directly in pretraining.

Learned routing. Mixture-of-Recursions goes further: a lightweight router assigns each token its own recursion depth, and the KV cache is restructured to match. Two routing schemes, with a genuine trade-off. Expert-choice has each depth select its own top-k tokens, which gives perfect load balance by construction but leaks information across the causal boundary during training — fixed with an auxiliary loss that teaches the model to detect the top-k threshold at inference. Token-choice commits each token to a depth up front, which is causally clean but needs a balancing loss to stop every token stampeding to the same depth. Of four weight-sharing layouts, Middle-Cycle — unique first and last layers, shared middle — consistently wins.

Dashed cells are passes the router declined to spend. The consequence is not only cheaper compute: under recursion-wise caching, only the tokens routed to depth d write KV entries at depth d, which brings cache memory down to (N_r + 1)/2N_r of a vanilla model's. The alternative — recursive sharing, cache once at step 1 and reuse everywhere — reaches 1/N_r but pays it back in attention FLOPs.

The numbers are the strongest efficiency result in the field. At an equal training budget of 16.5e18 FLOPs, a 167M-parameter MoR model reaches 43.1% few-shot average against a 315M vanilla transformer's 42.3% — better, with 47% fewer parameters. It trains 19% faster, peaks 25% lower on memory, and reaches up to 2.06× inference throughput through "continuous depth-wise batching," which works precisely because tokens at different recursion depths can be packed into one batch. That last figure comes from the depth-4 configuration, which the authors note degrades slightly in quality — a throughput-for-accuracy trade, not a free lunch. (It also circulates as 2.18× rather more often than it should.)

Be precise about what changes here, because it is a different thing from looping itself. With a fixed r, compute per token is a deployment constant: you set it, you can log it, you can bill for it. With a learned router, how much thinking each token receives becomes a hidden, input-dependent decision inside the model. For capability that is the entire point. For anyone trying to observe the system from outside, the change is not that the reasoning became unreadable — it is that the amount of reasoning became an unlogged variable.

8. What the literature actually establishes

Strip away the Astra story and there is a real, unusually clean research result underneath, now confirmed by three groups using different methods at different scales: looping buys reasoning, not knowledge.

The headline numbers below use a metric — "% of the gap recovered" — that is worth unpacking, because 282% sounds like a typo and is not.

Loops buy thinking, not facts. If you want a model that knows more, you still need parameters. If you want one that thinks harder, you can now turn a dial — and pay for it in time rather than in memory.

I find this genuinely odd, and I do not think the field has absorbed it. Parameter scaling has always moved knowledge and reasoning together, so we never had cause to treat them as separable knobs. Recurrent depth separates them cleanly enough that three independent methodologies agree — and that result outlives the news cycle that made anyone look.

9. Is this the architecture that pushes models further?

Partly, and not in the shape the coverage implied. The evidence supports a narrow claim and does not support a broad one.

The narrow claim: recurrent depth is a genuine third scaling axis with a measured exchange rate of $r^{0.46}$, and it moves reasoning without moving knowledge. That is a scaling law, not a slogan, and the knowledge-versus-reasoning split has been reproduced independently — by the Virtual Logic Depth and Ouro teams, on different architectures, at different scales, with different objectives. Anyone shipping a small model should be looking at it today.

The broad claim — that this is the architectural change that takes frontier models to the next level — is not supported, and one specific piece of evidence cuts against it. Nanbeige trained at 28T tokens, an order of magnitude beyond any other published looped model, and found that going past two passes bought almost nothing while destabilising optimisation: "increasing the number of passes provides only marginal additional improvement, but substantially slows training and makes optimization less stable." At 28T tokens the sweet spot was r = 2. Huginn's own returns saturate between 8 and 32 iterations depending on how much context there is to chew on. The exchange rate is sublinear by construction. Three attempts were needed to train one 3.5B model at all. None of that is what a discontinuity looks like from the inside.

Where it clearly helps

• Depth-bound, serial reasoning. Arithmetic, state tracking, graph connectivity, algorithmic search. The 282%-vs-37% split between math word problems and closed-book QA is the shape of the whole effect.
• Parameter-constrained deployment. A 3B that behaves like a 12B is worth more on a laptop, a phone or a memory-bound serving tier than anywhere else. This is where the shipped models actually are.
• Test-time compute without context growth. No CoT data, no RL to elicit it, no context consumed, and it composes with a 4K window. LOTUS cuts thought-phase latency 2.5–6.9× against writing the reasoning out.
• Interconnect-poor training clusters. High FLOPs per parameter means data parallelism suffices. Huginn ran 4096 GPUs with no model parallelism at all.

Where it does not help

• Anything bound by memorised facts. Two independent results say knowledge capacity is flat in r and still tracks parameters. No amount of looping makes a model know more.
• Wall-clock latency. The iterations are a serial dependency chain. You pay the full effective depth in time. Storage is the saving; speed is not.
• Long-context serving. KV cache grows linearly in r. The sharing fix works and measurably costs quality — Nanbeige declined it.
• Large loop counts. Sublinear returns, saturating benchmarks, and optimisation that gets less stable the deeper you go.

What is genuinely unsolved

• Stability at large effective depth. Huginn needed three attempts, a sandwich-norm block and a 10× learning-rate cut. Nobody has published a recipe known to hold at frontier scale.
• Composition with sparsity. Every frontier model is a mixture-of-experts; every looped result is dense. MoEUT shows the combination can work below 1B and is arguably necessary, since shared layers have a poor parameter-to-compute ratio. Above that, nothing public.
• Whether depth extrapolation survives scale. It is demonstrated at 3.5B on 795B tokens. The one 28T-token data point suggests the useful range of r shrinks as training gets long.
• Post-training. Outcome-level RL still works, but the process supervision that drives current reasoning models has no purchase on a step you cannot read. LOTUS's answer — supervise latent positions against gold CoT tokens — works, and gives back the data dependency that was the point.

The likely future is not looped models replacing transformers. It is a small loop count, a learned router and a sparse MoE core, all in the same model — with recurrent depth as a component rather than a paradigm.

Which is roughly what Mixture-of-Recursions already is, at 1.7B. What is missing is everything above that scale.

10. Back to the argument, which was two arguments

Which brings us back to where this started. Both camps argued well and mostly past each other, because the question they had merged is really two questions with two different answers.

On question one, the deflationary camp is right, and everything above is one long demonstration of why. Nothing in the mechanism supports the alarming reading: extra iterations add computation between emitted tokens exactly as extra layers always have, and the model still writes its reasoning out. Pachocki's "within a factor of two of GPT-4" is a claim about effective depth, which for a looped model means a small r — the regime where, at $\varphi = 0.46$, this is a modest efficiency choice rather than a new kind of system.

On question two, the alarmed camp is pointing at something real, and it does not depend on the Astra rumour being true at all. Nobody has to decide to hide anything. The most striking thing in the whole episode is that OpenAI's chief scientist agrees the transcript is degrading; he just says the architecture is not why.

The architecture is not the threat. The gradient is. Both sides concede the same trend — they disagree only about its cause.

The consequence for anyone building on these models is unchanged by the technical detail, and possibly sharpened by it. If legible reasoning is something you depend on — for audit, for evaluation, for debugging an agent that touches production — stop treating it as a property of the architecture you happened to buy and start treating it as something you specify, price and test for. And note which direction the frontier is heading: per-token routed depth, where the amount of computation spent on a token is itself a learned and unlogged decision. That is a monitorability question with real content, and it is not the one anyone spent September arguing about.

11. What would actually settle this

1. Does depth extrapolation survive at frontier scale? The only public demonstration is 3.5B on 795B tokens. The only 28T-token data point says the useful loop range shrinks. This is the load-bearing empirical question for the whole architecture and it is open.
2. Does it compose with sparsity? MoEUT works below 1B. Every frontier model is a mixture-of-experts. Nobody has published a large looped-plus-sparse model, and the parameter-to-compute argument says that combination is where the real design lives.
3. What does the KV cache actually cost in production? Naïve caching scales with effective depth; both published mitigations trade quality or FLOPs. A serving-side study at long context would settle whether this architecture is deployable where it is most wanted.
4. Can latent steps be supervised without gold CoT? LOTUS closes the latent-versus-explicit gap by supervising each latent position against a written reasoning step. Doing it without that crutch is the difference between an efficiency trick and a new training paradigm.
5. Astra's loop count, if any. "Within a factor of two of GPT-4" implies a small r. Publishing the number would end that argument in a sentence — and, per everything above, probably in the deflationary direction.
6. Did reasoning tokens per task go down? The measurable version of the safety worry, testable from outside. If Astra writes visibly less reasoning than the o-series for equivalent tasks, the economics argument has teeth.

Not one of those turns on what Astra does. The rumour was never the interesting part. It was the thing that got everybody to look.

Appendix: notation and jargon, for reference

Nothing in this appendix is needed to follow the argument — it is here so that when a symbol shows up in the sections above, there is one place to look it up.

The architecture symbols

Two collisions worth flagging, because both appear in the source papers and neither is avoidable without misquoting them. A is the adapter matrix here, but also a fitted constant in the scaling law of section 6 — different objects, same letter. And the scaling law's loss is written $\mathcal{L}$ throughout this post, curly, to keep it distinct from L the layer count.

The vocabulary

The whole architecture as a config

Every number above comes from one file. This is Huginn's published config.json, trimmed to the fields that matter and annotated:

{
  "n_layers_in_prelude": 2,        // l_P — runs once
  "n_layers_in_recurrent_block": 4,// l_R — this is the part that repeats
  "n_layers_in_coda": 2,           // l_C — runs once
  "n_layers": 8,                   // real layers on disk: 2 + 4 + 2
  "effective_expected_depth": 132, // what r = 32 unfolds to: 2 + 4*32 + 2

  "mean_recurrence": 32,           // r-bar — the mean of the training distribution
  "mean_backprop_depth": 8,        // k — gradient truncation window
  "sampling_scheme": "poisson-lognormal-filling",  // how r is drawn each step
  "state_init": "like-init",       // s_0 is random, not the embedding
  "injection_type": "linear",      // the adapter A: concat(s, e) -> h

  "n_embd": 5280,                  // h — hidden size
  "n_heads": 55,                   // 55 x 96 = 5280
  "num_key_value_heads": 55,       // equal to n_heads: NO grouped-query attention
  "intermediate_size": 17920,      // MLP inner width
  "block_size": 4096,              // context window
  "vocab_size": 65536,

  "block_class_name": "SandwichBlock",  // norm before AND after each sub-layer
  "norm_class_name": "RMSNorm_llama",
  "norm_eps": 1e-06,
  "nonlin_name": "SiLU",
  "rope_base": 50000,
  "qk_bias": true,                 // biases on queries and keys only
  "bias": false,

  "init_values": { "std": 0.008703882797784892, "embed_scale": 72.6636084983398 }
}

That last line is worth multiplying out, because it is where the paper's σ_s² = 2/5 actually comes from: 0.008703882797784892 × 72.6636084983398 = 0.6324555, and 0.6324555² = 0.4 = 2/5 exactly. The initial state is drawn from a normal with standard deviation 0.632, truncated at three standard deviations.

Sources

Architecture

• Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Geiping, McLeish, Jain, Kirchenbauer, Singh, Bartoldson, Kailkhura, Bhatele, Goldstein. Huginn-0125; the only fully specified public recurrent-depth model, and the source of every architectural number above.

Theory

• Reasoning with Latent Thoughts: On the Power of Looped Transformers — Saunshi, Dikkala, Li, Kumar, Reddi (ICLR 2025)
• Chain of Thought Empowers Transformers to Solve Inherently Serial Problems — Li, Liu, Zhou, Ma (ICLR 2024). The $\mathrm{AC}^0$ / circuit-size result.
• A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers — Merrill, Sabharwal

Scaling

• Beyond Parameters: Exploring Virtual Logic Depth for Scaling Laws — Zhu, Zhang, Li, Shi, Duan, Wang, Zhou, Banerjee, Qin
• How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models — Schwethelm, Rueckert, Kaissis. The φ = 0.46 exponent.

Adaptive depth

• Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation — Bae et al. (NeurIPS 2025)
• Universal Transformers — Dehghani, Gouws, Vinyals, Uszkoreit, Kaiser. Shared-weight recurrence with per-position adaptive computation time.

Sparsity, counterweights and shipped models

• MoEUT: Mixture-of-Experts Universal Transformers — Csordás, Irie, Schmidhuber, Potts, Manning
• Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers — Fan, Svete, Lee (LOTUS)
• Scaling Latent Reasoning via Looped Language Models (Ouro, ByteDance) · Nanbeige4.2-3B
• Open weights: Huginn-0125, Nanbeige4.2-3B, Ouro-1.4B
• Reasoning Beyond Language: A Survey on Latent Chain-of-Thought Reasoning

The argument

• OpenAI Astra and Looped Transformers — Sebastian Raschka
• Fortune, on The Information's report and the safety response
• Jakub Pachocki, chief scientist, OpenAI — 2 September 2026
• @nrehiew_ on looped transformers and CoT monitorability · elie bakouch on why the advantage is storage

Architectural figures are quoted from the papers named beside them. Where a number is derived rather than quoted — the FLOP and KV-cache table in section 6 — the assumptions are stated in full so the arithmetic can be checked. The Astra architecture claim remains single-sourced and uncorroborated.