What Are Looped Transformers? A Technical Deep Dive

A looped transformer applies the same block of layers to a token more than once. Instead of stacking L distinct layers and running each exactly once, you keep a small stack of shared layers and run it in a loop: four real layers applied thirty-two times give you 132 layers of computation out of the parameters of eight. The idea goes back to the Universal Transformer in 2018, and the modern name for it is recurrent depth.

What makes it interesting is that the loop count is not baked into the checkpoint. In an ordinary transformer, depth is a property of the weight file — you ship eighty layers, and every prompt gets eighty layers. In a looped model the number of passes, r, is an argument you pass at call time, so the same weights can be told to think harder about a hard question than an easy one. And the extra thinking happens inside the hidden state, between emitted tokens, rather than in text anyone can read.

Which is why, on 2 September 2026, this became an argument about safety. The Information reported that OpenAI's Astra uses recurrent depth — the same weights applied several times over — and that this makes some of its internal reasoning harder to read. Astra shipped as a preview the next day, went generally available on the fourth, and by then the argument had sorted itself into two camps that were not really talking to each other.

One camp read it as a redline crossed. Ryan Greenblatt: Astra "fundamentally doesn't have a fully monitorable chain of thought in the same way as past models. Very very bad news." Steven Adler, until recently on OpenAI's safety staff, called it a violated redline. The other camp read it as nothing at all. @nrehiew_: a looped transformer has "zero impact, because it is simply spending more compute per token." Sebastian Raschka called it a tiny architectural tweak and noted, correctly, that reusing layers does not by itself suppress a visible chain of thought.

Then Jakub Pachocki posted, which is unusual for a chief scientist mid-news-cycle, and said something better than either camp: the depth of Astra's computation graph is within a factor of two of GPT-4 — and, separately, that chain-of-thought monitoring "is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes."

The architecture claim rests on one anonymous source. No technical report, no code, no patent, nobody outside OpenAI in a position to check it. So I stopped reading the argument and read the architecture, which — unlike Astra — is documented in detail, mostly by people who published long before any of this was a story. Thirteen papers later, what a loop actually buys turns out to be narrower than the alarmed reading requires and stranger than the deflationary one admits.

This post is the mechanical version: what the forward pass does line by line, how the test-time dial is manufactured during training, what the loop costs in FLOPs and KV cache with the arithmetic shown, what the literature establishes, and whether this is the thing that pushes models further. Every mechanism is drawn out as a diagram.

TL;DR

• Depth is the thing being bought. Constant-depth transformers are confined to $\mathrm{AC}^0$. Chain-of-thought buys serial depth in token space; recurrent depth buys the same thing in latent space. Two implementations of one complexity-theoretic fix.
• Three parts, not one. A prelude P, a recurrent core R applied r times, a coda C. Huginn is (2, 4, 2) — eight real layers that unfold to 132 at r = 32.
• Two details every summary drops: the embedded input is re-injected on every iteration, and the initial state is random, not the embedding. Together they produce path independence, which is the entire licence to change r at inference.
• The dial is made during training, not discovered at inference: r is sampled per step from a heavy-tailed log-normal Poisson with mean 32, and the backward pass is truncated to the last k = 8 iterations, so memory depends on k and not on r.
• Nothing about the loop is free. Storage is saved; FLOPs, wall-clock and KV cache are not. At r = 32 the KV cache is 10.6 GiB for one 4096-token sequence, against 7 GB of weights.
• The exchange rate is measured and sublinear: $\varphi = 0.46$. One recurrence is worth $r^{0.46}$ in equivalent unique parameters.
• The strange result: looping buys reasoning, not knowledge — 282% of the gap recovered on math word problems against 37% on closed-book QA, confirmed independently by three groups.
• The frontier is a router, not a loop count. Mixture-of-Recursions beats a 315M vanilla model with 167M parameters at equal FLOPs by giving each token its own depth.
• Verdict: a genuine third scaling axis, and a component rather than a paradigm. On monitorability, the deflationary camp is right about the mechanism and the alarmed camp is right about the economics.

1. A fixed-depth transformer has a shape of thought it cannot have

Start with why anyone wants depth, because that is what the whole architecture is answering.

A transformer with a fixed number of layers is a parallel object. Every token's computation runs through the same bounded chain of operations, and that chain does not get longer when the problem gets harder. Li, Liu, Zhou and Ma made this precise at ICLR 2024: constant-depth, constant-precision transformers are confined to $\mathrm{AC}^0$, a proper subset of $\mathrm{TC}^0$. That is a small class. It does not contain the kinds of problems that are inherently serial — composing a permutation group step by step, iterated squaring, evaluating a circuit whose gates depend on earlier gates.

The same paper gives the escape hatch, and it is why chain-of-thought works at all. Let the model emit T intermediate tokens and read them back, and with O(log n) embedding width it can solve anything computable by a boolean circuit of size T. Chain-of-thought is not a prompting trick that happens to help. It is a depth mechanism — the context window used as a tape, every emitted token one more step of serial computation. Merrill and Sabharwal supply the other half: transformers of depth Θ(log n) express regular-language recognition (state tracking) and graph connectivity (multi-step reasoning), neither of which fixed-depth transformers can express under standard complexity conjectures.

So there are exactly two ways to buy serial depth. Spend tokens, and the computation happens in the context window where you can read it. Spend iterations, and it happens in the hidden state where you cannot.

Recurrent depth takes the second route, and everything downstream — the parameter savings, the KV cache problem, the monitorability argument — falls out of that one choice.

Chain-of-thought buys serial depth in token space. Recurrent depth buys it in latent space. They are two implementations of the same complexity-theoretic fix.

One caveat to carry through the rest of this. The complexity results above are asymptotic — they need depth that grows with the input. A looped model run at a fixed loop count r is still a constant-depth transformer, exactly as expressive as the unrolled l_P + r·l_R + l_C-layer network it is equivalent to. Looping does not change the complexity class unless you let r scale with problem size. What it changes is the price of depth, and — at inference — who gets to choose it.

2. The forward pass, precisely

Almost every explainer stops at "it runs the same layers twice." That is true and it is not enough to implement anything.

The most detailed public architecture is Huginn — a 3.5B recurrent-depth model trained on 795B tokens on Oak Ridge's Frontier — and it has three parts, not one. A prelude P embeds tokens into latent space. A recurrent core R is applied r times, reusing one weight set. A coda C decodes the final latent state into logits. The paper writes the shape as a triplet (l_P, l_R, l_C); Huginn is (2, 4, 2). Eight real layers.

# Standard decoder-only transformer — L weight sets, one position
x = Embed(t)
for l in range(1, L + 1):
    x = Layer_l(x)          # each layer used exactly once
p = Unembed(Norm(x))
# depth L is a property of the checkpoint. Nothing picks it at call time.

# Recurrent depth — 3 weight sets, one position
e = P(t)                    # prelude, l_P layers
s = trunc_normal(0, 2/5)    # s is RANDOM, not e
for i in range(1, r + 1):   # r is an ARGUMENT
    s = R(e, s)             # core, l_R layers, ONE weight set
p = C(s)                    # coda, l_C layers
# depth = l_P + r*l_R + l_C

Two details in that second block are where the architecture actually lives, and both are routinely dropped in summaries.

The embedded input is re-injected at every iteration. The core does not take s alone; it takes R(e, s_{i−1}), and internally an adapter matrix A: ℝ^2h → ℝ^h projects the concatenation [s_{i−1} ; e] back down to the hidden dimension before the four transformer layers run. The paper's justification is an analogy to gradient descent: you cannot optimise a data-dependent objective by looking at the data once. Formally, if e entered only through s_0, R could not be a monotone operator on data-dependent functions, so its fixed point would depend only on the boundary condition rather than on the input.

The initial state is random. s_0 is drawn from a truncated normal with variance σ_s² = 2/5, cut at 3σ — fresh, per forward pass. Not from the embedded input, which is the obvious choice and the wrong one. Random initialisation together with input injection is what produces path independence (Anil et al., 2022): different initialisations converge to the same trajectory. That property is not cosmetic. It is the thing that makes it legitimate to change r at inference at all — without it, a loop count the model was never trained at is just an out-of-distribution query.

The bus on the left is the detail most summaries lose: the embedded input e enters the core on every pass, not once at the bottom. The prelude and coda are ordinary transformer layers with their own weights, used exactly once — only the core is shared, and only the core's cost scales with r.

Inside the core block:

# adapter: concatenate the state with the input, project back down to h
A : R^(2h) -> R^h
R(e, s) = Block^(l_R) ( A([ s ; e ]) )

# each layer uses "sandwich" RMSNorm (n1..n4) — load-bearing at scale
x_hat = n2( x     + Attn( n1(x)     ) )
x     = n4( x_hat + MLP ( n3(x_hat) ) )

Attention is standard causal self-attention with RoPE (base 50000) and a gated SiLU MLP; RMSNorm ε = 1e−6; learnable biases on queries and keys and nowhere else. Ordinary, in other words.

The third detail is a negative result worth more than most positive ones: the core block takes no step index. The authors tried the obvious variant — a step-conditioned core s_i = R_i(e, s_{i−1}) with a diffusion-style timestep embedding, so that iteration seven knows it is iteration seven — and report it "interacts badly with path independence, leading to models that cannot extrapolate." The block has to be blind to which iteration it is on. Whatever it has learned is therefore not a 32-step program; it is one step of something applicable an unspecified number of times.

3. What actually changes, line by line

"It loops" is not a difference an engineer can plan around. These are.

A looped model is not a smaller model. It is a model whose depth has been moved out of the weight file and into the call signature.

4. You do not train it at a depth. You train it over a distribution of depths.

This is the least-covered part of the architecture and the most interesting, because it is where the test-time dial is actually manufactured.

If you trained at a fixed r, you would get a model that works at that r and degrades away from it — which is just a deep model with tied weights. Instead the loss is an expectation over sampled depths:

Every optimiser step draws a new r from a log-normal Poisson distribution Λ with mean 32 and σ = 1/2:

The distribution is deliberately heavy-tailed: most steps run shallower than 32, and occasionally one runs far deeper. Which raises the obvious objection — if some steps unroll to 60 or 80 iterations, the activation memory should be catastrophic. It is not, because the backward pass is truncated: gradients flow through the last k = 8 iterations only. Memory and backward compute are therefore functions of k, not of r, and the heavy tail becomes affordable. Textbook truncated backpropagation through time, applied to depth instead of to time.

Depth extrapolation is manufactured here, not discovered at inference. Because the model has never been told which iteration it is on, and because it has seen many different r values with gradient only ever arriving through the last eight, the core learns an operator that is safe to apply an unspecified number of times.

Three consequences follow, and the third is the interesting one.

Stability is not automatic. The paper is unusually honest about this: two large runs failed before the third worked. A version with parameter-free RMSNorm, a parameter-free additive adapter A(s, e) = s + e and a peak learning rate of 4e−4 stalled with hidden-state collapse; a second collapsed the recurrence itself, with token correlation driving toward 1.0 — the model learning to ignore its own loop. The fix was the sandwich normalisation, a learned adapter, and dropping the peak learning rate by a factor of ten to 4e−5. At small scale every normalisation scheme worked. At scale, only one did.

Backprop is shorter than the forward pass, and that is fine. A model can be trained through 8 iterations and run at 32 or 64. The gradient never saw the deep regime. That should be alarming and mostly is not, because of the step-blindness above: what the core is being taught is not a 32-step program but a single step of a convergent operator.

Depth extrapolation at test time is the surprising part. The model was trained with a single r per sequence, and yet at inference different tokens converge at different rates — the authors show latent states settling quickly on filler and slowly on the semantically loaded words of a question. Nobody trained that. Nor is the convergence uniform in kind: PCA of the latent trajectories shows fixed points on most tokens, multi-dimensional orbits on arithmetic, and steady directional drift — "sliders" — that would let the model count how many iterations it has taken.

How far the extrapolation actually goes, stated honestly. The paper evaluates 1 to 64 iterations and plots convergence out to 128. It does not demonstrate unbounded depth. Saturation is task- and context-dependent: roughly 8–12 iterations zero-shot, about 20 with one in-context example, about 32 with 25–50 — the model uses more depth when there is more context to reason over. "Improves with more compute" is supported; "improves without limit" is not claimed.

5. Looping trades storage for time. It does not create compute.

The single most common error in the coverage of this architecture was treating the loop as free because it adds no weights. Here is the arithmetic for Huginn at each loop count, from its published parameter split.

Assumptions, so you can check this. Forward FLOPs taken as 2·(N_once + r·N_rec) with the paper's split N_once = 1.5B and N_rec = 1.5B. KV cache taken as (l_P + r·l_R + l_C) × 2 × h × 2 bytes in bf16 at h = 5280 with full multi-head attention — Huginn has no grouped-query attention. The "cache budget 4" column applies the paper's own zero-shot sharing scheme, which writes to cache slot i mod k. The r = 32 row lands on 49.5B, which is the derivation behind the paper's own statement that the model reaches "FLOP budgets equivalent to a standard 50B parameter fixed-depth transformer." Those parameter figures are compute equivalences, not claims of parity with a 50B model — and the two get conflated constantly.

Three things in that table deserve to be said out loud.

The KV cache is the sleeper problem. Every iteration runs real attention and writes its own keys and values, so the cache scales with effective depth, not with real layers. At r = 32 and a 4096-token context that is 10.6 GiB of cache for a single sequence — for a model whose weights are 7 GB in bf16. That is why the paper has a section on cache sharing at all, and the mitigation is not free either: Nanbeige tested sharing the KV cache across loop passes, found it "reduces the KV-cache by half" and that "its performance gains are consistently lower than those of the full, non-sharing loop configuration," and shipped the expensive version anyway.

Storage is saved; bandwidth is not. At batch-1 decode you still stream the core weights r times through the memory system.

elie bakouch, who builds these, put it in a line: "Why recurrent depth instead of just scaling depth? It's not faster at inference or training… since you still go through the full 'effective depth'. The advantage is storage."

The exchange rate is sublinear, and it has been measured. Schwethelm, Rueckert and Kaissis ran an iso-depth pretraining sweep over r ∈ {1, 2, 4, 8} across roughly 50× in training compute and fitted a joint scaling law in which recurrence enters as a power of r multiplying the recurrent parameter count:

One recurrence is worth $r^{0.46}$ in equivalent unique parameters — genuinely positive, and considerably less than one. That exponent is the honest centre of the subject. It is the reason a looped 3B behaves like something bigger, and the reason it does not behave like something much bigger.

Two practitioner quotes worth keeping alongside it. Sebastian Raschka, on a two-pass looped model: "almost 2× as expensive in terms of compute, because we run the embedded text through almost 2× as many layers." And the Nanbeige team, who actually shipped one, on what a second pass buys: "relative to a standard Transformer, it retains approximately 75% of the token efficiency and provides a significant capacity gain." A looped layer is not a real layer, and somebody measured by how much.

Set against the three ways a model can be given more compute, the trade becomes legible:

6. The interesting version is not a fixed loop count

A fixed r is a blunt instrument: the token the gets the same 32 passes as the operative word in a hard question. Every serious line of work in this area is converging on the same next move — let the model decide the depth per token — and there are now three distinct ways to do it.

Emergent, zero-shot. Huginn exits when the KL divergence between two successive latent states falls below 5e−4. Nothing was trained for this; the criterion is applied after the fact to a model that happened to converge. It works: on MTBench, 5.63 without early exit and 5.56 with. And the exit distribution is semantically sensible — the model leaves early on high-school mathematics and takes about 3.5 steps longer on moral scenarios.

Learned halting. The Universal Transformer had this in 2018, with per-position Adaptive Computation Time. Ouro, ByteDance's 2025 looped family, trains an entropy-regularised objective for learned depth allocation directly in pretraining.

Learned routing. Mixture-of-Recursions goes further: a lightweight router assigns each token its own recursion depth, and the KV cache is restructured to match. Two routing schemes, with a genuine trade-off. Expert-choice has each depth select its own top-k tokens, which gives perfect load balance by construction but leaks information across the causal boundary during training — fixed with an auxiliary loss that teaches the model to detect the top-k threshold at inference. Token-choice commits each token to a depth up front, which is causally clean but needs a balancing loss to stop every token stampeding to the same depth. Of four weight-sharing layouts, Middle-Cycle — unique first and last layers, shared middle — consistently wins.

Dashed cells are passes the router declined to spend. The consequence is not only cheaper compute: under recursion-wise caching, only the tokens routed to depth d write KV entries at depth d, which brings cache memory down to (N_r + 1)/2N_r of a vanilla model's. The alternative — recursive sharing, cache once at step 1 and reuse everywhere — reaches 1/N_r but pays it back in attention FLOPs.

The numbers are the strongest efficiency result in the field. At an equal training budget of 16.5e18 FLOPs, a 167M-parameter MoR model reaches 43.1% few-shot average against a 315M vanilla transformer's 42.3% — better, with 47% fewer parameters. It trains 19% faster, peaks 25% lower on memory, and reaches up to 2.06× inference throughput through "continuous depth-wise batching," which works precisely because tokens at different recursion depths can be packed into one batch. That last figure comes from the depth-4 configuration, which the authors note degrades slightly in quality — a throughput-for-accuracy trade, not a free lunch. (It also circulates as 2.18× rather more often than it should.)

Be precise about what changes here, because it is a different thing from looping itself. With a fixed r, compute per token is a deployment constant: you set it, you can log it, you can bill for it. With a learned router, how much thinking each token receives becomes a hidden, input-dependent decision inside the model. For capability that is the entire point. For anyone trying to observe the system from outside, the change is not that the reasoning became unreadable — it is that the amount of reasoning became an unlogged variable.

7. What the literature actually establishes

Strip away the Astra story and there is a real, unusually clean research result underneath, now confirmed by three groups using different methods at different scales: looping buys reasoning, not knowledge.

Loops buy thinking, not facts. If you want a model that knows more, you still need parameters. If you want one that thinks harder, you can now turn a dial — and pay for it in time rather than in memory.

I find this genuinely odd, and I do not think the field has absorbed it. Parameter scaling has always moved knowledge and reasoning together, so we never had cause to treat them as separable knobs. Recurrent depth separates them cleanly enough that three independent methodologies agree — and that result outlives the news cycle that made anyone look.

8. Is this the architecture that pushes models further?

Partly, and not in the shape the coverage implied. The evidence supports a narrow claim and does not support a broad one.

The narrow claim: recurrent depth is a genuine third scaling axis with a measured exchange rate of $r^{0.46}$, and it moves reasoning without moving knowledge. That is a scaling law, not a slogan, and the knowledge-versus-reasoning split has been reproduced independently — by the Virtual Logic Depth and Ouro teams, on different architectures, at different scales, with different objectives. Anyone shipping a small model should be looking at it today.

The broad claim — that this is the architectural change that takes frontier models to the next level — is not supported, and one specific piece of evidence cuts against it. Nanbeige trained at 28T tokens, an order of magnitude beyond any other published looped model, and found that going past two passes bought almost nothing while destabilising optimisation: "increasing the number of passes provides only marginal additional improvement, but substantially slows training and makes optimization less stable." At 28T tokens the sweet spot was r = 2. Huginn's own returns saturate between 8 and 32 iterations depending on how much context there is to chew on. The exchange rate is sublinear by construction. Three attempts were needed to train one 3.5B model at all. None of that is what a discontinuity looks like from the inside.

Where it clearly helps

• Depth-bound, serial reasoning. Arithmetic, state tracking, graph connectivity, algorithmic search. The 282%-vs-37% split between math word problems and closed-book QA is the shape of the whole effect.
• Parameter-constrained deployment. A 3B that behaves like a 12B is worth more on a laptop, a phone or a memory-bound serving tier than anywhere else. This is where the shipped models actually are.
• Test-time compute without context growth. No CoT data, no RL to elicit it, no context consumed, and it composes with a 4K window. LOTUS cuts thought-phase latency 2.5–6.9× against writing the reasoning out.
• Interconnect-poor training clusters. High FLOPs per parameter means data parallelism suffices. Huginn ran 4096 GPUs with no model parallelism at all.

Where it does not help

• Anything bound by memorised facts. Two independent results say knowledge capacity is flat in r and still tracks parameters. No amount of looping makes a model know more.
• Wall-clock latency. The iterations are a serial dependency chain. You pay the full effective depth in time. Storage is the saving; speed is not.
• Long-context serving. KV cache grows linearly in r. The sharing fix works and measurably costs quality — Nanbeige declined it.
• Large loop counts. Sublinear returns, saturating benchmarks, and optimisation that gets less stable the deeper you go.

What is genuinely unsolved

• Stability at large effective depth. Huginn needed three attempts, a sandwich-norm block and a 10× learning-rate cut. Nobody has published a recipe known to hold at frontier scale.
• Composition with sparsity. Every frontier model is a mixture-of-experts; every looped result is dense. MoEUT shows the combination can work below 1B and is arguably necessary, since shared layers have a poor parameter-to-compute ratio. Above that, nothing public.
• Whether depth extrapolation survives scale. It is demonstrated at 3.5B on 795B tokens. The one 28T-token data point suggests the useful range of r shrinks as training gets long.
• Post-training. Outcome-level RL still works, but the process supervision that drives current reasoning models has no purchase on a step you cannot read. LOTUS's answer — supervise latent positions against gold CoT tokens — works, and gives back the data dependency that was the point.

The likely future is not looped models replacing transformers. It is a small loop count, a learned router and a sparse MoE core, all in the same model — with recurrent depth as a component rather than a paradigm.

Which is roughly what Mixture-of-Recursions already is, at 1.7B. What is missing is everything above that scale.

9. Back to the argument, which was two arguments

Which brings us back to where this started. Both camps argued well and mostly past each other, because the question they had merged is really two questions with two different answers.

On question one, the deflationary camp is right, and everything above is one long demonstration of why. Nothing in the mechanism supports the alarming reading: extra iterations add computation between emitted tokens exactly as extra layers always have, and the model still writes its reasoning out. Pachocki's "within a factor of two of GPT-4" is a claim about effective depth, which for a looped model means a small r — the regime where, at $\varphi = 0.46$, this is a modest efficiency choice rather than a new kind of system.

On question two, the alarmed camp is pointing at something real, and it does not depend on the Astra rumour being true at all. Nobody has to decide to hide anything. The most striking thing in the whole episode is that OpenAI's chief scientist agrees the transcript is degrading; he just says the architecture is not why.

The architecture is not the threat. The gradient is. Both sides concede the same trend — they disagree only about its cause.

The consequence for anyone building on these models is unchanged by the technical detail, and possibly sharpened by it. If legible reasoning is something you depend on — for audit, for evaluation, for debugging an agent that touches production — stop treating it as a property of the architecture you happened to buy and start treating it as something you specify, price and test for. And note which direction the frontier is heading: per-token routed depth, where the amount of computation spent on a token is itself a learned and unlogged decision. That is a monitorability question with real content, and it is not the one anyone spent September arguing about.

10. What would actually settle this

1. Does depth extrapolation survive at frontier scale? The only public demonstration is 3.5B on 795B tokens. The only 28T-token data point says the useful loop range shrinks. This is the load-bearing empirical question for the whole architecture and it is open.
2. Does it compose with sparsity? MoEUT works below 1B. Every frontier model is a mixture-of-experts. Nobody has published a large looped-plus-sparse model, and the parameter-to-compute argument says that combination is where the real design lives.
3. What does the KV cache actually cost in production? Naïve caching scales with effective depth; both published mitigations trade quality or FLOPs. A serving-side study at long context would settle whether this architecture is deployable where it is most wanted.
4. Can latent steps be supervised without gold CoT? LOTUS closes the latent-versus-explicit gap by supervising each latent position against a written reasoning step. Doing it without that crutch is the difference between an efficiency trick and a new training paradigm.
5. Astra's loop count, if any. "Within a factor of two of GPT-4" implies a small r. Publishing the number would end that argument in a sentence — and, per everything above, probably in the deflationary direction.
6. Did reasoning tokens per task go down? The measurable version of the safety worry, testable from outside. If Astra writes visibly less reasoning than the o-series for equivalent tasks, the economics argument has teeth.

Not one of those turns on what Astra does. The rumour was never the interesting part. It was the thing that got everybody to look.

Sources

Architecture

• Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Geiping, McLeish, Jain, Kirchenbauer, Singh, Bartoldson, Kailkhura, Bhatele, Goldstein. Huginn-0125; the only fully specified public recurrent-depth model, and the source of every architectural number above.

Theory

• Reasoning with Latent Thoughts: On the Power of Looped Transformers — Saunshi, Dikkala, Li, Kumar, Reddi (ICLR 2025)
• Chain of Thought Empowers Transformers to Solve Inherently Serial Problems — Li, Liu, Zhou, Ma (ICLR 2024). The $\mathrm{AC}^0$ / circuit-size result.
• A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers — Merrill, Sabharwal

Scaling

• Beyond Parameters: Exploring Virtual Logic Depth for Scaling Laws — Zhu, Zhang, Li, Shi, Duan, Wang, Zhou, Banerjee, Qin
• How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models — Schwethelm, Rueckert, Kaissis. The φ = 0.46 exponent.

Adaptive depth

• Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation — Bae et al. (NeurIPS 2025)
• Universal Transformers — Dehghani, Gouws, Vinyals, Uszkoreit, Kaiser. Shared-weight recurrence with per-position adaptive computation time.

Sparsity, counterweights and shipped models

• MoEUT: Mixture-of-Experts Universal Transformers — Csordás, Irie, Schmidhuber, Potts, Manning
• Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers — Fan, Svete, Lee (LOTUS)
• Scaling Latent Reasoning via Looped Language Models (Ouro, ByteDance) · Nanbeige4.2-3B
• Open weights: Huginn-0125, Nanbeige4.2-3B, Ouro-1.4B
• Reasoning Beyond Language: A Survey on Latent Chain-of-Thought Reasoning

The argument

• OpenAI Astra and Looped Transformers — Sebastian Raschka
• Fortune, on The Information's report and the safety response
• Jakub Pachocki, chief scientist, OpenAI — 2 September 2026
• @nrehiew_ on looped transformers and CoT monitorability · elie bakouch on why the advantage is storage

Architectural figures are quoted from the papers named beside them. Where a number is derived rather than quoted — the FLOP and KV-cache table in section 5 — the assumptions are stated in full so the arithmetic can be checked. The Astra architecture claim remains single-sourced and uncorroborated.