Recurrent Looped Transformer: A New Architectural Idea for AI
A new technical report proposes extending the loop between tokens, making the decoder recurrent for each token to increase computational depth without increasing per-token cost.
Looped Transformer is currently one of the most discussed architectural ideas in the field of artificial intelligence.
In a new technical report, the proposal is to stretch the cycle already between tokens. The Recurrent Looped Transformer makes the decoder recurrent for each token, covering both the prompt and the response.
The causal encoder forms a shared KV memory. For each new token, the decoder combines:
- The representation of this token from the encoder.
- Its own final hidden state from the previous token.
- A cache of recent activations with a sliding window.
With a 48-layer decoder, the computation path after t tokens passes through 48t decoder blocks, although each individual token still performs a fixed number of blocks.
This means that the depth of computation grows with the sequence length, while the cost of processing a single token remains constant.
The same state transition is used for pre-training, SFT, generation, and replay in RL. The state is not reset at the boundary between the prompt and the response.
During RL replay, states are rebuilt with the current model weights instead of reusing outdated states from previous runs.
For now, this is purely an architectural proposal. The author explicitly states that improvements in reasoning, hardware acceleration, and RL scaling are goals that have not yet been measured.
Why it matters
AnalysisThis approach offers a way to scale the effective depth of neural networks dynamically based on sequence length, potentially improving model reasoning capabilities without linearly increasing inference costs.
Discuss in community
Share your questions and insights with developers
Related in Ecosystem
Practical guides, core terminology, and agent skills
We understand Hugging Face and transformers: from startup to architectures
A step-by-step guide for those who want to understand how modern language models (BERT, GPT and others) work, how to run them, use them and adapt them to their tasks - using the example of the Transformers library and the Hugging Face platform.
Architectural Erosion from Fast Coding
The unnoticed accumulation of hidden systemic contradictions and the degradation of architectural boundaries in a project due to patchy local code fixes by language models.
Transformer Architecture
The neural network architecture introduced by Google researchers in 2017 in the paper 'Attention Is All You Need'. It forms the foundation of all modern language models (GPT, Claude, Gemini, Llama), replacing slow recurrent networks and enabling parallel processing of entire text sequences.
How to Build a Custom Design System in Claude Design: From Moodboard to Animated Mascot
Complete practical guide to building a brand design system in Claude Design: visual references, 4 diverging concepts, tokenization, stress-testing, and animated SVG mascot design.