Looped transformer

A looped transformer reruns some of its own layers several times for each token, so it gets more computation without storing more parameters.

  • Models and architecture
  • Updated

A normal model passes each token through every layer once. A looped model reuses a block of layers with shared weights and runs it two or more times, which adds depth without adding stored parameters. NVIDIA's Dory1.0-4B reruns a 20-layer block one to four times, and you choose the count when you start the server. SenseTime's SenseNova-Loop-9B-Preview stores 32 layers, but layers 4 to 27 run twice.

The weights stay small, but each token costs more time, and in Dory each loop keeps its own KV cache by default, although a last-loop mode keeps memory at the one-loop size. A parameter count therefore understates the work, and runtime support is thin: SenseNova-Loop needs custom code, and Dory's GGUF route needs a pinned llama.cpp build. Check that your tools support a looped model before you plan around it.

Related terms

  • Parameters Parameters are the numbers inside a model that are adjusted during training; their count, such as 7B or 70B, is the usual measure of a model's size.
  • KV cache The KV cache is the memory a model uses to remember the earlier tokens of a conversation, and it grows with the length of the context.
  • Reasoning model A reasoning model works through a problem step by step in a 'thinking' phase, visible or hidden, before it gives its final answer.
  • Efficient attention (sparse, linear and hybrid) Efficient attention designs let a model read very long inputs with far less memory and compute than standard attention, by looking at fewer tokens or compressing the past.

All terms