Study Finds Looped and Depth-Grown Models Learn More From Context

A study by Ferdinand Kapl, Kaitlin Maile, Stefan Bauer and Johannes von Oswald examined how computation reuse across depth shapes in-context learning in language models up to 3B parameters. Grown and looped models gained more from later context than a dense reference of the same depth, scored higher on GSM8K at every demonstration count, predicted unseen formal languages more accurately, acquired in-context learning earlier in controlled tasks, and relied more on induction-ranked attention heads.
Event version 1.
Key insights
Stronger gains from context
At 3B parameters, grown and looped models showed larger early-to-late token-loss reductions than the dense baseline and higher GSM8K accuracy at every demonstration count. They also predicted unseen formal languages on RegBench more accurately, with the advantage appearing within the first string.
Earlier acquisition and stronger retention
In controlled Omniglot tasks, both model types acquired in-context learning earlier than the dense reference when labels were reassigned per sequence. When fixed class-label associations could also be stored in the weights, they kept higher in-context accuracy and lower in-weights accuracy.
Induction heads propagate across layers
Tracing attention heads through training showed that depth growth copies early induction heads across layers, and most of the strongest final heads descend from a few early ancestors. Looped models showed the same repeated pattern across applications of the shared block.
Greater reliance on induction-ranked heads
Ablations showed that grown and looped models depend more on induction-ranked heads for few-shot performance and later-token prediction. Function-vector heads remained the most important heads for few-shot accuracy in all models.