XNews

Model-architecture research: Nested Learning & Dynamic Concept Models

Model-architecture research: Nested Learning & Dynamic Concept Models

Research posts cover Google's Nested Learning (multi-frequency learning to address catastrophic forgetting) and Dynamic Large Concept Models (DLCM) which compress tokens into variable-length concepts, cutting FLOPs and improving reasoning.

Key insights

Nested Learning reframes continual learning with multi-frequency updates

Google's 'Nested Learning' paper argues current deep-learning designs treat all parameter updates equally, unlike the brain's multi-timescale plasticity. Key claims: continual learning benefits from multi-frequency update mechanisms; model 'parameters' include momentum and hidden states; training/test dichotomy should be reconsidered; catastrophic forgetting results from compression and can be mitigated by hierarchical timescale updates. The paper reports improved incremental learning and long-context stability on benchmark tasks.

DLCM: adaptive semantic compression into variable-length concepts

The Dynamic Large Concept Models (DLCM) paper proposes learning variable-length semantic boundaries to compress tokens into 'concepts' for deeper reasoning at concept-level. DLCM reports end-to-end learned boundary detection, a compression-aware scaling law, reduced FLOPs (reported 34% reduction), and average reasoning task improvement (reported 2.69%). The approach adapts compression granularity by content type and stabilizes training across heterogeneous architectures.

Community reading lists and Latent Space curation highlight priorities

Researchers share curated collections of 2025 essential papers and emphasize using LLMs to assist with literature review. Curated archives facilitate downloading and sharing of key papers for community study and reproducibility.

Implications for optimizer and architecture co-design

Nested Learning emphasizes that optimizers and architectures are interdependent: different architectures generate different gradient/activation patterns and require architecture-specific optimizer designs; parameterization should incorporate hidden states and momentum as learnable knowledge carriers.

Source posts