Boltzbit and Cambridge Introduce 'Infinite Parameters' LLM
Researchers from Boltzbit and the University of Cambridge proposed an 'Infinite Parameters' LLM where a lightweight hypernetwork dynamically generates low-rank (LoRA) weights from user input, writing new knowledge into the model's weights rather than relying on the context window. In long-document and multi-turn tests, it outperforms traditional methods while keeping compute cost constant.
Key insights
Concept and Mechanism of 'Infinite Parameters'
The approach uses a hypernetwork that compiles user-provided real-time interaction data into low-rank weight updates (LoRA-like) dynamically. A Bayesian mechanism allows these weights to evolve during conversation, accumulating context like a snowball effect. Unlike traditional MoE with a fixed set of experts, this system generates new experts on the fly from a continuous latent space, making the parameter space effectively unbounded.
Performance Advantages Over Traditional Context Window
In short-text scenarios the method slightly trails direct prompt reading, but for long documents, multi-turn dialogues, and scenarios with distracting elements, it surpasses traditional approaches. Accuracy on pronoun resolution tasks monotonically increases with turns, approaching near-perfect scores. Compute cost per turn remains constant from turn 1 through 100, compared to linearly increasing cost for traditional context-based models.
Comparison to Existing Approaches
This method differs fundamentally from context window extension, retrieval-augmented generation, or fine-tuning. It does not require storing static expert parameters; instead, it synthesizes specialized modules on demand. The weight updates are applied as a temporary 'neural circuit' that is dynamically adjusted without full retraining. This allows the model to learn new facts and corrections during a single conversation.