Boltzbit and Cambridge Introduce 'Infinite Parameters' Method for LLMs
Boltzbit and Cambridge researchers introduced an 'Infinite Parameters' method that uses a hypernetwork to dynamically compile user interaction data into LoRA weights, enabling real-time weight updates during conversation. The approach outperforms traditional context-window methods in long documents and multi-turn dialogues, with constant compute cost per turn and improving accuracy via Bayesian mechanisms.
Key insights
Limitations of Current LLM Context Windows
Current LLMs freeze weights after training, forcing users to inject new facts, corrections, or long documents into the context window. This requires re-reading the entire context each turn, consuming excessive memory and compute, and forgetting everything after the conversation ends. The 'Infinite Parameters' method addresses this by writing knowledge into weights instead of prompts.
Hypernetwork and Bayesian Mechanism for Dynamic Weight Generation
The method uses a lightweight hypernetwork to compile real-time user interaction data into low-rank LoRA weights, with a Bayesian mechanism that dynamically evolves these weights as the conversation progresses. This allows the model to 'temporarily write into its brain' rather than relying on a static context window, enabling continuous adaptation without resetting.
Comparison with Traditional Mixture-of-Experts (MoE)
Traditional MoE selects from a fixed set of 64 or 128 expert modules, limiting combinations to a finite convex hull. In contrast, 'Infinite Parameters' generates bespoke experts on demand from a continuous latent space, producing unbounded weight combinations. This eliminates the discrete routing constraint and allows the model to adapt to any input without predefined expert boundaries.
Experimental Results: Long Documents and Multi-Turn Superiority
Three core experiments showed that the method underperforms on short texts compared to direct prompt reading, but comprehensively outperforms traditional approaches on long documents, multi-distractor scenarios, and multi-turn dialogues. Accuracy monotonically increases with each turn due to the Bayesian mechanism, especially for ambiguous pronouns, while compute cost remains constant from turn 1 to 100.