XNews

Boltzbit and Cambridge propose infinite-parameter LLMs

Boltzbit and the University of Cambridge proposed a method for LLMs to dynamically generate low-rank weights from user input using a hypernetwork, instead of relying on fixed weights and context windows. This 'infinite parameter' approach outperforms traditional methods in long-document and multi-turn scenarios, with constant compute cost per turn.

Key insights

Problem: Fixed Weights and Context Window Inefficiency

Current LLMs have frozen weights after training. New information, corrections, or long documents must be placed in the context window, which is re-read from scratch each turn, consuming significant memory and compute. The model forgets everything after the conversation ends. This approach is inefficient for long documents and multi-turn dialogues.

Solution: Hypernetwork Generates Dynamic LoRA Weights

The proposed method uses a lightweight hypernetwork that compiles user-provided real-time interaction data into low-rank (LoRA) weights. A Bayesian mechanism allows these weights to evolve dynamically during conversation, updating like a snowball. This effectively writes knowledge into weights rather than prompts, enabling the model to 'remember' without re-reading context.

Infinite Parameters: Beyond Fixed Mixture of Experts

Traditional MoE selects from a fixed set of experts (e.g., 64 or 128), limiting combinations to a convex hull. The infinite-parameter LLM does not store fixed experts; instead, it generates specialized experts on demand from a continuous latent space based on input data. Since input data and latent space are continuous and unbounded, the generated weights are also unbounded, hence 'infinite parameters'.

Experimental Results: Superior in Long Context and Multi-Turn

Three core experiments showed that the method underperforms on short texts compared to direct prompt reading, but comprehensively outperforms traditional methods on long documents, multiple distractors, and multi-turn dialogues. Accuracy monotonically increases with turns, reaching near-perfect scores for pronoun resolution. Compute cost per turn remains constant from turn 1 to 100, as only fixed-dimension inner product updates are performed.

Potential Impact: Efficient Long-Context and Persistent Memory

This approach could dramatically reduce the computational cost of handling long documents and extended conversations. By dynamically updating weights, the model can maintain context without linearly increasing memory usage. It offers a path toward LLMs that learn continuously during inference, similar to human memory.

Source posts