Fine-Tuning Large Models on Low VRAM GPUs with Soup

Soup is a tool that allows fine-tuning large language models on GPUs with very limited VRAM by keeping the base model in system RAM and streaming one layer at a time into GPU memory. A laptop GPU with only 4 GB VRAM can fine-tune an 8-billion-parameter model, peaking at 3.32 GB VRAM, with training results identical to standard setups. Usage requires one configuration file and a single command; batch size and compression parameters are auto-selected. The authors provide a free Colab notebook for verification. No remote server or complex setup is needed, enabling custom model training on a gaming laptop.
Key insights
Soup breaks the VRAM barrier by layer-wise streaming, enabling fine-tuning on widely available hardware
The tool stores the base model in regular memory and loads one layer at a time to the GPU, so the GPU never holds more than a single layer. This design allowed fine-tuning an 8B model on a 4 GB VRAM GPU with only 3.32 GB peak usage, producing results indistinguishable from normal fine-tuning.
The one-command setup and auto-configuration lower the technical barrier to custom model training
Users only need a config file and a single command; Soup automaticelly handles batch sizes and model compression, removing the typical trial-and-error parameter tuning. No remote server login is required, and the tool runs on a local machine.
A free Colab notebook proves reproducibility and invites verification
The developers released a Colab notebook that lets anyone run the fine-tuning pipeline and confirm the results, adding credibility and making the approach auditable without requiring local setup first.