Databricks Cuts Qwen 3.8 27B Fine-Tuning Time 6x on AI Runtime

Databricks reduced Qwen 3.8 27B supervised fine-tuning time from 7 hours 44 minutes to 1 hour 17 minutes on its AI Runtime, a 6x speedup. Profiling showed the CPU, not the GPU, was the bottleneck, and the gains came from installing fused kernels (flash-linear-attention and causal-conv1d) for a 3.7x improvement and 1D sequence packing for a further 1.6x.
Event version 1.
Key insights
CPU scheduling, not GPU, was the bottleneck
Profiling showed CPU time per step (14.4s) exceeded GPU time (10.3s), with about 1.71 million CPU calls per step. GPU utilization of 98% masked the inefficiency because it only measures whether the GPU is working, not how well.
Fused kernels delivered a 3.7x speedup
Installing flash-linear-attention and causal-conv1d reduced training from 7h44m to 2h05m and cut per-step operator counts roughly 15x. Databricks rejected CUDA Graphs and torch.compile due to FSDP2 conflicts and slow compilation.
1D sequence packing added a further 1.6x
Concatenating samples into one dimension with correct position_ids cut training from 2h05m to 1h17m. The approach reduced batch count by about 36% and lowered JIT compilation overhead.
Kernel loading and training support require verification
Dynamic loading via HF Kernel Hub can silently fall back to unoptimized paths on version mismatch, and some kernels implement only forward passes, making them unusable for training.
Platform tooling supported the optimization
AI Runtime packages updated CUDA and Flash Attention, maintains profiler tooling, includes built-in GPU metrics, and Genie Code actively identifies such optimizations at the code level.