XNews

Stanford CS336 Lecture 10 Covers Inference Optimization from Architecture to Systems

Stanford CS336 Lecture 10 Covers Inference Optimization from Architecture to Systems

Stanford's CS336 course released Lecture 10, "Inference," presented by Professor Percy Liang between the Scaling Laws and Scaling Laws II lectures. The lecture traces a mathematical chain from autoregression through low arithmetic intensity to memory-bound execution, framing all optimization as reducing bytes moved per step across architecture, algorithm, and system layers. It concludes that the KV cache is a structural source of Transformer inference inefficiency and that architectures designed for inference from the start, such as SSMs, linear attention, and diffusion-based non-autoregressive generation, may offer the field's largest room for improvement.

Event version 1.

Key insights

Mathematical chain frames all optimization

The lecture orders its logic as autoregression, inability to parallelize along the sequence, low arithmetic intensity, memory-bound execution, and finally that all optimization equals reducing bytes moved per step.

Architecture-layer techniques are lossy

GQA, MLA, and CLA dimensionality reduction, local attention truncation, quantization, and pruning/distillation all serve to shrink the KV cache and parameters, with evaluations required to show accuracy is not harmed.

Speculative sampling offers lossless speedup

Speculative sampling exploits the asymmetry that verification is faster than generation to obtain exact acceleration at no accuracy cost.

System layer ports OS ideas to GPU serving

Continuous batching, paging, prefix sharing, and copy-on-write migrate classic operating system concepts to GPU serving.

KV cache identified as structural bottleneck

The lecture concludes that the KV cache is the structural root of Transformer inference unfriendliness, and that architectures designed for inference from day one, including SSMs, linear attention, and diffusion-based non-autoregressive generation, may be the field's largest improvement space.

Source posts