GLM-5.3-Flash Inference Optimization by AI Agent

Zhipu AI announced that its GLM-5.3-Flash model's production inference is fully running on over 100,000 domestic AI accelerators, with end-to-end throughput improved 3.2x within two weeks. The optimization was completed by an AI agent driven by GLM-5.3 itself, achieving cost parity with mainstream NVIDIA GPUs on a per-token basis despite challenging hardware conditions.
Key insights
Optimization Techniques and Trade-offs
The engineering environment was harsh: limited accelerator memory and interconnect bandwidth, 1M token context, multimodal requests, and immature software stack with missing kernels and guesswork documentation. All optimizations were trade-offs: ReplaySSM (compute for memory), node-intra tensor parallelism (communication for memory), mixed INT8/FP8/BF16 cache (precision for capacity), and encode-prefill-decode separation (flexibility for scheduling). The final per-token cost on domestic chips matched mainstream NVIDIA GPUs.
Agent's Debugging Methodology: Dense Feedback
The agent's main bottleneck was not coding but diagnosing performance degradation (e.g., 20% throughput drop). This was framed as a sparse reward and credit assignment problem. The solution was to explicitize human tacit knowledge into layered verification interfaces (dense feedback) with three signal types: correctness, system behavior, and performance. Each signal had to be local, cheap, and objectively verifiable. This effectively designed an environment for the agent rather than supplementing its capabilities.
Three Real-World Cases Discovered and Fixed
The agent identified and fixed three real engineering problems: 1) Precision drift in KDA kernels due to TF32 rounding errors accumulating in chained state matrix merges (merged into Flash Linear Attention). 2) KV transfer and DeepEP dispatch being unable to overlap because the intra-node path did not release GIL, causing overhead >30% reduced to <1%. 3) Decode kernel recomputing the same normalization four times due to block partitioning; restructuring achieved 1.71x speedup by applying optimization skeletons distilled from SGLang, FLA, and DeepGEMM.
Broader Implications: Environment Design as Data Flywheel
CEO Tang Jie stated that while humans define goals, build feedback environments, and review high-risk changes, the engineer's role shifts from problem solver to feedback designer. The layered verifiable feedback environment serves a dual purpose: it is both the agent's working environment and training material for the next generation model—each completed task becomes training data for successors. Tang concluded that while recursive self-improvement is far away, the smallest closed loop already exists: models optimize systems, systems run models.