GLM-5.3 Inference Optimization by AI Agent

Zhipu AI said GLM-5.3-Flash production inference now runs on over 100,000 domestic AI accelerators, with a GLM-5.3-driven agent completing optimization in under two weeks for 3.2x throughput. The agent found and fixed three real bottlenecks, one cutting overhead from over 30% to under 1%.
Key insights
GLM-5.3 Agent Rebuilt Its Own Production Inference in Under Two Weeks
Zhipu AI disclosed that all of GLM-5.3-Flash's production inference now runs on more than 100,000 domestic AI accelerators, going from the model's first successful run to full production in under two weeks with end-to-end throughput up 3.2 times. The bulk of the optimization work was done not by the engineering team but by an AI agent driven by GLM-5.3. CEO Tang Jie said the recurring thought was that the main workforce was the model itself, with one model helping optimize the system that runs it, and that domestic chips' per-token cost now matches mainstream Nvidia GPUs.
Constrained Hardware Forced Explicit Tradeoffs at Every Step
The deployment environment was demanding: domestic accelerators with limited memory and interconnect bandwidth, an immature software stack with missing kernels and documentation that had to be guessed at, plus requirements for 1 million token context and multimodal requests. Each optimization was a tradeoff rather than a free win: ReplaySSM traded compute for memory, intra-node tensor parallelism traded communication for memory, mixed INT8/FP8/BF16 caching traded precision for capacity, and encode-prefill-decode separation traded architectural separation for scheduling freedom.
Dense Feedback Interfaces Solved the Real Bottleneck: Diagnosis
Zhipu's stated core lesson is that the agent almost never stalled because it could not write code, but because it did not know why performance got worse: a statement like 'throughput dropped 20 percent' is a sparse reward and credit assignment problem, and end-to-end benchmarks take hours, making exploration expensive. The company made senior engineers' implicit process rewards explicit as layered verification interfaces, called dense feedback, with three signal classes: correctness answering whether computation is right, system behavior answering where time went, and performance answering which option wins under what conditions. Each signal must be local, cheap, and objectively verifiable, essentially designing an environment for the agent rather than adding capabilities to it.
Three Real Bottlenecks Found and Fixed by the Agent
The agent found and fixed three engineering problems. First, precision drift: TF32 rounding errors accumulated through chained state matrix merges in the KDA kernel's context-parallel path, growing with sequence length; the fix was merged into the open-source Flash Linear Attention project. Second, a concurrency bottleneck: KV cache transfer never overlapped with DeepEP dispatch because the intra-node path did not release the GIL, traced across the Python/C++ boundary; after the fix, overhead fell from over 30 percent to under 1 percent. Third, redundant computation: a decode kernel computed the same normalization four times due to chunking; restructuring brought a 1.71x speedup using optimization skeletons distilled from reading SGLang, FLA, and DeepGEMM kernel code.
Three Cases Span Distinct Agent Capabilities
The three documented cases were chosen to represent different capabilities rather than easy wins. The KDA precision drift case demonstrates numerical correctness attribution, tracing TF32 rounding error accumulation through chained state matrix merges. The KV transfer and DeepEP dispatch case demonstrates system-level debugging across a cross-language boundary, following the call chain through Python and C++ to find the unreleased GIL. The decode kernel case demonstrates cross-codebase pattern transfer, where the agent applied optimization skeletons distilled from reading kernel code in SGLang, FLA, and DeepGEMM to restructure repeated normalization computations for a 1.71x gain.
Human Boundaries and a Feedback Environment That Doubles as Training Data
Zhipu draws a clear boundary: humans define goals, build the feedback environment, and review every high-risk change, shifting the engineer's role from problem solver to feedback designer. Tang Jie also pointed to a deeper implication: a layered, verifiable feedback environment built on real infrastructure tasks is exactly what training the next generation of models requires, since every completed task becomes training material for successors, merging environment design with the data flywheel. His closing statement: 'We are far from recursive self-improvement, but the minimal loop already exists. The model optimizes the system, and the system serves the model.'