XNews

Zhipu AI uses GLM-5.3 to optimize its own inference system

Zhipu AI uses GLM-5.3 to optimize its own inference system

Zhipu AI optimized its GLM-5.3-Flash inference system running on over 100,000 domestic AI accelerators using an AI agent powered by GLM-5.3 itself. The agent achieved 3.2x throughput improvement in under two weeks. Key optimizations included ReplaySSM, tensor parallelism, mixed precision caching, and encode-prefill-decode separation, enabled by a dense feedback system with local, cheap, verifiable signals.

Key insights

Dense feedback environment empowers the agent

The Zhipu AI team built a dense feedback system with three types of signals: correctness (does it compute right?), system behavior (where is time spent?), and performance (which solution wins under which conditions). Each signal is local, cheap, and objectively verifiable. This enables the agent to identify root causes of performance degradation without relying on sparse final rewards. The key insight is that designing the environment for the agent is more important than giving the agent more capabilities.

Three real engineering issues discovered and fixed by the agent

Case 1: KDA kernel's contextual parallel path accuracy drifted with sequence length due to TF32 rounding errors accumulating through chain state matrix merges. Case 2: KV transfer and DeepEP dispatch never overlapped because the in-node path did not release GIL, causing >30% overhead reduced to <1%. Case 3: A decode kernel computed the same normalization four times due to tiling; the agent reorganized computations for 1.71x acceleration, borrowing optimization patterns from SGLang, FLA, and DeepGEMM. Each case demonstrates a different capability: numerical correctness attribution, cross-language debugging, and cross-repo pattern transfer.

Engineer role shifts from solver to feedback designer

Zhipu AI CEO Tang Jie stated that the most important experience is not any specific optimization, but the change in engineer role. Humans now define goals, build feedback environments, and review high-risk changes. The engineer becomes a designer of feedback systems rather than a solver of problems. The team's layered verification interfaces externalize the implicit process reward that senior engineers carry in their heads.

Recursive self-improvement loop is forming

The feedback environment built for the agent also serves as training material for the next generation of models. Each completed task becomes training data for successors. Tang Jie commented that while recursive self-improvement is still far away, the smallest closed loop already exists: the model optimizes the system, and the system serves the model.

Constraints and trade-offs of domestic accelerator deployment

The deployment conditions were challenging: domestic accelerators with limited memory and interconnects, 1M token context, multimodal requests, immature software stack (missing kernels, poor documentation). Every optimization involved a trade-off: ReplaySSM traded compute for memory, intra-node tensor parallelism traded communication for memory, mixed INT8/FP8/BF16 cache traded precision for capacity, and encode-prefill-decode separation traded separation for scheduling flexibility. Ultimately, the per-token cost on domestic chips matched mainstream Nvidia GPUs.

Source posts