XNews

Zhipu GLM-5.3 Optimizes Its Own Inference Infrastructure

Zhipu GLM-5.3 Optimizes Its Own Inference Infrastructure

Zhipu's GLM-5.3-Flash used an AI agent driven by the model itself to optimize its inference infrastructure, achieving a 3.2x throughput increase in two weeks. The system now runs on over 100,000 domestic AI accelerators, supporting 1 million token context with cost per token matching mainstream NVIDIA GPUs.

Key insights

Self-Optimization Achievement

Zhipu disclosed that GLM-5.3-Flash's entire production inference now runs on over 100,000 domestic AI accelerators. From first run to production, the model achieved a 3.2x end-to-end throughput improvement in under two weeks. The optimization work was performed by an AI agent driven by GLM-5.3 itself, not by human engineers. CEO Tang Jie noted that the model helped optimize the system running it.

Engineering Constraints and Trade-offs

The deployment environment was challenging: domestic accelerators had limited memory and interconnect bandwidth, the software stack was immature with missing kernels and poor documentation. GLM-5.3-Flash also needed to support 1 million token context and multimodal requests. Every optimization involved trade-offs: using computation for memory (ReplaySSM), communication for memory (intra-node tensor parallelism), precision for capacity (mixed INT8/FP8/BF16 cache), and separation for scheduling flexibility (encode-prefill-decode separation).

Specific Optimizations Implemented

The agent implemented several key optimizations: ReplaySSM to trade computation for memory, intra-node tensor parallelism to trade communication for memory, mixed INT8/FP8/BF16 caching to trade precision for capacity, and encode-prefill-decode separation to trade architectural separation for scheduling freedom. These optimizations allowed the domestic chips to achieve per-token cost comparable to mainstream NVIDIA GPUs.

Dense Feedback System Design

Zhipu built a layered verification interface called 'dense feedback' to guide the agent. It provides three types of signals: correctness (is the computation correct?), system behavior (where is time spent?), and performance (which solution wins under what conditions). Each signal must be local, cheap, and objectively verifiable. This approach turns the implicit experience of senior engineers into explicit, callable interfaces for the agent.

Real Bugs Discovered and Fixed

The agent identified and fixed three real engineering problems. First, precision drift in KDA context parallel paths: TF32 rounding errors accumulated through chained state matrix merging, fixed and merged into Flash Linear Attention. Second, KV transfer and DeepEP dispatch not overlapping: the agent traced the call chain across Python/C++ boundaries and found the intra-node path did not release the GIL, reducing overhead from over 30% to under 1%. Third, a decode kernel repeated the same normalization four times due to tiling; the agent reorganized computation for 1.71x speedup, using optimization skeletons distilled from reading SGLang, FLA, and DeepGEMM kernel code.

Human Role and Future Implications

Zhipu emphasized that humans still set goals, build the feedback environment, and review high-risk changes. However, the engineer's role is shifting from problem-solver to feedback-designer. Tang Jie noted that the layered verifiable feedback environment serves dual purpose: it is both the agent's working environment and training data for future models. He stated, 'We are far from recursive self-improvement, but the smallest closed loop already exists: the model optimizes the system, and the system serves the model.'

Source posts