XNews

GLM-5.3-Flash inference optimization by Zhipu AI

GLM-5.3-Flash inference optimization by Zhipu AI

Zhipu AI announced that its GLM-5.3-Flash inference system, running on over 100,000 domestic accelerators, was optimized by an AI agent powered by GLM-5.3. The agent achieved a 3.2x end-to-end throughput improvement in under two weeks, using techniques like ReplaySSM and mixed-precision caching to overcome hardware constraints.

Key insights

AI Agent Achieves 3.2x Throughput Gain on Domestic Accelerators

Zhipu AI's GLM-5.3-Flash model, deployed on over 100,000 domestic AI accelerators, was optimized by an AI agent driven by GLM-5.3. The agent completed the optimization in under two weeks, achieving a 3.2x end-to-end throughput improvement. Key techniques included ReplaySSM (trading compute for memory), node-internal tensor parallelism (trading communication for memory), and mixed INT8/FP8/BF16 caching (trading precision for capacity). The deployment faced constraints like limited memory, bandwidth, and immature software stacks, but the agent's work brought single-token costs in line with mainstream NVIDIA GPUs.

Agent Debugging Framework Uses Dense Feedback Signals

The optimization process revealed that the agent's main challenge was not writing code but diagnosing why performance degraded. Zhipu AI addressed this by creating a layered verification interface with dense feedback signals: correctness (is the computation right), system behavior (where time is spent), and performance (which solution wins under what conditions). Each signal met criteria of being local, cheap, and objectively verifiable. This framework enabled the agent to identify and fix three real issues: precision drift in KDA kernels, a GIL bottleneck in DeepEP's node-internal path, and redundant normalization in a decode kernel, leading to a 1.71x speedup in one case.

Engineer Role Shifts from Problem-Solver to Feedback Designer

Zhipu AI CEO Tang Jie stated that the key takeaway is not any single optimization but the shift in engineer roles. Humans now define goals, build feedback environments, and review high-risk changes, while the AI agent handles the actual optimization work. This approach turns the agent's implicit experience into explicit, layered verification interfaces. Tang noted that this environment also serves as training data for next-generation models, creating a data flywheel. He emphasized that while recursive self-improvement is far off, the smallest closed loop already exists: models optimize systems, and systems run models.

Source posts