XNews

Zhipu AI optimizes GLM-5.3-Flash inference with GLM-5.3-driven Infra Agent in two weeks

Zhipu AI optimizes GLM-5.3-Flash inference with GLM-5.3-driven Infra Agent in two weeks

Zhipu AI announced its GLM-5.3-Flash inference runs on over 100k domestic accelerators, with a 3.2x throughput gain in two weeks achieved by a GLM-5.3-driven Infra Agent. Optimizations include ReplaySSM, tensor parallelism, mixed caching, and pipeline separation to overcome hardware limits.

Key insights

GLM-5.3-powered Infra Agent achieves 3.2x throughput in two weeks

The Infra Agent, driven by GLM-5.3, optimized GLM-5.3-Flash inference across over 100,000 domestic AI accelerators, achieving a 3.2x end-to-end throughput improvement from first run to full production in two weeks. This demonstrates a model-driven agent can significantly accelerate infrastructure optimization.

Optimization trade-offs due to domestic hardware limitations

Severe engineering constraints included limited accelerator memory and bandwidth, 1M token context, multimodal requests, and immature software stack. Each optimization was a trade-off: ReplaySSM (compute for memory), node-level tensor parallelism (communication for memory), mixed INT8/FP8/BF16 cache (precision for capacity), and encode-prefill-decode separation (separation for scheduling flexibility).

Dense feedback system provides layered verification signals

The agent's success relies on a dense feedback framework with three signal types: correctness (is it computed right?), system behavior (where is time spent?), and performance (which solution wins and under what conditions?). Each signal is localized, cheap, and objectively verifiable, transforming the engineer role from problem solver to feedback designer.

Agent identifies and fixes three concrete engineering issues

The agent discovered three real problems: (1) TF32 rounding error accumulation in KDA context parallel paths causing precision drift with sequence length; (2) GIL blocking KV transfer overlap with DeepEP dispatch; (3) duplicate normalization in a decode kernel due to block partitioning. Each case demonstrates a distinct capability: numerical correctness attribution, cross-language debugging, and pattern transfer across codebases.

Dense feedback also serves as training data for future models

Zhipu CEO noted that the layered verification environment built for the agent also produces training data for next-generation models. Each completed task becomes training material for successors. This creates a data flywheel: the feedback environment is both the agent's workspace and the future model's training ground, though recursive self-improvement remains distant.

Domestic chip token cost matches mainstream NVIDIA GPU

After optimization, the per-token cost on domestic accelerators matched that of mainstream NVIDIA GPUs. This was achieved despite hardware limitations, demonstrating that algorithmic innovations can compensate for hardware constraints and achieve cost parity.

Source posts