XNews

Zhipu AI's GLM-5.3-Flash Optimizes Its Own Inference System

Zhipu AI's GLM-5.3-Flash Optimizes Its Own Inference System

Zhipu AI announced that GLM-5.3-Flash's production inference runs on over 100,000 domestic AI accelerators, with end-to-end throughput improved 3.2x in under two weeks. The optimization was performed by an AI agent powered by GLM-5.3 itself, which discovered and fixed three real engineering issues including precision drift and concurrency bottlenecks.

Key insights

Self-Optimization Achievement

Zhipu AI disclosed that GLM-5.3-Flash's entire production inference now runs on over 100,000 domestic AI accelerators. From first run to production deployment took less than two weeks, with end-to-end throughput improved 3.2x. The optimization work was done by an AI agent driven by GLM-5.3, not by human engineers. CEO Tang Jie emphasized that the model helped optimize the system running itself.

Engineering Challenges and Trade-offs

The deployment environment was harsh: limited memory and bandwidth on domestic accelerators, immature software stack with missing kernels and guesswork documentation. The model supports 1 million token context and multimodal requests. Optimizations involved trade-offs: ReplaySSM (compute for memory), intra-node tensor parallelism (communication for memory), mixed INT8/FP8/BF16 caching (precision for capacity), and encode-prefill-decode separation (separation for scheduling flexibility).

Dense Feedback System

The key innovation was building a layered verification interface providing 'dense feedback' to the agent. Three signal types: correctness (is the computation correct?), system behavior (where is time spent?), and performance (which solution wins under what conditions). Each signal is local, cheap, and objectively verifiable. This externalizes the implicit process reward that senior engineers use to debug.

Three Real Bugs Found and Fixed

The agent discovered and fixed three engineering issues. First, precision drift in KDA kernel: TF32 rounding errors accumulated through chained state matrix merging, worsening with longer context. Second, concurrency bottleneck: KV transfer and DeepEP dispatch never overlapped because the intra-node path didn't release the GIL. Third, redundant computation: a decode kernel recomputed the same normalization four times due to block partitioning; the agent restructured it for 1.71x speedup.

Engineer Role Transformation

CEO Tang Jie noted that the bottleneck was not writing code but understanding why performance degraded. The dense feedback system shifts the engineer's role from problem-solver to feedback-designer. Humans still define goals, build the feedback environment, and review high-risk changes. The agent handles the iterative optimization loop.

Recursive Self-Improvement Vision

Tang Jie highlighted a deeper implication: the layered verifiable feedback environment built for this task is also ideal training data for next-generation models. Each completed task becomes a training example for successors. He stated, 'We are far from recursive self-improvement, but the smallest closed loop already exists: the model optimizes the system, and the system serves the model.'

Source posts