Perplexity Open-Sources Lily Local Inference Engine for Mac

Perplexity open-sourced Lily, a local inference engine for Mac Hybrid Compute. - Built in Rust runtime + custom Metal kernels; targets only Qwen3.6-35B-A3B and Apple Silicon; no PyTorch/MLX. - On M5 Max: prefill 1.23x MLX-LM, decode 1.35x. - Single process handles loading, session management, OpenAI-compatible API, Metal kernel execution. - Purpose: cloud handles research/inference; local model handles private files/apps. - Model: 35B params, ~3B active/token; router selects 8 of 256 experts + shared expert; 10 full-attention GQA layers; 30 Gated DeltaNet layers. - Apple Silicon: unified memory; prefill GEMM via Neural Accelerator; batch-1 decode GEMV via vector ALU. - Strategy: match GPU path to phase; minimize data movement; choose kernels by measured load. - Prefill gains: 4-bit weights 70GB→19.4GB on-chip dequant +77.4%; GPU routing +89%; 32-row tile +13.2%; register-resident recurrent state +5.6%; chunked prefill. - Decode gains: row-parallel GEMV; GPU token handoff; concurrent Metal pass for 795 kernels/555 serial stages; four-chain fusion; KV read coalescing; GQA packing +23.8% at 32K. - Long-context attention: +7.7% at 32K, +27.4% at 64K, +40.2% at 128K. - Speculative decoding 18% slower; reducing GPU launches, stage overlap, larger prefill tile, etc. did not improve. - Near hardware limit: MoE GEMM 97.9%, GEMV 90.3% of max sustained weight read; sparse GEMV arithmetic removal 0.2% change. - Official post: open-sourcing Lily powers newly introduced hybrid compute feature in Mac app.
Key insights
Lily is a single-model, single-hardware inference engine
Perplexity built Lily in Rust with custom Metal kernels, targeting only Qwen3.6-35B-A3B and Apple Silicon. It does not depend on PyTorch or MLX and runs as a single process that handles model loading, session management, an OpenAI-compatible API, and Metal kernel execution. On M5 Max, it reaches average prefill throughput of 1.23x MLX-LM and decode throughput of 1.35x MLX-LM. The project deliberately gives up generality so its execution plan, kernel selection, and runtime can be coordinated around one model architecture and one hardware family.
Hybrid Compute requires local inference speed
Perplexity's Hybrid Compute uses cloud resources for research and inference while the local model handles private files and applications. The stated reason for building a specialized engine is that if local inference cannot keep pace, the overall task experience breaks. Lily is designed to process prompts quickly during prefill and maintain high generation rate during decode, matching the Mac app's hybrid compute demands.
Prefill optimizations reuse weights and keep routing on GPU
Lily compresses 4-bit weights from about 70GB to 19.4GB and dequantizes them on-chip during GEMM, gaining 77.4% at 512-token prompt. Routing histogram, prefix sum, scatter, and block map stay in one GPU command buffer, gaining 89%. A 32-row tile with 4 simdgroups adds 13.2% at 2K prompt. Recurrent state remains in registers with fp32, adding 5.6%. Chunked prefill limits peak memory by keeping current chunk activations while weights, recurrent state, and KV cache continue across chunks.
Decode optimizations target CPU synchronization and KV bandwidth
Lily uses row-parallel GEMV with simdgroups and keeps token handoff in GPU by using double command buffers and persistent token slots, eliminating per-token CPU synchronization. One decode step launches 795 kernels and replaces 555 serial stages with a concurrent Metal pass based on real data dependencies. It fuses four chains, coalesces KV cache reads, and uses GQA packing so each KV row is loaded once and reused for four query heads, gaining 23.8% at 32K context. Long-context attention switching gives +7.7% at 32K, +27.4% at 64K, and +40.2% at 128K.
Trade-offs and hardware limits are documented
Speculative decoding was 18% slower because verification processes 2-5 rows, an inefficient shape on this hardware, and selected experts across rows increased weight reads. Reducing GPU launches, whole-stage overlap, larger prefill tiles, broader fusion, accelerated routing, and merging output projection with token selection did not improve end-to-end. MoE GEMM and GEMV reached 97.9% and 90.3% of the fastest sustained weight-read rate for their access patterns; removing arithmetic from sparse GEMV changed throughput by only 0.2%, indicating weight reading is the bottleneck.
Lily is open-source and powers Mac hybrid compute
Perplexity announced that Lily is being open-sourced and powers the newly introduced hybrid compute feature in the Perplexity Mac app. The GitHub code is available, and the release positions Lily as the local serving layer for Mac Apple Silicon devices.