XNews

Meta Deploys JiTTests LLM Bug-Catching System Across Engineering Infrastructure

Meta Deploys JiTTests LLM Bug-Catching System Across Engineering Infrastructure

Meta deployed Catching Just-in-Time Tests (JiTTests), an LLM-based system that generates executable tests designed to fail on code changes before they land, into its production infrastructure. The company reported that 589 engineers engaged with the system and reported a 60% true-positive rate, and that it caught 74 privacy SEVs at the diff stage in the first half of 2026, lowering overall privacy SEV occurrence by 20%.

Event version 1.

Key insights

Tests designed to fail surface behavioral differences

JiTTests automatically generate executable tests that are designed to fail on a code change and pass on the parent version, exposing behavioral differences before code lands. The approach differs from LLM-based code review by producing concrete failing tests rather than reviewing code like a human reviewer.

Two tandem workflows target likely failure points

An intent-aware workflow infers the intention and risks of a code change and targets test generation toward likely failure points. A dodgy diff workflow treats every code change as a mutant and generates tests to expose behavioral differences without reasoning about intent.

Filtering reduces false positives from weak catches

The system uses the RubFake rule tool and a panel of multiple large language models to rank failing tests and distinguish strong catches from weak catches. On 600 changes with weak catches, the panel reduced manual review burden by about 70%.

Production results reported after initial deployment

An initial paper reported 41 candidates to engineers, with 8 confirmed true positives at about 19.5%, 4 of which could have caused severe failures if not intercepted. Later production figures show 589 engineers engaged, a 60% true-positive rate, and 74 privacy SEVs caught at the diff stage in the first half of 2026.

Engineers remain in the loop for judgment

The workflow requires engineers to set standards and review summaries, typically confirming whether behavioral changes match expectations. The system runs overnight using idle compute and prioritizes high-risk changes from the previous day using a Diff Risk Score-like mechanism.

Source posts