XNews

Databricks Genie Benchmark: 91% Pass Rate Against Two Frontier Agents

Databricks Genie Benchmark: 91% Pass Rate Against Two Frontier Agents

Databricks benchmarked Genie, powered by Genie Ontology, against two frontier agents on enterprise data tasks in a live Databricks workspace, running each agent five times per task. Genie passed 91% of runs versus 45% and 58% for the other agents, using less than half the tokens and finishing 3–4× faster. The benchmark, developed with Turing, tested use of business knowledge across notebooks, dashboards, and saved SQL queries rather than table queries alone.

Event version 1.

Key insights

Genie leads on pass rate

Genie passed 91% of runs, compared with 45% and 58% for the two other agents. Each agent ran five times per task.

Lower token use and faster completion

Genie used less than half the tokens of the other agents and finished 3–4× faster.

Benchmark tests business context, not just SQL

Agents had to use business knowledge spread across notebooks, dashboards, and saved SQL queries in a live Databricks workspace. The benchmark was developed with Turing.

Business context drives correctness

A valid SQL query can still produce the wrong answer if it uses the wrong definitions or calculation rules. Genie Ontology retrieves and prioritizes business knowledge so agents apply the right context instead of relying on table schemas alone.

Source posts