XNews

DeepEval Releases JevEval, Replacing Generative LLM Judges with Structured Decision Making

DeepEval Releases JevEval, Replacing Generative LLM Judges with Structured Decision Making

DeepEval released JevEval, an evaluation framework that replaces generative LLM judges with Jev to convert LLM evaluation from generative verdicts into structured decisions. The framework separates evaluation logic, semantic judgment, and score computation, with Jev returning probability distributions instead of generated text and DeepEval applying fixed deterministic formulas. It supports three decision types—Noul for binary propositions, Score for ordered grades, and Choice for unordered behavior selection—and is applied to agent evaluation such as Tool Faithfulness.

Event version 3.

Key insights

Separation of evaluation stages

JevEval divides evaluation into three stages: humans define explicit and reviewable evaluation logic, Jev performs semantic judgment by returning probability distributions without generating text, and DeepEval computes scores using fixed deterministic formulas.

Three decision types

Noul handles binary propositions by returning P(true) for checklist-style checks. Score handles ordered grades by returning probabilities for each level and taking the normalized expected position. Choice handles unordered behavior selection by assigning 0–1 credit per option and supporting a not-applicable option when probability is at least 50%.

Agent evaluation application

For Tool Faithfulness, JevEval decomposes evaluation into four explicit questions covering whether facts come from tools (weight 2), numerical transcription accuracy, overall groundedness, and handling of information not provided by tools. Weights and priorities are written in code rather than implicit in prompts.

Limitations of traditional judges

Traditional LLM-as-a-judge has a generative model define evaluation logic, make judgments, and compute scores in one call. This produces uninterpretable score differences such as 0.71 versus 0.66 and unstable results where the same input can yield different scores across runs.

Source posts