On this page
Read metrics and compare runs¶
Find changed questions¶
Evaluate A and B on the same dataset, retrieval depth, rank cutoff and judge. Change one setting to measure its effect.
Under Comparison, choose a completed baseline A and a completed candidate B from the same dataset. The displayed change is B minus A. Open the configuration comparison to see what was recorded for each run.
Open improved and regressed questions. Compare relevant document ranks, or read both answers and their verdicts. Check the number of comparable questions before deciding.
Use the paired results where both runs recorded a verdict on the same question. They count improvements, regressions and ties. Where available, the paired confidence interval estimates uncertainty in the average change. A small sample or an interval that includes zero calls for more evidence. Paired answer views use each run’s original judge reading. The response identifies both models so you can spot a judge change. A historical run may have stored means without question-level answers.
The Next Product Lab question about the release notes deadline has one relevant document. If reranking moves it from second place in A to first in B, reciprocal rank rises from 0.5 to 1. Recall stays at 1 if both positions are within the cutoff.
Path |
Use |
|---|---|
|
Differences in stored means across completed runs of this dataset.
Pass |
|
Means and paired results for two completed runs in one Space.
Pass |
|
Paginated question pairs. Also choose |
See the paired comparison for the calculation inputs.
Reading metric values¶
Check why a metric family was skipped before comparing means. Missing relevance labels, an unavailable judge or an omitted family leave its measurements absent.
Metric |
What it measures |
|---|---|
MRR |
How high the first relevant document appears. A first hit at rank 2 contributes 0.5. |
Recall |
The share of the labeled documents retrieved within the cutoff. |
nDCG |
The order of relevant documents within the cutoff, weighting higher positions more. |
Faithfulness |
The share of scored answers supported by their actual grounding. |
Relevancy |
The share of scored answers that address the question. |
Correctness |
The mean judge score on a scale of 1–5 for questions with a reference answer. |
Ranking metrics deduplicate chunks to their dataset document IDs before
measuring rank. In the run response, metrics.retrieval.cutoff is the
document depth and metrics.retrieval.query_count is the number of
labeled questions included in the average. Answer metrics count empty
answers as failures, and use the lowest correctness score for an empty
answer with a reference. Check
empty_count, faithfulness_count, relevancy_count and
reference_count beside their means. For example, a high correctness
average on a handful of labeled answers covers only those questions.
Rejudge existing answers¶
To evaluate the saved answers of a completed run with another judge, send
{"judge_model":"PROVIDER/MODEL"} to
POST /api/v1/spaces/{space_id}/evaluations/{run_id}/assessments.
The new reading evaluates the stored answers and their grounding. It
preserves the original reading and skips retrieval and generation. This
isolates a change in the judging model. Use GET at the same path to
list readings and check completion. In the run detail, Judges’ readings
shows the readings and scores. For individual criteria and pinned
versions, see Evaluate against a rubric.