On this page

Read metrics and compare runs

Find changed questions

  1. Evaluate A and B on the same dataset, retrieval depth, rank cutoff and judge. Change one setting to measure its effect.

  2. Under Comparison, choose a completed baseline A and a completed candidate B from the same dataset. The displayed change is B minus A. Open the configuration comparison to see what was recorded for each run.

  3. Open improved and regressed questions. Compare relevant document ranks, or read both answers and their verdicts. Check the number of comparable questions before deciding.

Use the paired results where both runs recorded a verdict on the same question. They count improvements, regressions and ties. Where available, the paired confidence interval estimates uncertainty in the average change. A small sample or an interval that includes zero calls for more evidence. Paired answer views use each run’s original judge reading. The response identifies both models so you can spot a judge change. A historical run may have stored means without question-level answers.

The Next Product Lab question about the release notes deadline has one relevant document. If reranking moves it from second place in A to first in B, reciprocal rank rises from 0.5 to 1. Recall stays at 1 if both positions are within the cutoff.

Path

Use

GET /api/v1/eval-datasets/{dataset_id}/compare

Differences in stored means across completed runs of this dataset. Pass run_a and run_b.

GET /api/v1/spaces/{space_id}/evaluations/compare

Means and paired results for two completed runs in one Space. Pass run_a and run_b.

GET /api/v1/spaces/{space_id}/evaluations/compare/queries

Paginated question pairs. Also choose metric and optionally filter by change.

See the paired comparison for the calculation inputs.

Reading metric values

Check why a metric family was skipped before comparing means. Missing relevance labels, an unavailable judge or an omitted family leave its measurements absent.

Metric

What it measures

MRR

How high the first relevant document appears. A first hit at rank 2 contributes 0.5.

Recall

The share of the labeled documents retrieved within the cutoff.

nDCG

The order of relevant documents within the cutoff, weighting higher positions more.

Faithfulness

The share of scored answers supported by their actual grounding.

Relevancy

The share of scored answers that address the question.

Correctness

The mean judge score on a scale of 1–5 for questions with a reference answer.

Ranking metrics deduplicate chunks to their dataset document IDs before measuring rank. In the run response, metrics.retrieval.cutoff is the document depth and metrics.retrieval.query_count is the number of labeled questions included in the average. Answer metrics count empty answers as failures, and use the lowest correctness score for an empty answer with a reference. Check empty_count, faithfulness_count, relevancy_count and reference_count beside their means. For example, a high correctness average on a handful of labeled answers covers only those questions.

Rejudge existing answers

To evaluate the saved answers of a completed run with another judge, send {"judge_model":"PROVIDER/MODEL"} to POST /api/v1/spaces/{space_id}/evaluations/{run_id}/assessments. The new reading evaluates the stored answers and their grounding. It preserves the original reading and skips retrieval and generation. This isolates a change in the judging model. Use GET at the same path to list readings and check completion. In the run detail, Judges’ readings shows the readings and scores. For individual criteria and pinned versions, see Evaluate against a rubric.

ContextBase0.1.0

Search the documentation