On this page

Evaluate against a rubric

Check for a deadline from the source in an answer about travel rules. For a structured response, check that the text parses as JSON and satisfies a schema. Save related criteria as a named rubric and publish a version.

Publish the criteria before running an assessment

  1. Open Rubric library and create a set of criteria.

  2. Add a unique key, an instruction and a required flag for each criterion. Choose its kind. An LLM check uses the instruction and needs an explicit judge model. A regex check uses a pattern, match mode, flags and expected match. A JSON Schema check uses a schema in Draft 2020-12.

  3. Publish a version and select that exact version for your assessment. Publishing a change creates a new version. Existing runs and pipeline checks keep their pinned version. Archiving removes a rubric from new selections while preserving past references.

Send POST /api/v1/rubrics/ with a name, then POST /api/v1/rubrics/{id}/versions with criteria to do this via REST. Authenticate both requests with the signed-in account’s access token in Authorization: Bearer <access_token>. The version response provides its UUID and content hash. See the criterion schema for the exact rule fields.

Measure the pipeline or assess saved answers

  • New run. In the evaluation form, turn on Answer criteria, select the rubric version, and supply a judge for LLM criteria. The Space must produce answers. To score the rubric alone, select Criteria only (no numeric metrics). The REST equivalents are the rubric field with rubric_version_id and optional judge, and selected_metrics: [] for rubric-only scoring.

  • Completed run. Use Assess stored answers in the run detail. Choose a new rubric version or judge for the stored answers. The operation keeps the existing results and skips retrieval and generation. Use POST /api/v1/spaces/{space_id}/evaluations/{run_id}/rubric-assessments to enqueue it and GET to list assessments. Append /{assessment_id}/results for per-question results.

  • Live answers. A rubric node in the answer pipeline observes or enforces one pinned criterion as each answer is produced. Its action controls publication and any correction attempt. The offline assessment above measures stored candidates and records its own verdicts.

Interpret and compare criteria

A single question can have a publication outcome and a rubric verdict. For a blocked publication, assessment can grade the last retained attempt while leaving the public answer blocked. Where an attempt is unavailable, the result identifies the question as unassessed. The criterion pass rate is passed / (passed + failed). Inspect the separate counts for unevaluable results, unassessed examples and partial context before using the rate to choose a pipeline.

In the comparison screen, pick the two completed assessments explicitly. Match the dataset, rubric version and content hash, and judge configuration. Some question-and-criterion pairs remain incomparable when a verdict is missing or semantic context is incomplete. Read the reported reason alongside the before and after verdicts. The API view is GET /api/v1/spaces/{space_id}/evaluations/rubric-comparison with assessment_a and assessment_b.

See rubric comparison rules.

ContextBase0.1.0

Search the documentation