On this page
Evaluate against a rubric¶
Check for a deadline from the source in an answer about travel rules. For a structured response, check that the text parses as JSON and satisfies a schema. Save related criteria as a named rubric and publish a version.
Publish the criteria before running an assessment¶
Open Rubric library and create a set of criteria.
Add a unique key, an instruction and a required flag for each criterion. Choose its kind. An LLM check uses the instruction and needs an explicit judge model. A regex check uses a pattern, match mode, flags and expected match. A JSON Schema check uses a schema in Draft 2020-12.
Publish a version and select that exact version for your assessment. Publishing a change creates a new version. Existing runs and pipeline checks keep their pinned version. Archiving removes a rubric from new selections while preserving past references.
Send POST /api/v1/rubrics/ with a name, then
POST /api/v1/rubrics/{id}/versions with criteria to do this via
REST. Authenticate both requests with the signed-in account’s access token
in Authorization: Bearer <access_token>. The version response provides
its UUID and content hash. See
the criterion schema
for the exact rule fields.
Measure the pipeline or assess saved answers¶
New run. In the evaluation form, turn on Answer criteria, select the rubric version, and supply a judge for LLM criteria. The Space must produce answers. To score the rubric alone, select Criteria only (no numeric metrics). The REST equivalents are the
rubricfield withrubric_version_idand optionaljudge, andselected_metrics: []for rubric-only scoring.Completed run. Use Assess stored answers in the run detail. Choose a new rubric version or judge for the stored answers. The operation keeps the existing results and skips retrieval and generation. Use
POST /api/v1/spaces/{space_id}/evaluations/{run_id}/rubric-assessmentsto enqueue it andGETto list assessments. Append/{assessment_id}/resultsfor per-question results.Live answers. A rubric node in the answer pipeline observes or enforces one pinned criterion as each answer is produced. Its action controls publication and any correction attempt. The offline assessment above measures stored candidates and records its own verdicts.
Interpret and compare criteria¶
A single question can have a publication outcome and a rubric verdict.
For a blocked publication, assessment can grade the last retained attempt
while leaving the public answer blocked. Where an attempt is unavailable,
the result identifies the question as unassessed. The criterion pass rate
is passed / (passed + failed). Inspect the separate counts for
unevaluable results, unassessed examples and partial context before using
the rate to choose a pipeline.
In the comparison screen, pick the two completed assessments explicitly.
Match the dataset, rubric version and content hash, and judge configuration.
Some question-and-criterion pairs remain incomparable when a verdict is
missing or semantic context is incomplete. Read the reported reason
alongside the before and after verdicts. The API view is
GET /api/v1/spaces/{space_id}/evaluations/rubric-comparison
with assessment_a and assessment_b.