On this page

Evaluate retrieval and answers

Run a first evaluation

  1. In Datasets → New dataset, upload Next Product Lab sample or use the built-in sample. Your account owns the dataset. Reuse it in any space you may evaluate.

  2. In the space’s Evaluation screen, select the dataset under Run an evaluation and click Run evaluation. Check that workers are running. The UI runs in a temporary space with deployment defaults.

  3. Open the run from Runs. Wait for Done, then open Metrics. Use the REST example below for per-question results from one run, or Read metrics and compare runs for paired rows from two completed runs.

Dataset and metric inputs

Inspect a document and its question in the dataset detail. Each corpus item needs a unique id within the dataset and text. Each item in queries needs a question. Supply relevant_ids pointing to corpus IDs for retrieval metrics. Answer metrics need generation and a judge. Supply golden_answer when a metric needs a reference. Use representative inputs to choose a production configuration.

  1. Dataset

    Prepare the corpus, questions and any reference labels.

  2. Run

    ContextBase records the pipeline and scores the questions.

  3. Results

    Review measured metrics and any skipped metric families.

  4. Comparison

    Compare completed runs on the same dataset.

To collect real questions from a space you own, use its query log.

Create and watch a run over REST

Set CONTEXTBASE_BASE_URL, CONTEXTBASE_SPACE_ID and CONTEXTBASE_API_KEY as in Use ContextBase from an application. Use a manage key issued by the current space owner. Save next-product-lab-evaluation.json in the terminal’s working directory.

curl --fail-with-body "$CONTEXTBASE_BASE_URL/api/v1/eval-datasets/" \
  -H "X-API-Key: $CONTEXTBASE_API_KEY" \
  -H 'Content-Type: application/json' \
  --data-binary @next-product-lab-evaluation.json

Save the dataset id from the response as DATASET_ID. The response also reports corpus and query counts and whether relevance labels and reference answers are present. This run measures retrieval alone in a temporary space.

EVALUATIONS_URL="$CONTEXTBASE_BASE_URL/api/v1/spaces/$CONTEXTBASE_SPACE_ID/evaluations"
curl --fail-with-body "$EVALUATIONS_URL" \
  -H "X-API-Key: $CONTEXTBASE_API_KEY" \
  -H 'Content-Type: application/json' \
  --data "{\"dataset_id\":\"$DATASET_ID\",\"selected_metrics\":[\"retrieval\"]}"

The server returns 202 with a run id. Save it as RUN_ID and read the run until it reaches done or failed. Fetch the question rows once the run is ready.

curl --fail-with-body "$EVALUATIONS_URL/$RUN_ID" \
  -H "X-API-Key: $CONTEXTBASE_API_KEY"

curl --fail-with-body "$EVALUATIONS_URL/$RUN_ID/queries?page=1&page_size=20" \
  -H "X-API-Key: $CONTEXTBASE_API_KEY"

The dataset is posted as one JSON request. The API validates the corpus and rejects an unknown relevance ID or duplicate corpus ID. The server’s shipped request body limit is 96 MiB. If upload returns 413, reduce the size or ask the operator to check DATA_UPLOAD_MAX_MEMORY_SIZE and the edge proxy limit. Read the corpus and questions separately through the paginated /api/v1/eval-datasets/{id}/corpus and /api/v1/eval-datasets/{id}/queries resources.

Set the run conditions

  • top_k: chunks retrieved for each question. The installation defaults to 30 in the shipped configuration. The cutoff for reported ranking metrics counts distinct documents, defaults to 10, and is configured per installation. The API rejects a depth below the configured cutoff when retrieval metrics are requested.

  • query_concurrency: maximum questions in flight for this run. The shipped default is 4. Reduce concurrency when the model provider or database serves requests more slowly than the run sends them.

  • judge_model: optional model for numeric answer metrics. Omission uses the Space model when it can judge. Configure provider access on the server.

  • selected_metrics: omit to request both numeric metric families. To request only one family, set ["retrieval"] or ["answer_quality"] in the REST request. The interface offers the default and the criteria-only option, equivalent to [] when a rubric is supplied. See Evaluate against a rubric.

  • space_use: determines where scoring runs. Set it in the REST request to override the temporary Space used by default in the interface.

Mode

Suitable when

throwaway

You want an isolated corpus scored in a temporary Space. ContextBase removes the scoring Space after the run. This mode is the default.

kept

You want to inspect the loaded scoring Space or reuse it. ContextBase keeps it as an ordinary Space you can open or delete.

in_place

You have a Space filled by an earlier evaluation with the same corpus. This mode skips ingestion and graph building.

To prepare an in-place run, first run the dataset in kept mode. After the run completes, find evaluation-corpus-<run ID> under Spaces. Open that Space, wait for its indexes and any required knowledge graph to be ready and current, and start an in_place run with the same dataset. The server verifies the corpus fingerprint saved by evaluation. A regular upload of identical documents leaves that fingerprint unset and cannot be used for an in-place run.

For a reproducible comparison, save config_snapshot together with the dataset ID, top_k, the ranking cutoff in metrics.retrieval, effective_judge_model, and platform_build if recorded. The snapshot captures the pipeline and the identifiers of its embedding model and vector backend. Signed-in users can read the active deployment-wide defaults at GET /api/v1/installation/settings with Authorization: Bearer <access_token>.

Understand progress and failures

pending is queued, running is claimed by a worker, and done or failed is terminal. The progress stage reports queued, corpus loading, graph construction, or scoring questions. It counts units of that stage when the total is known. Corpus fill and embedding work grow with the corpus size. Building a graph also makes calls to its language model. Scoring grows with the number of questions and passes. Generation and model-based judging add requests when the run evaluates answers.

Use failure_reason to decide what to check.

Reason

Action

configuration_changed

The index changed after the run was queued. Check the space configuration.

embedder_unreachable or judge_unreachable

Check connectivity and provider quota.

timed_out

Try a smaller dataset or check the worker’s execution limits.

error

Read the worker logs.

abandoned

Restore the worker and queue, then start a new run.

The scoring task defaults to 6 hours soft and 6 hours 30 minutes hard. The settings are CELERY_CORPUS_TASK_SOFT_TIME_LIMIT and CELERY_CORPUS_TASK_TIME_LIMIT.

An interrupted corpus load can leave an evaluation-fill-… Space in the owner’s listing. The next temporary or kept run can continue from it if the same owner, dataset and fill configuration match. Leave a live run’s Space in place. Once an attempt stops, reading or listing runs can settle it after its heartbeat has expired. See the evaluation settings for defaults and use the worker logs to confirm that recovery is needed.

ContextBase0.1.0

Search the documentation