On this page

Run and recover background work

Follow the operation, then inspect its deliveries

Read GET /api/v1/spaces/{space_id}/document-operations/{operation_id} with space read access before retrying. Match the stored state to the worker and Beat logs. Beat can recover the same operation after a lost delivery. Celery PENDING may mean its result backend has no final record.

  1. PostgreSQL

    Core records the accepted document operation.

  2. Redis

    The broker delivers a task to its configured queue.

  3. Celery worker

    A worker consumes the delivery and processes the operation.

  4. Application status

    Core reports the operation’s stored state and result.

The document lifecycle recovery entry runs each minute when Beat is running and a worker listens on its configured queue. A preparation claim is considered lost after five minutes without a heartbeat. Wait time also includes schedule cadence, queue backlog and the remaining work. A responsive worker process establishes neither Beat scheduling nor a completed domain operation.

For paid evaluations, preserve existing run IDs when diagnosing an interruption. Pending admission has a default one-hour deadline (CELERY_EVALUATION_PENDING_TIMEOUT_SECONDS). Expiry applies to work that did not start, and does not stop a running evaluation or launch a replacement. Retry an expired request deliberately in evaluation.

Split the Quickstart worker pool

The default Quickstart uses one worker pool. The optional worker overlay separates standard work and document lifecycle work. From the checkout, supply the same Compose project name and configuration overrides used at install.

docker compose --project-directory . -f compose/docker-compose.quickstart.yml \
  -f compose/docker-compose.quickstart.workers.yml up --build -d --wait \
  --scale queue-worker=2

With these defaults there is one worker replica subscribed to celery and two queue-worker replicas subscribed to documents. Each replica runs one prefork child in this profile. The base Core worker has a different default concurrency. Inspect the running command and replies before resizing.

docker compose --project-directory . -f compose/docker-compose.quickstart.yml \
  -f compose/docker-compose.quickstart.workers.yml \
  exec -T worker celery -A server inspect active_queues

Verify that every queue used by producers has a live consumer. Producer routing for document preparation uses CELERY_DOCUMENT_PREPARATION_QUEUE. The overlay’s subscription settings CONTEXTBASE_DEFAULT_WORKER_QUEUES and CONTEXTBASE_QUEUE_WORKER_QUEUES accept comma-separated names. A subscription does not change where a producer sends work. To retire or rename a queue, pause producers including Beat and drain broker backlog and active, reserved and scheduled deliveries before stopping its last consumer. Use a warm worker shutdown and an adequate termination budget.

Run a dedicated worker from source

Set CELERY_DOCUMENT_PREPARATION_QUEUE=documents on the API, Beat and workers. Leave the ordinary worker on its existing queues. From the repository root, start the document worker.

export CELERY_DOCUMENT_PREPARATION_QUEUE=documents
poetry run celery -A server worker --pool=prefork --concurrency=4 -Q documents

Core defaults to celery for document tasks. The Quickstart worker overlay sets documents. Check PostgreSQL capacity before adding prefork processes.

Workers prepare documents in parallel. A writer permit serializes snapshot, staging and publication within each Space.

Setting

Default (seconds)

Effect

DOCUMENT_PUBLICATION_WRITER_WAIT_SECONDS

30

Wait for the writer permit.

DOCUMENT_PUBLICATION_WRITER_POLL_SECONDS

0.2

Recheck the permit. SQL connection timeouts bound each check.

CELERY_DOCUMENT_REDELIVERY_SECONDS

1

Retry a deferred operation that is still pending in SQL.

Periodic recovery can pick up an accepted operation after a broker failure.

Scale the bottleneck

Measure completed tasks, queue age, PostgreSQL backend waits and worker memory with the providers and storage you run in production. A prefork child may load its own model weights and hold a database connection. In the source production overlay, GRANIAN_WORKERS controls the number of HTTP processes and therefore the maximum number of copies of a loaded model. The packaged Quickstart starts one HTTP process per container through its Compose command. An external embedding endpoint can centralize dense embedding. Other configured local pipeline steps still load their own models.

Review the task soft and hard limits together with Redis CELERY_BROKER_VISIBILITY_TIMEOUT. The visibility timeout must be longer than the longest configured hard limit. Core uses late acknowledgements, then recovers retained domain intent for lost document work. These settings do not guarantee exactly-once delivery or that Redis snapshots retained the most recent notification. The delivery and recovery reference lists defaults, retry boundaries, retention and audit workflows.

ContextBase0.1.0

Search the documentation