On this page
Run and recover background work¶
Follow the operation, then inspect its deliveries¶
Read GET /api/v1/spaces/{space_id}/document-operations/{operation_id}
with space read access before retrying. Match the stored state to the
worker and Beat logs. Beat can recover the same operation after a lost delivery.
Celery PENDING may mean its result backend has no final record.
PostgreSQL
Core records the accepted document operation.
Redis
The broker delivers a task to its configured queue.
Celery worker
A worker consumes the delivery and processes the operation.
Application status
Core reports the operation’s stored state and result.
The document lifecycle recovery entry runs each minute when Beat is running and a worker listens on its configured queue. A preparation claim is considered lost after five minutes without a heartbeat. Wait time also includes schedule cadence, queue backlog and the remaining work. A responsive worker process establishes neither Beat scheduling nor a completed domain operation.
For paid evaluations, preserve existing run IDs when diagnosing an
interruption. Pending admission has a default one-hour deadline
(CELERY_EVALUATION_PENDING_TIMEOUT_SECONDS). Expiry applies to work
that did not start, and does not stop a running evaluation or launch a
replacement. Retry an expired request deliberately in
evaluation.
Split the Quickstart worker pool¶
The default Quickstart uses one worker pool. The optional worker overlay separates standard work and document lifecycle work. From the checkout, supply the same Compose project name and configuration overrides used at install.
docker compose --project-directory . -f compose/docker-compose.quickstart.yml \
-f compose/docker-compose.quickstart.workers.yml up --build -d --wait \
--scale queue-worker=2
With these defaults there is one worker replica subscribed to celery
and two queue-worker replicas subscribed to documents. Each
replica runs one prefork child in this profile. The base Core worker has a
different default concurrency. Inspect the running command and replies
before resizing.
docker compose --project-directory . -f compose/docker-compose.quickstart.yml \
-f compose/docker-compose.quickstart.workers.yml \
exec -T worker celery -A server inspect active_queues
Verify that every queue used by producers has a live consumer. Producer
routing for document preparation uses CELERY_DOCUMENT_PREPARATION_QUEUE.
The overlay’s subscription settings
CONTEXTBASE_DEFAULT_WORKER_QUEUES and
CONTEXTBASE_QUEUE_WORKER_QUEUES accept comma-separated names. A
subscription does not change where a producer sends work. To retire or
rename a queue, pause producers including Beat and drain broker backlog
and active, reserved and scheduled deliveries before stopping its last
consumer. Use a warm worker shutdown and an adequate termination budget.
Run a dedicated worker from source¶
Set CELERY_DOCUMENT_PREPARATION_QUEUE=documents on the API, Beat and
workers. Leave the ordinary worker on its existing queues. From the
repository root, start the document worker.
export CELERY_DOCUMENT_PREPARATION_QUEUE=documents
poetry run celery -A server worker --pool=prefork --concurrency=4 -Q documents
Core defaults to celery for document tasks. The Quickstart worker overlay
sets documents. Check PostgreSQL capacity before adding prefork processes.
Workers prepare documents in parallel. A writer permit serializes snapshot, staging and publication within each Space.
Setting |
Default (seconds) |
Effect |
|---|---|---|
|
30 |
Wait for the writer permit. |
|
0.2 |
Recheck the permit. SQL connection timeouts bound each check. |
|
1 |
Retry a deferred operation that is still pending in SQL. |
Periodic recovery can pick up an accepted operation after a broker failure.
Scale the bottleneck¶
Measure completed tasks, queue age, PostgreSQL backend waits and worker
memory with the providers and storage you run in production. A prefork child
may load its own model weights and hold a
database connection. In the source production overlay,
GRANIAN_WORKERS controls the number of HTTP processes and therefore
the maximum number of copies of a loaded model. The packaged Quickstart
starts one HTTP process per container through its Compose command.
An external embedding endpoint can centralize dense embedding. Other
configured local pipeline steps still load their own models.
Review the task soft and hard limits together with Redis
CELERY_BROKER_VISIBILITY_TIMEOUT. The visibility timeout must be
longer than the longest configured hard limit. Core uses late
acknowledgements, then recovers retained domain intent for lost
document work. These settings do not guarantee exactly-once delivery
or that Redis snapshots retained the most recent notification. The
delivery and recovery reference lists
defaults, retry boundaries, retention and audit workflows.