Document processing

Document processing is durable background work. It should stay separate from interactive chat so uploads have explicit status and retries while user questions stay on the live request path.

What the operator owns

Operators are responsible for three things:

• keeping the worker path healthy
• keeping source-file storage available
• making sure ingestion settings match document shape

Processing flow in practice

What users will notice

• New documents are not searchable immediately after upload.
• Updated documents may briefly show an in-progress state before the new revision replaces the old one.
• Failed processing shows up as a document state problem before it becomes a chat quality problem.
• Metadata extraction is applied asynchronously, so a document can be searchable for a short time before its extracted date tags appear.
• Enrichment failures do not fail the document. The document stays searchable without enriched metadata and records safe provenance for operators.

Important controls

• DOCUMENT_STORAGE_DRIVER selects local filesystem or GCS storage for uploaded source files.
• DOCUMENT_UPLOAD_MAX_BYTES limits accepted upload size.
• workspace ingestion settings control chunking strategy and chunk sizes.
• documentEnrichmentEnabled controls the workspace default for metadata extraction. Source overrides and reprocess overrides can force extraction on or off for narrower runs.
• The document type catalog in Knowledge → Ingestion decides what extraction classifies against and which fields it pulls. Edits apply to processing that happens after the save; Reprocess source applies them to documents a source already ingested. Extract document metadata walks through authoring a type and using its fields.
• documentMetadata on a document source holds the tags stamped onto every chunk that source produces. A document’s own metadata wins on any key the two share, and the tags land on existing chunks the next time those documents process — which is what a source reprocess is for.
• WORKER_DISPATCH_DRIVER decides whether jobs stay local or are dispatched to Cloud Tasks.
• WORKER_TASK_AUTH_TOKEN authenticates every Cloud Tasks and recovery request at the worker application boundary; use the same value on dispatchers and workers and preserve X-Radioso-Worker-Token through any proxy.
• DOCUMENT_PROCESSING_JOB_LEASE_MS defines how long a claimed job stays leased before later recovery logic can reclaim it.

Reprocessing can target a single document, a single source, or the whole workspace. Source reprocessing queues eligible documents for that source only and reports queued and skipped counts.

Queue-backed deployments keep the same dispatch contract for extraction reprocesses. The Cloud Tasks or AMQP message wakes a worker by job id; the extraction override is loaded from the durable processing job row.

Retrieval eligibility

Retrieval eligibility decides whether a processed document is a search candidate. It is separate from processing status. An excluded or expired document still shows in the document list with its normal state; it is left out of retrieval until it is eligible again.

There are two controls, both on the document itself:

• Available for retrieval is a manual switch. While it is off, the document is never retrieved, whatever its expiry.
• Auto-exclude on is an optional date. The document stays retrievable through that day and is excluded once the date passes.

Turning a document back on also clears an auto-exclude date that has already passed, so a stale date cannot immediately exclude a document the operator just re-enabled. A future date is kept.

Both controls apply to every document type, including imported files and crawled pages, and take effect without re-processing. The API is PATCH /api/v1/document/{documentId} with retrievalEnabled and/or retrievalExpiresAt.

In practice, an excluded or expired document is filtered out at candidate selection, so it will not appear in agent answers or direct retrieval search.

Embedding coverage

A processed chunk is only a search candidate once it has an embedding filed under the workspace’s current model. Settings → Providers shows the count on the Embeddings row — “14,989 of 19,318 chunks indexed” — and names the two states that stop it moving: jobs that failed keep their place in the queue and will not retry on their own, and a workspace with no embedding model set produces no indexing work at all.

GET /api/v1/settings/ingestion/embedding-coverage returns the same counts for scripting. Eligible chunks are the ones retrieval would serve, so a document that is excluded, expired, or still processing is left out of both the covered and the total figure.

First checks during incidents

1. Confirm the API is still accepting uploads.
2. Confirm the worker runtime is alive.
3. Confirm source-file storage is reachable.
4. Confirm embeddings are succeeding for new jobs.
5. Confirm embedding coverage is climbing rather than parked with failed jobs.
6. Confirm the queue is draining rather than growing.

Local versus cloud mode

Local development is optimized for a simple single-machine workflow. The bootstrap path writes .env, starts Docker Compose, and defaults document storage to the local filesystem.

Cloud deployment changes two important things:

• uploaded files move to GCS
• request-driven processing dispatch moves to Cloud Tasks and the dedicated worker service

Read next

• Extract document metadata — author document types, choose where extraction runs, and read its provenance.
• Document upload — what an upload does and what to verify afterwards.
• Document processing lifecycle — the job model, chunk metadata merge, and tag ownership underneath these controls.