Document processing

Document processing is durable background work. It should stay separate from interactive chat so uploads have explicit status and retries while user questions stay on the live request path.

What the operator owns

Operators are responsible for three things:

• keeping the worker path healthy
• keeping source-file storage available
• making sure ingestion settings match document shape

Processing flow in practice

What users will notice

• New documents are not searchable immediately after upload.
• Updated documents may briefly show an in-progress state before the new revision replaces the old one.
• Failed processing shows up as a document state problem before it becomes a chat quality problem.
• Metadata extraction is applied asynchronously, so a document can be searchable for a short time before its extracted date tags appear.
• Enrichment failures do not fail the document. The document stays searchable without enriched metadata and records safe provenance for operators.

Important controls

• DOCUMENT_STORAGE_DRIVER selects local filesystem or GCS storage for uploaded source files.
• DOCUMENT_UPLOAD_MAX_BYTES limits accepted upload size.
• workspace ingestion settings control chunking strategy and chunk sizes.
• documentEnrichmentEnabled controls the workspace default for metadata extraction. Source overrides and reprocess overrides can force extraction on or off for narrower runs.
• WORKER_DISPATCH_DRIVER decides whether jobs stay local or are dispatched to Cloud Tasks.
• DOCUMENT_PROCESSING_JOB_LEASE_MS defines how long a claimed job stays leased before later recovery logic can reclaim it.

Reprocessing can target a single document, a single source, or the whole workspace. Source reprocessing queues eligible documents for that source only and reports queued and skipped counts.

Queue-backed deployments keep the same dispatch contract for extraction reprocesses. The Cloud Tasks or AMQP message wakes a worker by job id; the extraction override is loaded from the durable processing job row.

Retrieval eligibility

Retrieval eligibility decides whether a processed document is a search candidate. It is separate from processing status. An excluded or expired document still shows in the document list with its normal state; it is left out of retrieval until it is eligible again.

There are two controls, both on the document itself:

• Available for retrieval is a manual switch. While it is off, the document is never retrieved, whatever its expiry.
• Auto-exclude on is an optional date. The document stays retrievable through that day and is excluded once the date passes.

Turning a document back on also clears an auto-exclude date that has already passed, so a stale date cannot immediately exclude a document the operator just re-enabled. A future date is kept.

Both controls apply to every document type, including imported files and crawled pages, and take effect without re-processing. The API is PATCH /api/v1/document/{documentId} with retrievalEnabled and/or retrievalExpiresAt.

In practice, an excluded or expired document is filtered out at candidate selection, so it will not appear in assistant answers, direct retrieval search, or the MCP grounded tools.

First checks during incidents

1. Confirm the API is still accepting uploads.
2. Confirm the worker runtime is alive.
3. Confirm source-file storage is reachable.
4. Confirm embeddings are succeeding for new jobs.
5. Confirm the queue is draining rather than growing.

Local versus cloud mode

Local development is optimized for a simple single-machine workflow. The bootstrap path writes .env, starts Docker Compose, and defaults document storage to the local filesystem.

Cloud deployment changes two important things:

• uploaded files move to GCS
• request-driven processing dispatch moves to Cloud Tasks and the dedicated worker service