Self-hosting operations

Self-hosted Radioso has two durable data surfaces:

• PostgreSQL, including accounts, workspaces, settings, conversations, audit events, documents, chunks, vectors, and worker queues
• uploaded source-file storage, either the local document-storage volume or the configured object-storage bucket

Back up both surfaces together. A database backup without source files can leave document records whose original uploads cannot be reprocessed. Source files without the database are not enough to restore accounts, settings, embeddings, or chat history.

Redis or Valkey used for live dashboard updates is transient. Restore PostgreSQL and source files, then let the realtime runtime rebuild subscriptions and admission leases as browsers reconnect.

Minimum operating baseline

For a small private deployment, the minimum baseline is:

• run the backend, frontend, document worker, crawler worker, PostgreSQL, and document storage as separate services or containers
• put the frontend and backend behind HTTPS
• keep the default PostgreSQL host publication on 127.0.0.1; use the private container network or a private database network for service access
• give the backend and HTTP task workers the same WORKER_TASK_AUTH_TOKEN, and preserve X-Radioso-Worker-Token through any reverse proxy
• keep .env and secret values outside source control
• back up PostgreSQL and document storage every day
• test restore before relying on the backup
• deploy a tagged release and pin the image to it, rather than tracking a branch
• apply upgrades manually after reading the migration list

Live dashboard updates are optional. An enabled deployment also runs Redis or Valkey 7+ and the dedicated realtime process described below.

Backup

For Docker Compose deployments, back up:

• the PostgreSQL database with pg_dump or a physical volume snapshot
• the radioso_document_storage volume when DOCUMENT_STORAGE_DRIVER=local
• the deployed .env or equivalent secret inventory, stored in a secure password manager or secrets system

For cloud deployments, back up:

• Cloud SQL or the managed PostgreSQL database
• the GCS bucket or object-storage bucket used by DOCUMENT_STORAGE_BUCKET
• Secret Manager values or the external secret inventory

Keep database and source-file backups from the same time window. That makes document reprocessing predictable after restore.

Restore

Upgrade

Use this order for self-hosted upgrades:

1. Read the release notes for the version you are moving to. They list every SQL migration the release adds.
2. Back up PostgreSQL and document storage.
3. Build or pull the new backend, frontend, worker, and optional realtime images.
4. Start the backend and let migrations run.
5. Start the document worker and crawler worker.
6. If enabled, start Redis or Valkey and the realtime process and confirm /health/ready succeeds.
7. Start or refresh the frontend.
8. Upload a small document and confirm it becomes searchable.
9. Ask one grounded question and verify citations or retrieval trace data.
10. curl <backend_url>/health and confirm version matches the release you meant to deploy.

/health reports the build it is running:

{ "status": "ok", "version": "1.4.0", "commit": "5434e0eb6f2c1d..." }

version is the release the image was built from. A build made outside a release reads as <last release>+<short commit>, and a build with no deploy stamp at all reads as development — so a service can never claim a release it does not carry. When you run more than one stack, curl each of them: this is the fastest way to see that one is behind another.

Do not auto-pull main into production without a backup. A failed application upgrade is usually reversible; a partially applied schema or mismatched storage restore is harder to recover.

The backend runs SQL migrations during startup before it opens the HTTP port. The document worker and crawler worker only check for pending migrations. Start the backend first, then start workers after the backend has completed migrations and /health responds.

A rolling deploy or a multi-replica start can bring up two backend instances at once. Before touching schema_migrations, each instance takes a session-level PostgreSQL advisory lock, so exactly one instance applies the pending migrations while the rest wait their turn.

Migration startup metadata checks have their own timeout controls:

• DB_MIGRATION_LOCK_TIMEOUT_MS
• DB_MIGRATION_STATEMENT_TIMEOUT_MS

Keep these values shorter than your platform startup-probe window. That way a blocked migration metadata check appears as a clear application log instead of a silent port-listen timeout. A waiting instance that exceeds DB_MIGRATION_LOCK_TIMEOUT_MS exits with a log line naming the migration lock instead of hanging past the probe; the platform’s restart then finds the migrations already applied by the instance that held the lock and starts cleanly. Large migration SQL bodies, such as index builds or backfills, are not capped by these local metadata timeouts.

Live dashboard updates

The default REALTIME_MODE=disabled needs no Redis service and leaves dashboard freshness to polling. To enable live updates for a small deployment, use the same configuration on the backend, workers, and realtime process:

REALTIME_MODE=standalone
REALTIME_REDIS_URL=redis://redis:6379
REALTIME_ROLLOUT_MODE=default-on

Build the backend and run pnpm --dir backend run start:realtime. Configure the frontend with the realtime process’s private base URL:

REALTIME_INTERNAL_URL=http://realtime:8080

Keep the realtime process on the private service network. The browser opens GET /backend/api/v1/events on the frontend, and the frontend forwards it to the realtime process’s /api/v1/events route. Preserve the request’s cookie and x-workspace-id headers when placing another proxy in front of the frontend.

This stream accepts the dashboard session cookie rather than personal or service API credentials. It is used by the dashboard itself and is not an SDK or MCP event subscription API.

For rollback, set REALTIME_MODE=disabled and REALTIME_ROLLOUT_MODE=disabled across the application runtimes, then stop the realtime process and Redis. Visible dashboard queries continue through their polling fallback.

Migration startup incidents

If a new backend revision fails before /health is reachable, check the backend logs first. A migration metadata lock or statement timeout should name startup migrations as the failing phase.

Look for database migration lock held by another instance; waiting — a normal line during a rolling deploy, logged once by the instance that lost the race for the session-level migration lock. If that instance’s log ends with Timed out waiting for the database migration lock held by another instance (DB_MIGRATION_LOCK_TIMEOUT_MS), the instance holding the lock took longer than DB_MIGRATION_LOCK_TIMEOUT_MS to finish, or never released it. Check that instance’s own logs for the migration it was running when it stalled.

If the worker starts but the backend does not, that is a useful signal. Workers use a read-only pending-migration check, while the backend is the runtime that applies SQL migrations.

To inspect blocking sessions in PostgreSQL, use a privileged database console and check locks around the migration metadata table and the session-level migration advisory lock:

SELECT a.pid, a.state, a.wait_event_type, l.mode, l.granted, left(a.query, 140) AS query
FROM pg_locks l
JOIN pg_class c ON c.oid = l.relation
JOIN pg_stat_activity a ON a.pid = l.pid
WHERE c.relname = 'schema_migrations';
 
SELECT a.pid, a.state, l.granted, left(a.query, 140) AS query
FROM pg_locks l
JOIN pg_stat_activity a ON a.pid = l.pid
WHERE l.locktype = 'advisory';
 
SELECT pid, pg_blocking_pids(pid) AS blocked_by, state, left(query, 140) AS query
FROM pg_stat_activity
WHERE cardinality(pg_blocking_pids(pid)) > 0;

If a stale session is clearly blocking a rollout, terminate only that backend session:

SELECT pg_terminate_backend(<pid>);

Use this carefully. Confirm the session is stale or belongs to the failed rollout before terminating it. Restarting the database is the broader fallback when session-level recovery is not clear.

What the worker runs

The document worker is where work that outlives a request runs, so run it wherever you run the backend. Besides processing uploaded documents, it:

• sends a routine’s queued actions: contact requests, handoff notifications, approval requests
• runs background jobs, such as the one each answered turn writes to refresh the conversation’s rolling summary and title
• drains the email channel’s inbound mail and review turns, when a provider is configured
• extracts message facets for Audience Pulse
• removes data past its retention window: old Ray conversations, test executions, Slack event ids, and dead-lettered background jobs
• deletes the stored files of a deleted workspace or organization, so the worker’s storage credentials need to list and delete objects (roles/storage.objectAdmin on the GCS bucket)
• deletes at ElevenLabs the call recordings of deleted conversations, visitors, workspaces and organizations, so the worker needs outbound HTTPS to the ElevenLabs API and the same CONNECTOR_ENCRYPTION_KEY as the backend

A deployment without a running worker still answers chat, but documents stay queued, routine actions are never sent, conversation summaries and titles stop updating, and a deleted workspace’s files and deleted calls’ recordings wait until a worker runs.

Background jobs live in the background_jobs table. A completed job is deleted, so the table holds only work that is waiting, running, or has failed for good. A failed job is retried with a growing delay. After its last attempt it stays for 14 days with its kind, key, attempt count, and the class of the error that stopped it, such as BackgroundJobTimeoutError or AppError [provider_unavailable]. Its payload is dropped when it fails for good, and neither the table nor your error tracking receives the error’s message, because a payload or a message can carry conversation text:

SELECT kind, idempotency_key, status, attempts, available_at, last_error
FROM background_jobs
ORDER BY created_at DESC
LIMIT 20;

Call recordings left at ElevenLabs

A deletion writes each call recording to delete into voice_recording_erasure_items, beside an encrypted copy of the workspace’s ElevenLabs key in voice_recording_erasures, and the voice.recording_erasure job deletes them, retrying each recording on its own for about a day (Deleting conversation data). A recording ElevenLabs kept refusing, or one still pending a week after its deletion, stays there with status failed, a reason and failed_at, and its key copy is gone by then.

Each batch of recordings given up on is reported to your error tracking as voice.recording_left_at_vendor, with the erasure, organization and workspace ids, the vendor, a count and the reasons, never a recording id or a key. A live organization’s owners and admins see their own in GET /api/v1/account/voice-recordings/left-at-vendor. Once the organization is deleted, you are the only ones who can see them; list them, newest first, to delete each in ElevenLabs or to tell the organization:

SELECT e.account_id, e.workspace_id, i.conversation_id, i.vendor_region,
       i.vendor_conversation_id, i.failure_reason, i.attempts, i.failed_at
FROM voice_recording_erasure_items i
JOIN voice_recording_erasures e ON e.id = i.erasure_id
WHERE i.status = 'failed'
ORDER BY i.failed_at DESC;

The organization and workspace ids stay after both are deleted. A row you have dealt with can be deleted; the backlog keeps a failed recording until someone does. Rows with status pending are still being worked through, each due again at its next_attempt_at. The voice-recording-erasure-recovery periodic task (on Cloud Run, the *-voice-recording-erasure-recovery Cloud Scheduler job, every 15 minutes) queues the job again for any erasure whose pending recordings are more than 15 minutes overdue. If they still grow old, check that the worker runs and look for voice.recording_erasure in background_jobs.

Stored files of deleted workspaces

The worker deletes a workspace’s files from the store this deployment is configured with: the DOCUMENT_STORAGE_BUCKET bucket, or the DOCUMENT_STORAGE_LOCAL_PATH folder. Files a workspace left in a bucket the deployment used before stay there. Give each deployment its own bucket or folder. Two deployments sharing one store cannot tell each other’s workspaces from deleted ones, and the cleanup below would delete the other deployment’s files.

Storage can still hold files whose workspace this database no longer has, such as those of workspaces deleted on an earlier release. The workspaceStorageSweep command finds them and, once you confirm the count, queues one purge per workspace: the same file deletion a workspace deletion does. It runs only when you run it, and it needs what the worker has: the backend environment (DATABASE_URL, DOCUMENT_STORAGE_DRIVER, DOCUMENT_STORAGE_BUCKET or DOCUMENT_STORAGE_LOCAL_PATH, and the rest of the worker’s settings), a connection to the database, and a storage identity that may list objects.

• Container deployments run it in the backend image, with the worker’s environment and identity: node ./dist/src/workspaceStorageSweep.js.
• Cloud Run, as the Terraform under infra/terraform sets it up, runs it as a one-off Cloud Run job on the backend image. Give the job the worker’s service account, which holds roles/storage.objectAdmin on the documents bucket, the worker’s environment and secrets, and the backend’s VPC connector, because Cloud SQL has only a private address. Pass the arguments below as the job’s arguments.
• From a checkout with the deployment’s environment loaded, pnpm --dir backend run storage:sweep-orphans takes the same arguments.

The command exits when it is done, so a job run finishes with it.

node ./dist/src/workspaceStorageSweep.js

Document storage check: verified: the store holds files of this database's workspaces
Workspace prefixes scanned: 212
Orphaned workspaces: 9 (1730 objects)
Kept, workspace exists in this database: 203
Kept, an object written in the last 24 hours: 0
Kept, not a canonical workspace id: 0
Dry run: nothing was queued. To queue these purges, run again with --apply --expect-orphans 9.

SELECT event_type, count(*) FROM audit_events
WHERE event_type IN ('workspace.deleted', 'account.delete') AND event_status = 'success'
GROUP BY event_type;

gcloud storage buckets describe gs://<bucket> --format="value(soft_delete_policy.retentionDurationSeconds)"

node ./dist/src/workspaceStorageSweep.js --apply --expect-orphans 9

A purge that failed for good stays in background_jobs for 14 days as a dead documents.workspace_storage_purge job, keyed by what queued it and the workspace id: workspace_deletion:<id>, orphan_sweep:<id> or operator:<id>. The workspace.deleted audit event carries the same id as deletedWorkspaceId. Queue it again for that one workspace, dry run first:

node ./dist/src/workspaceStorageSweep.js --workspace <workspace id>
node ./dist/src/workspaceStorageSweep.js --workspace <workspace id> --apply

The command refuses a workspace that exists in this database. A purge you queue this way deletes whatever the files’ age, and records a workspace.stored_files.purge_queued audit event.

Worker incidents

If uploads succeed but new documents do not become searchable, treat it as a worker-path incident first.

Check in this order:

1. The backend can accept uploads and write document records.
2. The document worker is running and can connect to PostgreSQL.
3. Source-file storage is reachable from the worker.
4. The embedding provider credentials and model settings are valid.
5. The queue is draining rather than growing.
6. Failed jobs have enough logs to identify parser, storage, embedding, or timeout failures.

For local polling deployments, WORKER_DISPATCH_DRIVER=noop means the worker claims durable jobs from PostgreSQL. For queue-backed deployments, also check Cloud Tasks or AMQP delivery and retry configuration.

The document worker also drives background work that has nothing to do with documents: the machine-access credential expiry scan, and any task an application module registers for a recurring sweep, such as the Enterprise usage-limit alert and auto top-up dispatchers. If the worker is down, these sweeps pause along with document processing, even though chat and the dashboard keep answering requests normally. A cloud deployment with no continuously running worker process drives the same sweeps on a schedule, pushed to the worker task service instead.

Secrets

Rotate these values deliberately and keep historical impact in mind:

• SESSION_COOKIE_SECRET affects browser sessions.
• RADIOSO_MCP_SIGNING_SECRET keys standalone MCP source budgets and source proofs. A single in-memory MCP process can omit it and uses a process-random key that resets budgets on restart. Give the identical value to the backend and standalone MCP whenever the backend must trust source proofs, and always when Redis is configured; changing it starts a new budget window for every source. A backend without standalone MCP derives its source-budget key from SESSION_COOKIE_SECRET, so rotating that value has the same budget-window effect.
• WORKSPACE_TOKEN_SECRET derives the per-agent signing keys used for signed visitor identity.
• PUBLIC_CHAT_SESSION_SECRET affects anonymous public-chat sessions and website embeds.
• CONNECTOR_ENCRYPTION_KEY encrypts stored credentials: workspace provider API keys, connector secrets, OAuth tokens, external MCP credentials, and webhook signing secrets. The value must be 32 random bytes, base64-encoded — generate one with openssl rand -base64 32. The bootstrap command generates a key automatically when .env does not already have one. Replace it with the procedure in Rotate the credential encryption key: swapping the value in place leaves every stored credential unreadable.
• RADIOSO_EDGE_PROOF_SECRET lets the frontend’s server-side proxies vouch for a visitor’s IP address and geo headers when they forward public chat, embed session, and dashboard /backend requests to the backend. At least 32 characters, shared verbatim by both services. Leave it unset and the proof path never runs: the backend still serves every request, but it records that visitor’s IP, country, region, and city as unknown instead of trusting an unsigned header, and it budgets relayed requests by a proxy’s address rather than the visitor’s. docker-compose.yml and docker-compose.dev.yml already set a matching dev value on both services, so the local stack exercises the signed path with no extra setup.
• provider API keys affect chat, rewrite, rerank, and embedding calls.

Changing encryption or token secrets without a migration or rotation plan can invalidate existing sessions, tokens, or saved connector credentials.

Rotate the credential encryption key

Every stored credential is encrypted under CONNECTOR_ENCRYPTION_KEY, and the backend also decrypts with each key in CONNECTOR_ENCRYPTION_PREVIOUS_KEYS. That lets you replace the key, after a suspected leak or on a schedule, without anyone re-entering a credential: the old key keeps decrypting while the credentialReencryption command moves each stored value onto the new one.

Reports name keys by key id, 16 hex characters derived from the key. A key always has the same id, and the id is no help in recovering the key, so the command prints the id of each configured key instead of the key. It finds the key that wrote each value by decrypting it.

Start only when every backend and worker instance runs a release that reads CONNECTOR_ENCRYPTION_PREVIOUS_KEYS. An instance without it decrypts with CONNECTOR_ENCRYPTION_KEY alone, so it can’t read credentials written under the new key.

The command runs only when you run it, with the backend’s environment: DATABASE_URL, CONNECTOR_ENCRYPTION_KEY, CONNECTOR_ENCRYPTION_PREVIOUS_KEYS, and the rest of the backend’s settings.

• Container deployments run it in the backend image: node ./dist/src/credentialReencryption.js.
• Cloud Run, as the Terraform under infra/terraform sets it up, runs it as a one-off Cloud Run job on the backend image, with the backend’s environment and secrets and the backend’s VPC connector, because Cloud SQL has only a private address. Pass the arguments below as the job’s arguments. That Terraform has no input for CONNECTOR_ENCRYPTION_PREVIOUS_KEYS, so give it to the services and the job the same way as CONNECTOR_ENCRYPTION_KEY.
• From a checkout with the deployment’s environment loaded, pnpm --dir backend run credentials:reencrypt takes the same arguments.

The order below keeps every instance able to read every credential at each step, including while a deploy rolls out.

openssl rand -base64 32

node ./dist/src/credentialReencryption.js

Current key: 08ae134562f05a24. Previous keys: ccecbc25ba3365c2.
connector_configs.config_data
  previous key ccecbc25ba3365c2: 2 to re-encrypt
integration_oauth_connections.credential_ciphertext
  current key 08ae134562f05a24: 1
  previous key ccecbc25ba3365c2: 3 to re-encrypt
integration_oauth_connections.oauth_client_ciphertext
  previous key ccecbc25ba3365c2: 4 to re-encrypt
integration_oauth_connections.oauth_flow_ciphertext
  no encrypted values
mcp_connections.credential_ciphertext
  previous key ccecbc25ba3365c2: 1 to re-encrypt
mcp_connections.oauth_client_ciphertext
  no encrypted values
mcp_connections.oauth_flow_ciphertext
  no encrypted values
workspace_provider_credentials.ciphertext
  previous key ccecbc25ba3365c2: 7 to re-encrypt
voice_recording_erasures.sealed_api_key
  no encrypted values
workspace_webhook_destinations.secret_ciphertext
  previous key ccecbc25ba3365c2: 2 to re-encrypt
To re-encrypt: 19. Unreadable: 0.
Dry run: nothing was changed. Run again with --apply to re-encrypt these values under the current key.

node ./dist/src/credentialReencryption.js --apply

To re-encrypt: 19. Unreadable: 0.
Re-encrypted 19 values under the current key 08ae134562f05a24.

The command exits 1 while any stored credential column holds a value it can’t decrypt. It counts those values per column on unreadable lines and leaves them as they are. Connector settings are the exception: a connector secret that no configured key opens looks like a plain setting, so the command neither counts it nor fails on it, and the connector’s dashboard page shows it as [re-enter secret]. Usually the key that wrote an unreadable value was removed from CONNECTOR_ENCRYPTION_PREVIOUS_KEYS too early: put it back and run the command again. A value that no key you hold decrypts has to be re-entered in its workspace.

Database TLS

The Compose stack connects to Postgres without TLS, over the private Docker network. To encrypt that connection, set PGSSLMODE=no-verify on the backend and worker, or put sslmode=verify-full&sslrootcert=<path> in DATABASE_URL to also check the certificate. Encryption covers both and the certificate errors to expect.

Visitor request facts

Two more settings shape what a conversation records about the request that started it.

RADIOSO_TRUSTED_PROXY_HOPS turns a forwarded-for chain into one client address. Set it to the number of proxies in front of the backend that add the caller’s address to the end of X-Forwarded-For — nginx with $proxy_add_x_forwarded_for, Caddy’s reverse_proxy, or a cloud load balancer — and the backend reads the entry the outermost of them added as the client. Anything to the left of that entry is whatever the caller sent. With no such proxy in front, keep 0 and the backend uses the socket address. The bundled frontend relays requests to the backend as they arrived, so it adds nothing to the count; its signed edge facts carry the chain it received, and the backend resolves that chain with the same value. It is the backend’s only hop count, so it decides every rate-limit source budget and the IP address on sign-up and password-reset audit records, and the standalone MCP server reads it for its own budgets. Counting wrong reads the wrong entry as the client: one too many trusts an address the caller wrote, one too few lands on a proxy.

That one value reads two chains: the one the backend receives directly and the one the frontend signs into its edge facts. So the frontend and the backend need to sit behind the same number of proxies that add the caller’s address. Say Caddy fronts the API while the frontend answers on :3000 with nothing in front of it: a visitor who reaches :3000 writes every entry of the chain the frontend signs, and with a hop count of 1 they pick their own rate-limit budget and the IP address their conversation records. Put both services behind the same proxy, or make the frontend’s port reachable only through it.

Country, region, and city need no setting of their own. Radioso reads whichever geo header the deployment’s load balancer, CDN, or reverse proxy already stamps on the request: Cloudflare’s CF-IPCountry, Vercel’s x-vercel-ip-country family, and Google App Engine’s X-AppEngine-Country all work unmodified. Any other proxy can rename its own header to x-client-region or x-client-city to be picked up the same way.

Health checks

At minimum, monitor:

• backend /health
• frontend reachability
• worker process liveness
• realtime /health/live and /health/ready when enabled
• stream connection counts, rejected admissions, reconnect rates, and Redis transport loss when realtime is enabled
• PostgreSQL connection errors
• document-processing failures
• embedding provider failures
• disk or bucket storage errors
• HTTP 429 and 5xx rates for public chat and website embeds

Read next

• Deployment — the production contract, required secrets, and rollout checklist this runbook assumes.
• Enterprise usage limits — cap indexed storage, monthly content, and monthly answers per account on Enterprise deployments.