Skip to content
Radioso Docs home
Open app

Evaluate answer quality

An eval is a real conversation you froze so you can replay it and check the agent still answers the way you expect. It is how you catch a regression before a visitor does — after you add documents, retune retrieval, or edit a directive. You manage evals from the Eval section in the dashboard.

The Eval section showing 1 of 1 case passing, a Run all button, and one case row with a passing status and its last run and expectation count.

Cases come from real conversations

You do not write an eval case from scratch. You capture one from a conversation that already happened. In the workbench, hover an assistant answer (the agent’s reply — API paths say assistant, same thing) and click the flask icon. From Activity → Quality, choose Add to Eval. Capturing takes a snapshot: it freezes the conversation and the settings that produced the answer, so a later replay compares against a fixed starting point rather than a moving one.

The Quality action uses one idempotent endpoint. Repeating it, including from two browser tabs, opens the same case instead of creating duplicates:

text
PUT /api/v1/evals/cases/by-source-message/{assistantMessageId}
GET /api/v1/evals/cases/by-source-message/{assistantMessageId}

The response includes the linked case and immutable snapshot, so an API client does not need to scan the case list. GET only looks up an existing link; PUT returns it or creates it. The case starts without assertions; open it and add what should stay true before running it. After creation, the dashboard action changes to Open Eval and Quality shows the latest case/run status with its run time.

The lower-level endpoints remain useful when you want to choose the case name or assertions yourself:

text
POST /api/v1/evals/snapshots        { conversationId, messageId? }
POST /api/v1/evals/cases            { snapshotId, name, assertions? }

messageId picks the assistant turn to capture; omit it to snapshot the latest. The case request accepts assertions when you want the new case ready to run. The workbench’s flask action seeds an LLM-judge check from the captured answer; the Quality Add to Eval action leaves assertions for you to choose in the editor.

From review to verified fix

In Activity → Quality, resolve or dismiss a turn. You can optionally classify the decision so it appears in resolution reporting. After you change knowledge, retrieval, behavior, or platform code, run the linked Eval. Quality shows timestamped passing or failing evidence beside the turn. A pass offers Review and resolve, but a test result never closes operator work automatically.

Assertions: what a run checks

A case passes when all its assertions pass. Add them in the eval editor. Two families cover most needs — what retrieval found, and what the answer said.

Retrieval assertions:

  • retrieval_includes_document / retrieval_excludes_document — a document must, or must not, appear in the retrieved set. Names a documentId.
  • retrieval_top_k_includes_document — a document must appear within the top k results. Names a documentId and a k.
  • retrieval_document_order — the named documentIds must appear in that relative order.
  • retrieval_chunk_metadata — a retrieved chunk from documentId must carry the expected metadata fields.

Answer assertions:

  • answer_cites_document — the answer must cite documentId.
  • answer_contains / answer_does_not_contain — the answer must, or must not, match a pattern. Set matchMode to substring or regex, and caseSensitive when it matters.
  • llm_judge — a model compares the new answer to an expectedAnswer, with optional criteria describing what counts as correct. Use it when exact text is too brittle but meaning matters.

A case holds up to 20 assertions.

For the shipping case in the screenshot above, three assertions pin the behavior from different angles: answer_contains with pattern 3–5 business days pins the number, answer_cites_document pins it to the Shipping policy document, and llm_judge with the original answer as reference catches drift in meaning that exact text matching would miss.

Run a case

Running replays the snapshot against your current corpus and settings — that is the point: the frozen conversation meets today’s data.

text
POST /api/v1/evals/cases/{id}/runs   { mode, overrides? }

mode is retrieval_only — replay just the retrieval step and score retrieval assertions — or full_assistant, which produces a full answer and scores every assertion. Use retrieval_only to isolate whether a change moved retrieval before you spend a model call on the whole answer.

Run the whole suite

The Eval list header shows how many cases pass — “1 of 1 case passing” — and a Run all button. Run all replays every case with assertions and reports each case’s outcome plus the workspace pass rate. Cases with no assertions are skipped, because there is nothing to score. The suite runs case by case, so a large suite takes a moment.

Watch the pass rate the way you watch a test suite: a change that drops it tells you something regressed, and the failing cases point at where.

Common failure modes

  • A case shows nothing to score: it has no assertions. Add at least one in the editor, or capture from an answer to seed an LLM-judge check.
  • A retrieval assertion fails after an ingestion change: the document moved in or out of the result set. That is the eval doing its job — decide whether the new ranking is correct, then update the assertion or the settings.
  • An answer_contains regex never matches: check matchMode is regex and caseSensitive is set as you intend.
  • A run reflects old behavior: a run replays against current corpus and settings, but the conversation is the snapshot. Capture a new case if the conversation itself should change.