Evaluations & feedback

An eval set is a named group of test questions, called cases, that you score pipelines against. To create one, go to Quality → Evaluations and choose New set. Give it a Name (required) and an optional Description, then choose Create set.

Click the set to open it, then choose Add case:

• Question: required, what the evaluation asks the pipeline.
• Expected answer: optional, a reference answer to compare against.
• Expected source IDs: optional, the sources a good answer draws on. Press Enter or a comma after each one.

Add from feedback lists answers that were rated not helpful or marked unanswered and aren't resolved yet. Use fills in the question, the answer given as the expected answer, and the sources it cited, for you to correct.

Each case's ⋯ menu has Edit case and Delete, which asks you to confirm. Managing eval sets needs the Manage evaluations permission, which every built-in role except Viewer has.

An evaluation run asks the pipeline every question in the set, two at a time, then has the processing model set in Settings → General judge each answer. Without a processing model, runs fail. Each run gets four scores, from 0% to 100%:

• Faithfulness: the share of the answer's claims that the retrieved passages support.
• Relevance: how directly and completely the answer addresses the question, whether or not it's correct.
• Precision: whether the passages that ranked highest were the useful ones. With expected sources, a passage counts when it comes from one of them; otherwise the judge decides.
• Recall: the share of expected sources that were retrieved. A case without expected sources uses the share of its expected answer that the passages support instead, and a case with neither gets no recall score.

A run's scores are averages over its cases, skipping cases without that score. Runs don't appear in Request Logs, but the pipeline's and the judge's model calls use your provider accounts.

To run one, go to Quality → Evaluations and choose Run evaluation. Pick the Eval set and the Pipeline, then choose Queue run. The run uses the pipeline's latest published version, or its saved draft if it hasn't been published.

The Runs tab lists every run with its set, pipeline and version, status (Queued, Running, Completed or Failed), the four scores and when it was created. Scores fill in when the run completes. Hover over Failed to see why a run stopped. Filter by status, or search by set or pipeline name. Run a set again after changing a pipeline to compare the scores.

The Sets tab lists your eval sets with their number of cases. Each set's ⋯ menu has Edit set and Delete, which removes the set and its cases after you confirm and can't be undone.

Quality → Feedback collects the answers people rated in the Playground and the questions they marked unanswered. The figures across the top count Thumbs up, Thumbs down and Unanswered, and Satisfaction: the share of rated answers that were rated helpful.

Three tabs sort the list: Negative (thumbs down, the default), Unanswered and All. Negative and Unanswered show only items that aren't resolved yet. Each row shows the question, the answer, the pipeline, the note left with a thumbs down, and the date. Search the questions and answers, or filter by pipeline.

Each row's ⋯ menu has:

• Open conversation: reads the whole conversation in Conversations.
• Add to eval set: turns the item into a case in an existing set, or in a new one with + New set. The answer given becomes a draft Expected answer and its citations the Expected source IDs; edit them, then choose Add case.
• Resolve or Reopen: marks the item handled, or brings it back.

Adding to an eval set and resolving need the Manage evaluations permission.