Skip to main content
The Evaluations tab on a Digital Worker lets you measure how the worker handles real situations before you change it. You replay a case set (a fixed list of accounts or customer messages) against the worker, and Statisfy records what it did with each case. An AI assistant connected to Statisfy over MCP, such as Claude, then scores the results, and the tab shows its verdicts. Statisfy doesn’t score runs itself. A run never sends anything to customers and never changes the worker. Each run keeps the exact settings it used, so you can compare two runs and trace a difference back to a setting. Use evaluations when you want to:
  • Check a new worker before you set it to Active.
  • Test a change to the greeting, persona voice, guardrails, disclosure mode, or knowledge bases before you save it.
  • Track whether the worker still answers, hands off, or reaches out the way your team expects.
To open the tab, go to Digital Workers, open a worker, click Configure, and select Evaluations. The tab opens on the list of Evaluation runs.

Case sets

A case set is a named list of cases with an expected outcome for each. Every set is one of two kinds:

View case sets

On the Evaluations tab, click Case sets. The list shows each set’s Name, Kind, number of Cases, Created date, and Created by. Hover over a name to see the set’s description. Click a set to open it. The set page lists its cases:
  • Proactive sets show each case’s Account, As of date (the day the case is replayed as), Expected outcome, Rationale, and Source thread (when the thread it came from happened).
  • Reactive sets show each case’s From, Message, Expected outcome, Reference reply (the reply the author would accept), and Source thread.
Click a case to see everything its author recorded. A reactive case also shows any Context it carries about the sender. To run the set, click Run this set.

Create a set case by case

  1. On the Case sets page, click New set.
  2. Enter a Name (for example, Quiet accounts, 45+ days).
  3. Choose Proactive or Reactive, and add an optional Description. After you add the first case, the kind is locked.
  4. Fill in the case and click Add case:
    • Proactive: choose an Account (required), an optional Contact the worker would write to, the Expected outcome (Reach out or Stay quiet), and an optional Rationale (for example, quiet 60 days, renewal in March).
    • Reactive: enter the customer’s Message (required), an optional From name and Account, the Expected outcome (Answer or Hand off), and an optional Reference reply.
  5. Repeat for each case. Use the edit and remove buttons on a case card to change the list.
  6. Click Create set. Statisfy opens the new set.
A set needs a name and at least one case, and holds at most 200 cases. A set can’t be changed after it’s created: to change one, write a new set.

Create a set with an assistant

An AI assistant connected to Statisfy over MCP can build a set from your workspace’s own threads and accounts. The worker is then measured on work it actually sees.
You need an assistant, such as Claude, connected to the Statisfy MCP server under your own user, with Make changes in Statisfy allowed. See Connect to Claude.
  1. On the Case sets page, click Write with an assistant.
  2. Under Connect an assistant, check that your connection is listed. If no assistant is connected for you yet, copy the server URL shown, add Statisfy as a connector in your assistant, connect, and come back.
  3. Under Give it the brief, choose Proactive or Reactive.
  4. Click Copy brief and paste it into your assistant. Copy brief stays disabled until a connection is listed.
  5. Click Done. The new set appears in the Case sets list once the assistant has created it.
The brief tells the assistant which worker to target and how to pick 15 to 20 good cases:
  • Proactive: a mix of accounts the worker covers that warrant a check-in (quiet for 45 days or more, a renewal within 120 days, an open issue) and accounts that should be left alone.
  • Reactive: real customer questions from the last 90 days, each paired with the reply your team actually sent, mixing questions that were answered with ones that were handed off.
The assistant creates the whole set, with every case, in one call and then reports the set’s name and case count.

Start a run

  1. On the Evaluations tab, click Run evaluation. You can also click Run this set on a set page to open the form with that set selected.
  2. Choose Proactive or Reactive, then pick a Case set.
  3. Read the readiness check below the set. Ready confirms how many accounts or cases will run against the worker. Nothing is sent. A set with no cases can’t be picked.
  4. Optionally enter a Label (for example, Warmer opening). Runs are listed by label, so say what you changed.
  5. Optionally change settings under Settings for this run (see below).
  6. Click Start run. Statisfy queues the run and opens it. If something blocks the run, the readiness check changes to Can’t run yet and lists why. For example, a proactive run needs the worker to have proactive tasks that apply to the set’s accounts.

Override settings for one run

Settings for this run lets you try a change and see what the worker would do with it, without saving it to the worker. You can override five settings:
  • Greeting
  • Persona voice
  • Guardrail instructions
  • AI disclosure mode
  • Knowledge bases
Each field opens on the worker’s saved value. When you edit a field, its label shows Changed (or Cleared if you emptied it). Click View changes to compare the saved value with the run’s value. Click Reset to saved to discard the edit. Fields you don’t change run with the worker’s saved values. The worker’s saved settings are never written.

Read results

Run statuses

The runs list shows each run’s label (or Untitled run), Set, Kind, Size, Result, Mean score, Started, Took, and Started by. The Result column follows the run’s status: Each case is a full worker session, so a run takes a few minutes per case. You can close the page; the list updates on its own.

Run summary

Open a run to see summary tiles, the list of cases, and the settings the run used (not the worker’s current settings).
  • Proactive runs: Would contact, Left alone, Good enough to send (passed judged drafts), and Errors.
  • Reactive runs: Passed, Mean score (out of 5), Handed off (hand-offs and loop-ins), and Errors.
Until the run is scored, a proactive run shows Judged in place of Good enough to send, and the Passed tile shows a dash.

Judge verdicts

Once an assistant scores the run, each case shows its verdict. A run can be scored by more than one judge.
  • Passed or Failed, with a Score from 1 to 5 and the judge’s Reason.
  • Not judged when nobody has scored the case yet. A case that errored shows a dash.
When more than one judge ruled on a run, use the judge picker to show All judges or a single judge’s verdicts. Click a case to see its expectation, what the worker did, and every verdict.

Outcomes

The outcome is what the worker did with a case, before any judgement. For Reactive runs: For Proactive runs, the outcome is Drafted (the worker would reach out, and the draft is shown), Left alone, or Error.

Configuring a Digital Worker

Set the persona, knowledge, tools, and autonomy a run tests.

Statisfy MCP Server

Connect an assistant to Statisfy to write case sets and score runs.