> ## Documentation Index
> Fetch the complete documentation index at: https://help.statisfy.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluating a Digital Worker

> Build case sets, replay them against a Digital Worker with optional settings changes, and read the verdicts on the Evaluations tab.

The **Evaluations** tab on a Digital Worker lets you measure how the worker handles real situations before you change it. You replay a **case set** (a fixed list of accounts or customer messages) against the worker, and Statisfy records what it did with each case. An AI assistant connected to Statisfy over MCP, such as Claude, then scores the results, and the tab shows its verdicts. Statisfy doesn't score runs itself.

A run never sends anything to customers and never changes the worker. Each run keeps the exact settings it used, so you can compare two runs and trace a difference back to a setting.

Use evaluations when you want to:

* Check a new worker before you set it to **Active**.
* Test a change to the greeting, persona voice, guardrails, disclosure mode, or knowledge bases before you save it.
* Track whether the worker still answers, hands off, or reaches out the way your team expects.

To open the tab, go to **Digital Workers**, open a worker, click **Configure**, and select **Evaluations**. The tab opens on the list of **Evaluation runs**.

## Case sets

A case set is a named list of cases with an expected outcome for each. Every set is one of two kinds:

| Kind | What it replays | Expected outcome per case |
| - | - | - |
| **Proactive** | Outreach the worker starts on its own. Each case is an account. | **Reach out** or **Stay quiet** |
| **Reactive** | Replies to what customers send. Each case is a customer message. | **Answer** or **Hand off** |

### View case sets

On the **Evaluations** tab, click **Case sets**. The list shows each set's **Name**, **Kind**, number of **Cases**, **Created** date, and **Created by**. Hover over a name to see the set's description.

Click a set to open it. The set page lists its cases:

* **Proactive sets** show each case's **Account**, **As of** date (the day the case is replayed as), **Expected** outcome, **Rationale**, and **Source thread** (when the thread it came from happened).
* **Reactive sets** show each case's **From**, **Message**, **Expected** outcome, **Reference reply** (the reply the author would accept), and **Source thread**.

Click a case to see everything its author recorded. A reactive case also shows any **Context** it carries about the sender. To run the set, click **Run this set**.

### Create a set case by case

1. On the **Case sets** page, click **New set**.
2. Enter a **Name** (for example, *Quiet accounts, 45+ days*).
3. Choose **Proactive** or **Reactive**, and add an optional **Description**. After you add the first case, the kind is locked.
4. Fill in the case and click **Add case**:
   * **Proactive**: choose an **Account** (required), an optional **Contact** the worker would write to, the **Expected** outcome (**Reach out** or **Stay quiet**), and an optional **Rationale** (for example, *quiet 60 days, renewal in March*).
   * **Reactive**: enter the customer's **Message** (required), an optional **From** name and **Account**, the **Expected** outcome (**Answer** or **Hand off**), and an optional **Reference reply**.
5. Repeat for each case. Use the edit and remove buttons on a case card to change the list.
6. Click **Create set**. Statisfy opens the new set.

A set needs a name and at least one case, and holds at most 200 cases. A set can't be changed after it's created: to change one, write a new set.

### Create a set with an assistant

An AI assistant connected to Statisfy over MCP can build a set from your workspace's own threads and accounts. The worker is then measured on work it actually sees.

> [!NOTE]
>
> You need an assistant, such as Claude, connected to the Statisfy MCP server under your own user, with **Make changes in Statisfy** allowed. See [Connect to Claude](https://help.statisfy.com/mcp/connecting.md).

1. On the **Case sets** page, click **Write with an assistant**.
2. Under **Connect an assistant**, check that your connection is listed. If no assistant is connected for you yet, copy the server URL shown, add Statisfy as a connector in your assistant, connect, and come back.
3. Under **Give it the brief**, choose **Proactive** or **Reactive**.
4. Click **Copy brief** and paste it into your assistant. **Copy brief** stays disabled until a connection is listed.
5. Click **Done**. The new set appears in the **Case sets** list once the assistant has created it.

The brief tells the assistant which worker to target and how to pick 15 to 20 good cases:

* **Proactive**: a mix of accounts the worker covers that warrant a check-in (quiet for 45 days or more, a renewal within 120 days, an open issue) and accounts that should be left alone.
* **Reactive**: real customer questions from the last 90 days, each paired with the reply your team actually sent, mixing questions that were answered with ones that were handed off.

The assistant creates the whole set, with every case, in one call and then reports the set's name and case count.

## Start a run

1. On the **Evaluations** tab, click **Run evaluation**. You can also click **Run this set** on a set page to open the form with that set selected.
2. Choose **Proactive** or **Reactive**, then pick a **Case set**.
3. Read the readiness check below the set. **Ready** confirms how many accounts or cases will run against the worker. Nothing is sent. A set with no cases can't be picked.
4. Optionally enter a **Label** (for example, *Warmer opening*). Runs are listed by label, so say what you changed.
5. Optionally change settings under **Settings for this run** (see below).
6. Click **Start run**. Statisfy queues the run and opens it. If something blocks the run, the readiness check changes to **Can't run yet** and lists why. For example, a proactive run needs the worker to have [proactive tasks](https://help.statisfy.com/digital-workers/configuring_a_worker.md#proactive-work) that apply to the set's accounts.

### Override settings for one run

**Settings for this run** lets you try a change and see what the worker would do with it, without saving it to the worker. You can override five settings:

* **Greeting**
* **Persona voice**
* **Guardrail instructions**
* **AI disclosure mode**
* **Knowledge bases**

Each field opens on the worker's saved value. When you edit a field, its label shows **Changed** (or **Cleared** if you emptied it). Click **View changes** to compare the saved value with the run's value. Click **Reset to saved** to discard the edit.

Fields you don't change run with the worker's saved values. The worker's saved settings are never written.

## Read results

### Run statuses

The runs list shows each run's label (or **Untitled run**), **Set**, **Kind**, **Size**, **Result**, **Mean score**, **Started**, **Took**, and **Started by**. The **Result** column follows the run's status:

| Status | What it means |
| - | - |
| **Queued** | Waiting for a runner to pick it up. |
| **Running** | Working through the set one case at a time, shown as *Running · 3 of 5*. |
| **Done** | Finished. Shows *4 of 10 passed* once scored, or *10 recorded · awaiting verdicts* until an assistant scores it. |
| **Failed** | The run stopped. Open it to see the reason. |

Each case is a full worker session, so a run takes a few minutes per case. You can close the page; the list updates on its own.

### Run summary

Open a run to see summary tiles, the list of cases, and the settings the run used (not the worker's current settings).

* **Proactive runs**: **Would contact**, **Left alone**, **Good enough to send** (passed judged drafts), and **Errors**.
* **Reactive runs**: **Passed**, **Mean score** (out of 5), **Handed off** (hand-offs and loop-ins), and **Errors**.

Until the run is scored, a proactive run shows **Judged** in place of **Good enough to send**, and the **Passed** tile shows a dash.

### Judge verdicts

Once an assistant scores the run, each case shows its verdict. A run can be scored by more than one judge.

* **Passed** or **Failed**, with a **Score** from 1 to 5 and the judge's **Reason**.
* **Not judged** when nobody has scored the case yet. A case that errored shows a dash.

When more than one judge ruled on a run, use the judge picker to show **All judges** or a single judge's verdicts. Click a case to see its expectation, what the worker did, and every verdict.

### Outcomes

The outcome is what the worker did with a case, before any judgement.

For **Reactive** runs:

| Outcome | What it means |
| - | - |
| **Answered** | The worker wrote a reply. |
| **Answered, looped in** | The worker replied and handed the rest to a person on your team. The case shows the to-do, any **Draft follow-up**, and the guardrail's flag on it. Counted under **Handed off**. |
| **Handed off** | The worker deferred the message to a human. The case shows the defer reason. |
| **Held by guardrail** | A guardrail stopped the reply. |
| **No reply** | The worker chose not to reply. |
| **Error** | The case failed to run. The case shows the error. |

For **Proactive** runs, the outcome is **Drafted** (the worker would reach out, and the draft is shown), **Left alone**, or **Error**.

## Related

> [card] [Configuring a Digital Worker](https://help.statisfy.com/digital-workers/configuring_a_worker.md)
>
> Set the persona, knowledge, tools, and autonomy a run tests.

> [card] [Statisfy MCP Server](https://help.statisfy.com/mcp/overview.md)
>
> Connect an assistant to Statisfy to write case sets and score runs.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.