Testing workers is part of the Statisfy MCP server. Connect Statisfy in Claude first: see Connect to Claude. Your workspace needs AI CSM, and you need the Admin role.
Safe to run on your live workspace
A test run is a rehearsal. The worker reads your real accounts, contacts, and knowledge bases, so its answers reflect your workspace, but:- Nothing is sent. No email, chat reply, ticket, or task leaves the run. Everything the worker would have done is recorded instead.
- The worker doesn’t change. A run uses the worker’s saved settings. To try a change, ask Claude to run the test with different ones: they are recorded against that run only, and the live worker keeps working as before.
- Customers see nothing. The situations are copies of real messages. Nobody is contacted.
How it works
1
Build a test set
A test set is a list of situations, each with what the worker should do. Ask Claude to build one from your real conversations, emails, or support tickets. Claude reads them, picks a spread of questions, and writes what a good outcome looks like for each: answer it, or hand it to a person.Each test keeps a link to the thread it came from, so you can check any result against what really happened. A set holds up to 200 tests.
2
Run the worker
Claude starts a run. The worker handles every situation in a full session of its own, as it would live, so a run takes several minutes or more. Claude checks on it and tells you when it’s done.
3
Read what it did
For each test, Claude reads the worker’s reply, whether it handed off to a person and why, and what it looked up to get there, such as which knowledge-base articles it searched.
4
Score each test
Claude judges each result against what the worker should have done: was the outcome right, was the answer accurate and backed by what the worker found, did it link the right article, did it ask a clarifying question only when it needed one. The scores are saved with the run, so you can come back to them.
5
Change and compare
Ask Claude to run the same set again with a change and compare the two runs case by case. A run can try different guidelines, voice, greeting, how the worker says it’s an AI, or knowledge bases. Changes to its tools, skills, or channels can’t be tried this way. When you’re happy, change the worker itself.
Two kinds of test
Prompts to try
Replace the names with your workers and accounts.Build a test set
Build a test set
- “Build a test set for our Digital CSM from last month’s support tickets: 50 real questions, a mix of how-to questions it should answer from the help center and problems it should hand to the team.”
- “Add the questions customers asked our digital worker in chat this week to a test set, with what it should have done for each.”
- “Write an outreach test set for our Digital CSM from ten at-risk accounts as they looked on the first of the month.”
Run and read the results
Run and read the results
- “Run the help-center test set against our Digital CSM and tell me when it’s done.”
- “Show me every test where the worker handed off a question it should have answered.”
- “For the tests it got wrong, show the customer’s question, the worker’s reply, and what it searched.”
Score and compare
Score and compare
- “Score each result: right outcome, accurate, backed by what it found, links the article. Give me the pass rate by type of question.”
- “Run the same set again with these guidelines instead, and compare the two runs.”
- “Which tests passed in the first run but failed in the second?”
What you’re allowed to do
- Admins only. Building test sets, starting runs, and scoring need the Admin role, like changing a worker.
- Read-only connections can read. If you connected without Make changes, Claude can read existing test sets, runs, and scores, but can’t build a set or start a run.
- Changes are recorded. Test sets, runs, and scores Claude creates are in Settings → Audit logs, under your name, with MCP as the source.
Troubleshooting
A run stays queued
A run stays queued
Runs wait their turn with other background work in your workspace, such as indexing knowledge-base documents. Ask Claude to check again in a few minutes. A queued run is never lost.
A test shows an error instead of a result
A test shows an error instead of a result
The worker’s session for that test couldn’t run, so there’s nothing to judge. It’s counted apart from the tests the worker got wrong. To try it again, start a new run: a finished run isn’t resumed.
Claude can't start an outreach run
Claude can't start an outreach run
An outreach test needs the worker to have proactive tasks that apply to the accounts in the set. Give the worker a task first, or test its replies instead.
Claude says it can't build a set or start a run
Claude says it can't build a set or start a run
- You may have connected without Make changes. Disconnect Statisfy in Claude and connect again, with the box ticked.
- Testing workers needs the Admin role. Check with your Statisfy admin.