Evaluations

A case is one YAML document that drives a workflow through one session and asserts what happened. A workflow’s cases are its suite. An evaluation is the record of running one case or the whole suite in an environment.

Cases are files of the workflow, at workflows/<slug>/tests/*.test.yaml (or *.test.yml). That path and the document format are the archmax harness’s own, so the harness’s testing guide is the reference for every key and assertion. This page covers how the platform lists, edits and runs them.

A case

A case names its start conditions (trigger, variables, workspace), the tool results it mocks (mocks), and a flat list of steps: actions (send, decide, deliver) and the assertions that check the view after the nearest action above them. This one, from the harness’s bundled example, enters through a trigger, makes every read of the order data fail, and checks that the agent ends the session as a failure instead of inventing a report:

workflows/order-lookup/tests/orders-unavailable.test.yaml

# Every read of the order records is mocked to fail the way a store outage would.
# The report cannot be built without inventing data, so the session has to end with
# archmax_raise, and it must not write a report it could not have compiled.
title: Order store down - the report session raises
description: >-
  Every read of the order records fails, so the agent ends the report session with
  archmax_raise instead of fabricating a report or completing it.
trigger:
  id: report_requested
variables:
  company: "Acme Corp"
mocks:
  - tool: read_file
    whenInput:
      file_path: "/orders\\.json/"
    result: >-
      Error: skills/order-data/assets/orders.json could not be read: the order
      store is unavailable (ECONNREFUSED).
steps:
  - send: "A delayed-orders report was requested."
  - raised: true
  - triggerArrival: report_requested
  - notCalledTool:
      name: write_file
      input:
        file_path: "/report\\.json/"
  - grade:
      closedQA: >-
        ends the session as a failure because the order records could not be read,
        without inventing any order data
      atLeast: 0.7

A grade step has a model score the turn against a yes/no criterion; a score below atLeast fails the step and the case. It needs the suite’s grading model enabled in workflow.yaml:

tests:
  maxConcurrency: 1
  caseTimeoutMs: 120000
  judge: {}

A grade is the case’s bar. A grading rubric is something else: the standard a state’s exit is measured against inside every session.

The Evaluations view

Open a workflow and switch to Evaluations. In editor mode the rail lists the suite under Cases, each with the verdict of its latest case session, beside the workflow’s graph. Selecting a case opens Case settings, where everything about it is editable: title, description, Trigger (picked from the entry points the workflow declares), Variables, the Steps (drag to reorder), Tool mocks and File seeds. Edits save automatically and are draft changes of the environment, published with the next Publish. A case the design agent writes or removes appears or disappears in the rail without a reload.

• Add case asks for a title and a description and writes a valid case with a first step. For an event-triggered workflow it seeds triggers with one entry and opens with exactly what a real firing carrying that entry would send.
• Fill from a past session… rewrites the case’s trigger and variables from a session the workflow ran: its locked variables become the seed, and the ones the agent set itself are offered as a variables assertion instead, since a seed would hide the agent’s own write.
• A case is deleted from its row’s context menu in the rail. Its file can be read, not edited, in the Files page’s Agent view.

A case file the harness refuses still lists, carrying the parser’s message, so a broken case is visible. The design agent’s validate_workflow reports the same problems.

Running an evaluation

Run case runs the selected case; Run all cases runs the suite. Either writes any edit still waiting to be saved first, then starts the evaluation in the background and switches the view to its sessions mode. One evaluation of a workflow runs at a time, and Stop ends it after the current case.

Each case runs as a session of one turn with the reserved evaluation trigger and the testing origin. It runs in the environment you started it in, against that environment’s draft: the case and its fixtures are read from the draft, and the agent is assembled as a real turn would be, with the environment’s model, its connection tools, its mounts and its prompts. A case drives the trigger its trigger names; without one, the evaluation uses manual, else chat, else the first trigger the workflow declares. Cases run one after another, and a failing case does not stop the rest.

Case sessions spend like every other session and count toward the agent’s, the environment’s and the workflow’s spend caps. A reached cap refuses a new evaluation; one crossed mid-run lets the case in flight finish, fails it, and stops the evaluation, so the cases after it never start.

Verdicts

A case’s verdict is the harness’s: Passed, Failed or Skipped, or Stopped when the evaluation was stopped while it ran. An evaluation fails when any of its cases failed. Each step carries its own outcome, Passed, Failed, Skipped or Did not run, with the assertion’s failure detail beside a failed one. A failed structural assertion such as succeeded or reachedState halts the case, and the steps after it did not run; a content assertion such as reply or grade lets the case carry on (see which assertions halt a case). Every assertion is a hard failure: there is no tier that records a miss without failing.

Sessions mode lists the workflow’s case sessions, newest first, and no other sessions. A suite’s case sessions are grouped under a Suite header that says how many of its cases failed. Selecting a case session lights the states it visited on the graph; Summary opens its Case result and Case report (the verdict and every step’s outcome), and selecting a state shows what the agent did there. Run this case again re-runs the case behind the selected session.

Mocks, and what an unmocked case touches

Caution

A case calls the environment’s real connection tools unless it mocks them. An unmocked microsoft-outlook__reply-email sends real mail, from the connection’s real account, every time the case runs. Mock every tool with a side effect.

A mock intercepts the calls to one tool, all of them or only those whose input matches whenInput partially, and settles them with result. It intercepts the agent’s calls and the calls a script makes alike:

mocks:
  - tool: microsoft-outlook__reply-email
    whenInput: { to: "/@acmecorp\\.com/" }
    result: { "id": "mocked-reply" }

A mocked call is still checked against the tool’s input schema, so a call whose arguments the real tool would refuse fails as it would in production. A succeeded: true step fails when any tool call of the turn failed, even one the agent recovered from (a call governance refused does not count): when a case fails there, look for a failed tool call in the case session’s states before suspecting the workflow. See mocking tool results.