AgentWorks

AgentWorks documentation

Evaluation System

This doc describes how workflow evaluation works now.

The current system is file-backed and execution-oriented:

What Evaluation Is For

Evaluation is the workflow's goal-measurement layer for completed execution runs.

It answers:

It is separate from:

Division of labor: Pulse owns "did it run right"; evals own "did it achieve the goal." Eval steps should map one-to-one to success criteria and never duplicate operational checks — see the evaluation-plan guidance template for the authoring contract. Pulse reads the eval report each run and maps verdicts onto the per-criterion goal card in builder/improve.html, which is the durable goal signal (there is no separate numeric metrics layer).

Current Mental Model

Use this mental model:

  1. Build or edit the eval plan in evaluation/evaluation_plan.json.
  2. Point evaluation at a completed execution run folder such as iteration-8/production.
  3. Evaluation steps execute in an internal eval sandbox.
  4. Those eval steps read the original execution artifacts through {{TARGET_RUN_PATH}}.
  5. A scoring phase produces evaluation_report.json.
  6. The report is published back under the requested target run folder path inside evaluation/runs/.

Eval Plan Files

The core evaluation files are:

Frontend eval mode loads these files directly. The eval plan is not embedded in the main workflow manifest structure.

Eval steps can be scoped to the route or artifact they apply to:

{
  "id": "eval-bid-submission",
  "title": "Evaluate Bid Submission",
  "description": "Verify bid submission artifacts...",
  "applies_to_routes": [
    { "routing_step_id": "workflow-mode-router", "route_ids": ["route-bid"] }
  ]
}

When this field is present, the runtime checks the target run's routing-evaluation.json before launching the eval step. Non-applicable checks are skipped and recorded in the final report with max_score: 0, so a run is not penalized for route paths it did not take.

Current frontend behavior is implemented in useEvaluationPlanData.ts.

Eval Mode In The UI

The UI still has a separate eval mode.

Current behavior:

So evaluation is still a first-class workspace view, not just a hidden background feature.

Running Evaluation

The runtime entry point is ExecuteEvaluationOnly(...).

Current behavior:

This is implemented in evaluation_execution.go.

Internal Eval Sandbox

One of the biggest architecture changes is the eval run folder model.

Evaluation does not primarily execute inside:

Instead it executes inside an internal sandbox:

The target run folder is still important, but it is the thing being evaluated, not the main place eval steps execute.

This is why the code uses:

TARGET_RUN_PATH

TARGET_RUN_PATH is the main bridge between evaluation and the original workflow run.

Current behavior:

For example, the original run might be:

The eval step runs in the eval sandbox, but reads the original artifacts through:

This is the correct way to reference original execution outputs in eval steps.

Report Phase

After eval steps finish, the runtime builds a single evaluation_report.json covering every eval step — directly in Go (runEvaluationReportPhase in evaluation_execution.go). There is no scoring agent, no combined LLM scoring call, and no learn_code scoring fast path (all removed); each eval step's own structured output is its verdict.

Report shape

The on-disk report contains:

output_content is the source of truth for each eval step's verdict: the step should emit its own score, max_score, reasoning, and evidence (plus any domain-specific judgment dimensions) as structured output, enforced by the step's validation schema. Consumers — Pulse triage, the goal card in builder/improve.html, the scheduled improve loop, the UI — read the per-step verdicts; nothing numeric is aggregated downstream.

Where the report lands

Whichever path produces it, the runtime writes the report to:

The publish step is what makes evaluation reports line up with the execution run the user asked about.

Auto-Evaluation

Evaluation can also run automatically after workflow execution.

Current behavior:

This behavior is implemented in:

Evaluation Costs

The old mental model of evaluation writing a standalone evaluation/runs/.../token_usage.json as the main source of truth is no longer the right model.

Current cost architecture:

So:

For the full cost architecture, see cost_and_log_measurement.md.

Evaluation Reports API

The evaluation report UI is driven by /api/workflow/evaluation-reports.

Current behavior:

This is implemented in workflow.go.

Learnings And Step Config In Eval Mode

Evaluation mode has its own step config but shares the learnings namespace with execution steps.

Current behavior:

Eval steps therefore reuse workflow learnings (e.g. a _global SKILL.md written by execution steps is visible to eval steps), but keep an independent step_config.json.

Validation Of The Eval Plan

The system includes validate_evaluation_plan for checking the file after edits.

Current validation checks include:

This is implemented in evaluation_helpers.go.

What Changed From Older Docs

The older evaluation doc described a different architecture in a few places.

Current corrections:

Practical Summary

Use this mental model:

Related Docs