AgentWorks documentation

Cost And Log Measurement

This doc explains how workflow costs and logs are measured now.

It is not the same as workflow_monitoring.md.

• workflow_monitoring.md is about the user-facing observability surfaces
• this doc is about the storage and measurement architecture underneath those surfaces

Two Different Systems

There are two separate data planes:

1. Cost measurement

• driven by token_usage events from LLM calls
• persisted into the workflow costs/ ledger
• aggregated by run folder, phase, step, and model

2. Workflow logs

• written explicitly by workflow controllers as files under runs/.../logs/
• store execution results, conversations, validation results, learning traces, orchestration traces, and markers

Costs are not reconstructed from log files. Logs are not reconstructed from cost files.

Cost Measurement Source Of Truth

The source of truth for workflow cost measurement is the token_usage event stream.

Current flow:

1. An agent emits a token_usage event.
2. The context-aware bridge intercepts that event.
3. It extracts:
   • provider
   • model ID
   • prompt/input tokens
   • completion/output tokens
   • cache read and cache write tokens
   • reasoning tokens
   • LLM call count
4. It persists those values directly to the costs/ ledger.

This happens in context_aware_bridge.go.

What Gets Counted

The cost ledger tracks per-model usage with pricing fields:

• input tokens
• output tokens
• cache tokens
• cache read tokens
• cache write tokens
• reasoning tokens
• LLM call count
• input cost
• output cost
• reasoning cost
• cache read cost
• cache write cost
• total cost
• context window usage

The main types are defined in:

• base_orchestrator_types.go
• cost_storage.go

Important nuance:

• cache reads and cache writes are tracked separately
• pricing is normalized or recomputed from model pricing helpers when files are read or merged
• the ledger stores both per-model totals and per-step-per-model totals

Cost Storage Layout

The current storage layout is under costs/.

Phase-only costs

Used for phase agents such as planning and builder-style workflow-phase sessions.

Files:

• costs/phase/token_usage.json
• costs/phase/daily/YYYY-MM-DD.json

These are written by PersistPhaseTokenUsage(...) in base_orchestrator_tokens.go.

Builder workflow-phase sessions also write phase cost data directly in server.go.

Execution run costs

Used for normal workflow execution runs.

Files:

• costs/execution/<group-or-__ungrouped__>/YYYY-MM-DD.json

Each daily file stores a map of run folders:

• run_folders[runFolder] = TokenUsageFile

So one daily bucket can contain multiple runs for the same group on the same UTC date.

Evaluation run costs

Used for evaluation execution.

Files:

• costs/evaluation/<group-or-__ungrouped__>/YYYY-MM-DD.json

Evaluation uses the same ledger shape as execution, but under the evaluation scope.

Aggregation Shape

Run-level cost data is aggregated into two main maps:

• by_model
• by_step_and_model

by_model:

• total usage across the whole run for each model

by_step_and_model:

• nested usage keyed by step and model
• current key format is usually phase:stepID
• older fallback format may use phase:stepIndex

This is what lets the UI render:

• stage totals
• step-wise totals
• model-wise totals
• per-run totals

Legacy Compatibility

Older layouts still exist in some workspaces:

• runs/<runFolder>/token_usage.json
• evaluation/runs/<runFolder>/token_usage.json
• workspace-root token_usage.json for phase costs

Current behavior:

• these are migrated into the costs/ ledger when read or written
• migration is handled automatically by the cost storage helpers

This logic is in cost_storage.go and token_usage_store.go.

How /api/workflow/costs Works

The workflow cost API does not read a single per-run JSON file anymore.

Current behavior:

• reads costs/phase/token_usage.json
• reads costs/phase/daily/*
• reads all execution daily ledgers under costs/execution/*
• reads all evaluation daily ledgers under costs/evaluation/*
• merges run-folder totals across daily files
• returns execution and evaluation cost data side by side for each run folder

This happens in:

• workflow.go
• cost_storage.go

Step Token Usage Events

step_token_usage is a derived summary event, not the primary source of truth.

Current behavior:

• the system reads the persisted cost ledger for the current run
• aggregates the requested step across all models
• emits a summary event for UI display

This means:

• token_usage events drive persistence
• step_token_usage events summarize persisted step totals

See base_orchestrator_tokens.go.

Workflow Log Source Of Truth

Workflow logs are explicit artifact files written by workflow controllers.

They are not reconstructed from the database event stream.

The main log families are:

• execution result JSON
• execution conversation JSON
• execution prompts JSON
• validation JSON
• pre-validation JSON
• learning execution JSONL
• learning conversation JSON
• conditional evaluation JSON
• orchestration execution JSONL
• todo-task execution JSONL

Log File Layout

For a normal run folder, logs live under:

• runs/{runFolder}/logs/...

Common files:

• logs/{step-folder}/execution/execution-attempt-{N}-iteration-{M}.json
• logs/{step-folder}/execution/execution-attempt-{N}-iteration-{M}-conversation.json
• logs/{step-folder}/execution/execution-attempt-{N}-iteration-{M}-prompts.json
• logs/{step-folder}/validation.json
• logs/{step-folder}/validation-{N}.json
• logs/{step-folder}/pre_validation.json
• logs/{step-folder}/learning-execution.json
• logs/{step-folder}/learning-conversation.json
• logs/{step-folder}/conditional-evaluation.json
• logs/{step-folder}/orchestration-execution.json
• logs/{step-folder}/todo-task-execution.json

There are also builder/session logs under:

• builder/conversation/YYYY-MM-DD/session-{sessionId}-conversation.json

Who Writes The Logs

Main writers:

• execution result and conversation logs: controller_execution.go
• learning logs: controller_learning.go
• pre-validation logs: pre_validation.go
• workflow log API scanner: workflow.go

Archived Logs

When a step is re-executed, older log files can be archived instead of being discarded.

Current behavior:

• selected log files are moved into archived/{timestamp} under the step log folder
• this preserves older attempts for debugging while keeping the active log folder readable

This is handled in controller_progress.go.

How /api/workflow/logs Works

The workflow log API scans the run folder and reconstructs a structured view for the UI.

Current behavior:

• scans runs/{runFolder}/logs/
• scans runs/{runFolder}/execution/
• maps step folders back to plan metadata from planning/plan.json
• parses known file families into typed response sections
• exposes conversation file paths for lazy loading

This is why the execution log popup can show:

• per-step execution attempts
• validation history
• learning traces
• conditional results
• orchestration and todo-task traces

The implementation is in workflow.go.

Practical Summary

Use this mental model:

• costs come from token_usage events and live in costs/
• logs come from controller-written files and live in runs/.../logs/
• step_token_usage is a derived UI summary from persisted costs
• /api/workflow/costs merges the cost ledger
• /api/workflow/logs scans and structures the run artifacts

Related Docs

• workflow_monitoring.md
• iteration_run_folder_architecture.md
• evaluation_system.md