AgentWorks documentation

LLM Configuration & Resilience

This document outlines the system for managing LLM configurations, user-driven fallbacks, and the automated temporary LLM cascading flow for execution resilience.

1. User-Driven LLM Configuration

Status: Implementation Complete (2026-01-05)

📋 Overview

The Model Library combines provider discovery, setup status, available models, and reusable saved configurations. Workflows and Chief of Staff select models directly; saved configurations are shortcuts, not a required publishing layer. Provider authentication is stored separately in workspace provider-key config.

Key Principles:

• Simple mode: Stores a coding-agent provider profile. Current role defaults resolve from multi-llm-provider-go after every app update.
• Advanced mode: Stores direct provider/model/options/fallback selections for each role.
• Saved configurations: Optional reusable provider/model/options combinations.
• Separate Auth: Model metadata does not store secrets; provider auth is managed separately.
• Backend Defaults: Backend uses ambient credentials (AWS Bedrock) for internal operations only.
• Fallback Chain: Simple ordered fallback array configured by the user.

📁 Key Files & Locations

🔄 Workflow

User opens the unified Model Library
    ↓
Configures provider authentication when required
    ↓
Simple: selects a provider profile; Advanced: selects models directly
    ↓
Optional: saves a provider/model/options combination for reuse
    ↓
Agent execution resolves the role config plus provider auth

🏗️ Architecture

Data Flow

Types

// Optional saved configuration
interface SavedLLM extends LLMModel {
  id: string
  name: string
  options?: Record<string, unknown>  // reasoning_effort, thinking_level, etc.
}

// Agent configuration sent to backend
interface AgentLLMConfiguration {
  primary: LLMModel     // Selected directly or resolved from a provider profile
  fallbacks: LLMModel[] // Ordered fallback chain
}

⚙️ Configuration

Backend Defaults (run_server_with_logging.sh)

Note: Backend no longer creates an internalLLM at startup. Agent execution uses the selected workflow or Chief of Staff role config.

🛠️ UI Components

LLM Dropdown Display

┌─────────────────────────────────────────┐
│ OPENROUTER                              │
│   My GPT-4o Config                      │
│   gpt-4o                                │
│   📦 128k  💲$2.50/1M  🌡️ 0.7          │
│   Reasoning: medium                     │
├─────────────────────────────────────────┤
│ BEDROCK                                 │
│   Production Claude                     │
│   claude-sonnet-4.5                     │
│   📦 200k  💲$3.00/1M  🌡️ 0.0          │
└─────────────────────────────────────────┘

🔍 For LLMs: Quick Reference

Constraints:

• ✅ Workflows can select provider/model/options directly
• ✅ Simple profiles receive new provider defaults after an app update
• ✅ Saved configurations are optional reusable shortcuts
• ✅ Backend uses Bedrock (AWS credentials) for internal ops
• ❌ No hardcoded API keys in backend
• ❌ No internalLLM created at startup

Key Store Actions:

// Save current config for reuse
saveLLM(llm, name, modelName, authMethod)

// Provider/profile data comes from the backend manifest
loadProviderManifest()

Auto-refresh triggers:

• saveLLM() → calls refreshAvailableLLMs()
• deleteSavedLLM() → calls refreshAvailableLLMs()
• loadDefaults() → calls refreshAvailableLLMs()

2. Execution Model Selection

📋 Overview

Execution agents now select models from a fixed priority chain: step override, sub-agent override, tiered resolution, then workflow fallback. Retry state still matters for control flow, but it no longer changes the execution model.

Key Benefits:

• Explicit overrides: Step execution_llm wins whenever it is set
• Predictable tiering: Tiered mode resolves only when no higher-priority override exists
• Stable retries: Retry handling is separate from model selection

📁 Key Files & Locations

🔄 Flow Sequence

graph TD
    A[Execution Start] --> B{Check Conditions}
    B -->|step execution_llm set| C[Step Override]
    B -->|sub_agent_llm available| D[Sub-agent Override]
    B -->|tier resolver active| E[Tiered Resolution]
    B -->|otherwise| F[Workflow Execution LLM]
    C --> G[Run Step]
    D --> G
    E --> G
    F --> G
    G -->|retry if needed| B

Attempt Sequence

File: controller_agent_factory.go

Priority Order (checked in this sequence):

1. Step execution LLM - Used when agent_configs.execution_llm is set for the step.
2. Sub-agent override - Used when execution is running inside a sub-agent context that supplies sub_agent_llm.
3. Tiered execution resolution - Used when tiered mode is active and no step override is present.
4. Original LLM chain (preset/workflow execution LLM) - Used as the remaining fallback.

⚙️ Failure Criteria

For retry purposes

Only ExecutionStatus == "FAILED" counts as failure:

Validation Status Handling

File: controller_execution.go:46-54

Retry Decision: Uses IsSuccessCriteriaMet from validation response

• If IsSuccessCriteriaMet == true: Stop retry, step passes
• If IsSuccessCriteriaMet == false: Continue retry (regardless of status)

Retry handling: Uses ExecutionStatus == "FAILED" via isValidationFailure()

• Only ExecutionStatus == "FAILED" triggers isRetryAfterValidationFailure
• PARTIAL/INCOMPLETE/COMPLETED with unmet criteria still retry, but they do not change model selection

Special Cases:

• Loop Iterations: New loop iterations after failure (loopIterationCount > 1) trigger isRetryAfterValidationFailure.

🔄 Implementation Details

Key Logic

File: controller_execution.go

// Validation failure check
isRetryAfterValidationFailure := isValidationFailure(previousValidationResponse) &&
    (retryAttempt > 1 || (hasLoop(step) && loopIterationCount > 1))

File: controller_agent_factory.go

if stepConfig.ExecutionLLM != nil {
    // Use explicit step execution LLM
} else if subAgentLLMFromContext != nil {
    // Use sub-agent override when allowed
} else if tierResolver != nil {
    // Resolve execution tier from context and learning maturity
} else {
    // Use workflow/preset execution LLM fallback
}

Conditions

File: controller_agent_factory.go:131-236

🛠️ Common Issues & Solutions

🔍 For LLMs: Quick Reference

Constraints:

• ✅ Allowed: Set execution_llm on a step to force that execution model.
• ✅ Allowed: Let tiered mode select execution models when no step override is present.
• ✅ Allowed: Fall back to the workflow execution LLM when no higher-priority override exists.

Failure Detection:

• Only ExecutionStatus == "FAILED" triggers isRetryAfterValidationFailure
• Decision step false result also triggers isRetryAfterValidationFailure
• New loop iteration after failure (loopIterationCount > 1) triggers isRetryAfterValidationFailure
• PARTIAL/INCOMPLETE/COMPLETED with unmet criteria continue retry but do not change model selection

Priority Order:

1. Step execution_llm
2. sub_agent_llm
3. Tiered execution resolution
4. Workflow execution LLM fallback

Example Flows:

Step override present:

Attempt 1: Step execution LLM → FAILED → Continue
Attempt 2: Step execution LLM → SUCCESS → Complete

Tiered execution:

Attempt 1: Tier-selected model → FAILED → Continue
Attempt 2: Tier-selected model → SUCCESS → Complete

Decision Step False:

Decision Step: FALSE → isRetryAfterValidationFailure = true
Attempt 1: Selected execution model → SUCCESS → Complete

Loop Iteration After Failure:

Loop Iteration 1: Selected execution model → FAILED
Loop Iteration 2: Selected execution model → SUCCESS

📖 Related Documentation

• Workflow Docs - Overall execution system
• Controller Execution - Retry logic implementation
• Model Metadata - Pricing, context, capabilities
• LLM Store - State management