AgentWorks documentation

Human Feedback System

📋 Overview

The human feedback system is an interactive virtual tool that pauses LLM execution to request real-time input from users, enabling human-in-the-loop workflows. It provides browser notifications, real-time UI updates, and optional Slack integration for seamless feedback collection. The system supports 2FA/OTP input, confirmations, and clarifying questions through a thread-safe request/response coordination system.

Key Benefits:

• Enables human-in-the-loop workflows for critical decisions
• Supports 2FA/OTP input, confirmations, and clarifying questions
• Real-time notifications via browser push notifications
• Optional Slack integration with smart delayed notifications (2-minute delay)
• Thread-safe request/response coordination with timeout handling
• Respond via Slack threads or UI (no UI required for Slack responses)
• Integrates seamlessly with event-driven architecture

📁 Key Files & Locations

🔄 How It Works

Core System Lifecycle

1. Tool Registration

   • CreateHumanTools() defines the human_feedback virtual tool
   • Tool requires: {"message_for_user": string, "unique_id": string (UUID)}
   • Executor: handleHumanFeedback() function

2. LLM Invokes Tool

   • LLM calls human_feedback with message and unique UUID
   • handleHumanFeedback() receives the call

3. Backend Request Creation

   • Global HumanFeedbackStore singleton creates request entry
   • Store maps unique_id → HumanFeedbackRequest struct
   • Creates channel (chan string) for response coordination
   • If Slack enabled, starts 2-minute delayed notification timer

4. Frontend Notification

   • Event system automatically notifies frontend
   • HumanFeedbackToolCallDisplay component renders
   • Browser push notification shown (if permissions granted)
   • UI displays message + text input + submit button

5. Slack Notification (Optional, Delayed)

   • If Slack enabled and user hasn't responded within 2 minutes
   • System checks if user already responded
   • If no response, sends Slack notification to configured channel
   • Message includes question, context, and unique request ID
   • Maps Slack message timestamp to unique ID for thread replies

6. User Response

   • User responds via UI (types response, clicks "Submit Feedback" or Ctrl/Cmd+Enter)
   • OR user replies in Slack thread
   • Frontend sends POST to /api/human-feedback/submit (UI path)
   • OR Slack Events API sends webhook to backend (Slack path)
   • Backend verifies Slack webhook signature and extracts unique ID from thread

7. Backend Coordination

   • Response written to channel
   • WaitForResponse() unblocks with user's response
   • handleHumanFeedback() returns response to LLM

8. LLM Continuation

   • LLM receives user response as tool output
   • Execution continues in same turn with user input

Timeout Handling

• Default timeout: 5 minutes (tool executor)
• Orchestrator helpers: 10 minutes
• On timeout: Returns error, LLM handles gracefully

Delayed Notification Logic

The Slack integration uses smart delayed notifications:

1. Immediate: Feedback request appears in UI right away
2. 2-Minute Delay: System waits 2 minutes before sending Slack notification
3. Response Check: Before sending, system checks if user already responded
4. Conditional Send: Only sends Slack notification if user hasn't responded
5. Non-Blocking: All notification logic runs asynchronously

🏗️ Architecture

sequenceDiagram
    participant LLM
    participant Tool as handleHumanFeedback
    participant Store as HumanFeedbackStore
    participant Slack as SlackService
    participant API as /api/human-feedback/submit
    participant Frontend as React UI
    participant User

    LLM->>Tool: Call human_feedback(message, unique_id)
    Tool->>Store: CreateRequestWithSlack(unique_id, message)
    Store-->>Tool: Request created
    Store->>Frontend: Event: tool_call_start
    Frontend->>User: Show notification + UI
    Store->>Store: Start 2-minute timer (async)
    Tool->>Store: WaitForResponse(unique_id, 5min)
    Note over Tool,Store: Tool blocks here

    alt User responds via UI (< 2 min)
        User->>Frontend: Type response + submit
        Frontend->>API: POST /api/human-feedback/submit
        API->>Store: SubmitResponse(unique_id, response)
        Store->>Tool: Channel receives response
        Tool-->>LLM: Return user response
    else User doesn't respond (< 2 min)
        Store->>Store: Check if user responded
        Store->>Slack: SendFeedbackNotification() (if enabled)
        Slack->>User: Slack message in channel
        User->>Slack: Reply in thread
        Slack->>Store: Extract unique_id, SubmitResponse()
        Store->>Tool: Channel receives response
        Tool-->>LLM: Return user response
    end

🧩 Example Usage

LLM Tool Call

{
  "tool_name": "human_feedback",
  "arguments": {
    "message_for_user": "Please enter the 2FA code sent to your email",
    "unique_id": "550e8400-e29b-41d4-a716-446655440000"
  }
}

Backend Tool Handler

File: human_tools.go

func handleHumanFeedback(ctx context.Context, args map[string]interface{}) (string, error) {
    messageForUser := args["message_for_user"].(string)
    uniqueID := args["unique_id"].(string)

    feedbackStore := GetHumanFeedbackStore()

    if err := feedbackStore.CreateRequestWithSlack(ctx, uniqueID, messageForUser, "", nil); err != nil {
        return "", fmt.Errorf("failed to create feedback request: %w", err)
    }

    response, err := feedbackStore.WaitForResponse(uniqueID, 5*time.Minute)
    if err != nil {
        return "", fmt.Errorf("failed to get user feedback: %w", err)
    }

    return response, nil
}

Orchestrator Helper Functions

File: base_orchestrator.go

// Request human feedback with text input
approved, feedback, err := orchestrator.RequestHumanFeedback(
    ctx,
    "approval_123",
    "Please approve this plan",
    "Additional context",
    sessionID,
    workflowID,
)

// Request yes/no feedback
approved, err := orchestrator.RequestYesNoFeedback(
    ctx,
    "yesno_456",
    "Do you want to proceed?",
    "Approve",
    "Reject",
    "",
    sessionID,
    workflowID,
)

// Request multiple choice feedback
choice, err := orchestrator.RequestMultipleChoiceFeedback(
    ctx,
    "choice_789",
    "Select deployment environment",
    []string{"Development", "Staging", "Production"},
    "",
    sessionID,
    workflowID,
)

⚙️ Configuration

Tool Parameters

Timeout Configuration

Slack Configuration

Configuration Storage:

• Stored in slack_feedback_config database table
• Managed via UI: Sidebar → Slack icon
• API endpoints: GET/POST /api/human-feedback/slack/config

Browser Notifications

Permissions:

• Requested on component mount
• Shown when Notification.permission === 'granted'
• Properties: Title "Human Feedback Required", body from message_for_user, requireInteraction: true, silent: false

🛠️ Common Issues & Solutions

🔍 For LLMs: Quick Reference

Constraints

✅ Allowed:

• Any user-facing message (questions, requests for OTP/2FA, confirmations)
• UUID generation for unique_id parameter
• Waiting synchronously for user response (blocks LLM execution)
• Multiple feedback requests in sequence (different unique_ids)

❌ Forbidden:

• Reusing same unique_id for multiple requests
• Not providing unique_id (required parameter)
• Empty or missing message_for_user
• Expecting instant response (users may take time)

Example Pattern

Simple feedback request:

{
  "tool": "human_feedback",
  "arguments": {
    "message_for_user": "Please confirm the database migration plan before I proceed",
    "unique_id": "feedback_1701234567890"
  }
}

2FA/OTP use case:

{
  "tool": "human_feedback",
  "arguments": {
    "message_for_user": "Enter the 6-digit verification code sent to your email",
    "unique_id": "otp_1701234567891"
  }
}

LLM receives:

User response: "582491"

Integration with Workflow Agents

Planning agent pattern:

// 1. human_feedback - ask user for approval
// 2. Receive yes/no response
// 3. If approved: call update_plan_steps / add_plan_steps / delete_plan_steps

Variable extraction agent pattern:

// 1. human_feedback - "I detected variable {{API_KEY}}. Should I extract it?"
// 2. User responds: "Yes" or "No"
// 3. If yes: call update_variable tool
// 4. If no: don't modify variables

🔌 API Endpoints

Submit Human Feedback

POST /api/human-feedback/submit

Request:

{
  "unique_id": "550e8400-e29b-41d4-a716-446655440000",
  "response": "User's feedback text"
}

Response:

{
  "success": true
}

Slack Configuration

GET /api/human-feedback/slack/config

Response:

{
  "enabled": true,
  "channel_id": "C1234567890",
  "bot_token": "xoxb-...1234",
  "app_token": "xapp-...5678"
}

POST /api/human-feedback/slack/config

Request:

{
  "enabled": true,
  "bot_token": "xoxb-...",
  "app_token": "xapp-...",
  "channel_id": "C1234567890"
}

POST /api/human-feedback/slack/test

Response:

{
  "success": true,
  "message": "Slack connection test successful!"
}

🔒 Security Model

Thread Safety

Global Singleton:

var (
    globalHumanFeedbackStore *HumanFeedbackStore
    humanFeedbackStoreOnce   sync.Once
)

Mutex Protection:

• sync.RWMutex protects concurrent access to request map and waiters
• Write operations (CreateRequest, SubmitResponse) use Lock()
• Read operations (WaitForResponse lookup) use RLock()

Socket Mode Security

• WebSocket connection (no public webhook required)
• App-Level Token separate from Bot Token
• Automatic reconnection handling
• Webhook signature verification for thread replies

Token Storage

• Tokens stored in database (slack_feedback_config table)
• Tokens masked in API responses (last 4 characters only)
• Recommendation: Encrypt tokens in production environments

📊 Database Schema

slack_feedback_config Table

slack_feedback_messages Table

Data Structures

type HumanFeedbackRequest struct {
    UniqueID       string
    MessageForUser string
    UserResponse   string
    IsCompleted    bool
    CreatedAt      time.Time
}

type HumanFeedbackStore struct {
    requests map[string]*HumanFeedbackRequest
    waiters  map[string]chan string
    mu       sync.RWMutex
}

🚀 Slack Setup Instructions

1. Create Slack App at https://api.slack.com/apps
2. Configure Bot Token Scopes: chat:write, channels:read, channels:history
3. Enable Socket Mode: Generate App-Level Token with connections:write scope
4. Enable Events API: Subscribe to message.channels event
5. Get Channel ID: Right-click channel → View channel details
6. Configure in UI: Sidebar → Slack icon → Enter tokens and channel ID → Test connection → Save

📖 Related Documentation

• Workflow Monitoring - Reviews runs that may require human approval
• Todo Task Step Type - Uses human feedback for plan approval and variable confirmation
• Virtual tools - See the backend tool handlers referenced above
• Event System - How events coordinate frontend/backend