AssetOpsBench

Ground Truth Creation Best Practice

Overview

This document provides best practices for creating ground truth for an utterance. It can be viewed as a trajectory of the execution process of the utterance under an ideal execution process without hallucination or execution failure. The ground truth is the cornerstone to evaluate AI agent systems irresoective it domain or application. Ground truth represents the ideal trajectory - the expected sequence of planning decisions and execution steps that a system should follow to correctly answer a user’s natural language query, i.e. utterance.

What is Ground Truth?

Ground truth is a structured specification that defines:

  1. What the AI agent system should accomplish (expected output)
  2. How the AI agent system should accomplish it (execution trajectory)
  3. Why each step is necessary (planning rationale)

It serves as the soft ground truth for automated evaluation, enabling systematic comparison of actual agent behavior against expert-defined expected behavior. “Soft” means the framework accommodates both types, providing exact answers for deterministic tasks and acceptance criteria for non-deterministic tasks. For example, in a deterministic task like “How many work orders in 2017?”, the ground truth would specify the exact number of work orders. In a non-deterministic task like “Recommend maintenance actions”, the ground truth would specify the criteria for a valid recommendation, such as “The recommended actions should address the top 3 failure modes with the highest risk scores.”

Purpose

Enable rigorous, reproducible benchmarking of AI agent systems by providing:

Use Cases

While this document uses industrial asset management examples (from AssetOpsBench), the principles apply to any domain requiring agent-based automation, including energy systems, healthcare, finance, and customer service.

Two System Paradigms

Ground truth supports evaluation of two complementary paradigms:

1. Multi-Agent Systems

2. Multi-MCP Systems

Key Insight: Both paradigms share the same ground truth structure. The difference lies in semantic interpretation:

Benchmarking Context

Ground truth enables three critical benchmarking capabilities:

1. Architectural Comparison

2. Model Selection

3. Failure Mode Discovery

Community and Reproducibility

Ground truth serves as shared infrastructure for open source and research communities:

Document Scope

This document provides systematic workflow, structure definitions, best practices, and validation guidance for creating ground truth trajectories. Based on AssetOpsBench and industry best practices for agent evaluation.

1. High-Level Ground Truth Creation Workflow

This section provides a systematic, step-by-step process for creating ground truth trajectories. Follow this workflow to ensure completeness and consistency.

1.1 Overview: The Five-Phase Process

Phase 1: Understand the Utterance
    ↓
Phase 2: Define Expected Behavior
    ↓
Phase 3: Design the Ideal Trajectory
    ↓
Phase 4: Validate and Refine
    ↓
Phase 5: Document and Review

1.2 Phase 1: Understand the Utterance

Objective: Fully comprehend what the user is asking and what constitutes success.

Key Questions:

  1. What is the user’s intent?
  2. What information or action does the user need?
  3. Which agents/servers are required?
  4. What data sources must be accessed?
  5. What constraints or conditions apply?

Actions:

Example:

Utterance: "Get the work order of equipment CWC04013 for year 2017."

Analysis:
- Intent: Retrieve historical work order records
- Information needed: Work orders for specific equipment and time period
- Required agent: WOAgent (Work Order Agent)
- Data source: Work order database
- Constraints: Equipment ID = CWC04013, Year = 2017
- Group: retrospective (historical data retrieval)
- Category: Knowledge Query
- Type: Workorder
- Entity: Equipment
- Deterministic: true (33 records expected)

1.3 Phase 2: Define Expected Behavior

Objective: Specify what the correct output should be and how to validate it.

Key Decisions:

Decision 1: Is this deterministic or non-deterministic?

Ask: “Is there exactly one correct answer?”

Decision 2: What is the characteristic form?

For deterministic utterances:

"There will be 33 records. The expected response should retrieve
all work orders for equipment CWC04013 in the year 2017, ensuring
correct equipment ID and time filtering."

For non-deterministic utterances:

"The answer should contain one or more valid work order
recommendations with rationale. The recommendations should be
relevant to addressing anomalies in equipment CWC04009."

Decision 3: What is the expected output structure?

Define final_out:

Actions:

1.4 Phase 3: Design the Ideal Trajectory

Objective: Map out the step-by-step process the agent should follow.

Step 3A: High-Level Planning

Create planning_steps: Strategic decomposition of the problem

Template:

"planning_steps": [
  {
    "agent": "AgentName",
    "instruction": "What this agent should accomplish (high-level)"
  }
]

Guidelines:

Example:

"planning_steps": [
  {
    "agent": "WOAgent",
    "instruction": "Create Equipment instance for CWC04013"
  },
  {
    "agent": "WOAgent",
    "instruction": "Create DateRange for year 2017"
  },
  {
    "agent": "WOAgent",
    "instruction": "Retrieve work orders for equipment and date range"
  }
]

Step 3B: Detailed Execution Design

Create execution_steps: Specific tool calls and operations

For each step, define:

  1. name: Unique identifier (e.g., “create_equipment”, “get_work_orders”)
  2. action: Tool/function name (e.g., “Equipment”, “get_work_orders”)
  3. agent: Which agent/server executes this
  4. arguments: Input parameters (can reference previous outputs)
  5. outputs: Variable names produced
  6. deterministic: Flags for name, action, arguments, outputs

Template:

{
  "name": "descriptive_step_name",
  "action": "tool_or_function_name",
  "agent": "AgentName",
  "arguments": {
    "param1": "value1",
    "param2": "reference_to_previous_output"
  },
  "outputs": ["output_variable_name"],
  "deterministic": {
    "name": false,
    "action": true,
    "arguments": true,
    "outputs": true
  }
}

Key Considerations:

Create execution_links: Connect steps to form DAG

Template:

"execution_links": [
  {"source": "step1_name", "target": "step2_name"},
  {"source": "step2_name", "target": "step3_name"}
]

Rules:

Validation:

1.5 Phase 4: Validate and Refine

Objective: Ensure ground truth is correct, complete, and consistent.

Validation Checklist

Structural Validation:

Semantic Validation:

Logical Validation:

Domain Validation:

Refinement Process

  1. Test with Domain Expert: Walk through trajectory with SME
  2. Simulate Execution: Mentally execute each step to verify feasibility
  3. Check Edge Cases: Consider what could go wrong
  4. Optimize Path: Remove unnecessary steps, combine where possible
  5. Update Documentation: Ensure notes explain any non-obvious decisions

1.6 Phase 5: Document and Review

Objective: Finalize documentation and prepare for use.

Documentation Requirements

Required Documentation:

  1. characteristic_form: Clear, detailed description of expected output
  2. final_out_description: Explanation of what the output represents
  3. note field: Context, rationale, and metadata

Recommended Note Content:

"note": "Source: [where this scenario came from];
         Owner: [SME who created/validated it];
         Context: [important background information];
         Design rationale: [why this approach was chosen];
         Implementation notes: [technical details];
         Omitted: [what was intentionally left out]"

Final Review

Review with Stakeholders:

Quality Gates:

1.7 Quick Reference: Creation Process Flow

START: New Utterance
    ↓
Q1: What is user's intent?
    → Identify: RETROSPECTIVE, PREDICTIVE, or PRESCRIPTIVE
    ↓
Q2: Is there exactly one correct answer?
    → YES: deterministic = true
    → NO: deterministic = false
    ↓
Q3: Which agents/servers are needed?
    → Single: type = "AgentName"
    → Multiple: type = "multiagent"
    ↓
Q4: What are the high-level steps?
    → Create planning_steps (strategic intent)
    ↓
Q5: What are the detailed operations?
    → Create execution_steps (tool calls)
    ↓
Q6: How do steps connect?
    → Create execution_links (DAG)
    ↓
Q7: What is the expected output?
    → Define final_out and characteristic_form
    ↓
Q8: Does it pass validation?
    → YES: Document and finalize
    → NO: Return to appropriate phase and refine
    ↓
DONE: Ground Truth Complete

1.8 Common Pitfalls and How to Avoid Them

Pitfall 1: Vague characteristic_form

Pitfall 2: Missing intermediate steps

Pitfall 3: Incorrect deterministic flag

Pitfall 4: Circular dependencies

Pitfall 5: Ambiguous variable references

1.9 Effort Considerations

Ground truth creation effort varies significantly based on scenario complexity:

Factors Affecting Creation Time:

Observed Patterns from AssetOpsBench:

Efficiency Strategies:

Note: Specific time estimates depend on team expertise, tooling, and domain complexity. Track your own metrics to establish baselines for your context.

2. Ground Truth Structure

2.1 Core Components

Each ground truth scenario consists of the following key components:

{
  "id": <integer>,
  "uuid": "<optional-unique-identifier>",
  "text": "<natural language utterance>",
  "type": "<agent or MCP server type>",
  "category": "<utterance category>",
  "deterministic": <boolean>,
  "characteristic_form": "<expected response description>",
  "planning_steps": [...],
  "execution_steps": [...],
  "execution_links": [...],
  "final_out": {...},
  "final_out_description": [...]
}

2.2 Field Definitions

Required Fields

  1. id (integer): Unique identifier for the scenario
    • Must be unique across the entire ground truth dataset
    • Used for tracking and referencing scenarios
  2. text (string): The natural language utterance/query
    • Should be clear, unambiguous, and representative of real user queries
    • Must align with the utterance design guidelines
  3. type (string): Primary agent or MCP server responsible
    • Examples: IoT, FMSA, TSFM, Workorder, multiagent
    • In MCP context: filesystem, database, web-search, etc.
    • Use multiagent when multiple agents must collaborate
  4. category (string): Classification of the utterance
    • Examples: Knowledge Query, Data Query, Inference Query, Decision Support, Prediction, Complex Query
    • Should align with the 8 universal categories from utterance design guideline
  5. deterministic (boolean): Whether the utterance has a single correct answer
    • true: Single verifiable correct answer (e.g., data retrieval, factual queries)
    • false: Multiple valid responses possible (e.g., recommendations, predictions)
  6. characteristic_form (string): Description of expected response format and validation criteria
    • For deterministic: Specify exact expected output or validation rules
    • For non-deterministic: Describe acceptable response characteristics
    • Include data format, file references, or structural requirements

Planning Steps

  1. planning_steps (array of objects): High-level agent-level planning
    • Each step is a dictionary with keys: agent and instruction
    • Represents the strategic decomposition of the problem
    • Must use predefined agent names from the available agent list
    • Order matters - represents logical sequence of problem-solving

Structure:

"planning_steps": [
  {
    "agent": "AgentName",
    "instruction": "High-level description of what this agent should do"
  }
]

Best Practices:

Execution Steps

  1. execution_steps (array of objects): Detailed tool-level operations
    • Each step represents a specific tool/function call
    • Forms nodes in the execution DAG

Structure:

"execution_steps": [
  {
    "name": "unique_step_identifier",
    "action": "tool_or_function_name",
    "agent": "AgentName",
    "arguments": {
      "param1": "value1",
      "param2": "reference_to_previous_output"
    },
    "outputs": ["output_variable_name"],
    "deterministic": {
      "name": false,
      "action": true/false,
      "arguments": true/false,
      "outputs": true/false
    }
  }
]

Field Details:

  1. execution_links (array of objects): Defines the DAG structure
    • Connects execution steps to form a directed acyclic graph
    • Represents dependencies and execution order

Structure:

"execution_links": [
  {
    "source": "step_name_1",
    "target": "step_name_2"
  }
]

Best Practices:

Final Output

  1. final_out (object): Structured representation of the expected final result
    • JSON object containing the actual expected output
    • Can be null if not applicable
    • Should match the format described in characteristic_form
  2. final_out_description (array of strings): Textual description of the final output
    • Human-readable explanation of what the output represents
    • Can include validation criteria, interpretation notes, or context

Optional Fields

  1. uuid (string): Globally unique identifier (optional)
    • Use when scenarios need to be tracked across systems
    • Format: UUID v4 (e.g., “efc94d35-5236-410c-9e4f-5dcdfee818cc”)
  2. expected_result (any): Legacy field for expected results (optional)
    • Can be null or omitted
    • Prefer using final_out instead
  3. data (object): Additional metadata or context (optional)
    • Can store scenario-specific data
    • Usually empty {}
  4. possible_alternatives (object): Alternative valid execution paths (optional)
    • Documents other acceptable ways to solve the problem
    • Useful for non-deterministic scenarios

3. Special Execution Steps

3.1 Finish Step

The Finish step is a special terminal step that marks task completion and provides the final answer.

Characteristics:

Usage Patterns:

Pattern 1: Direct Finish (IoT Agent)

{
  "name": "finish",
  "action": "Finish",
  "agent": "IoTAgent",
  "argument": "The assets at the MAIN site are: CQPA AHU 1, CQPA AHU 2B, Chiller 4, Chiller 6, Chiller 9, Chiller 3.",
  "deterministic": {
    "name": false,
    "action": true,
    "arguments": true,
    "outputs": true
  }
}

Pattern 2: Finish with Structured Arguments (TSFM Agent)

{
  "name": "return_final_answer",
  "action": "Finish",
  "agent": "TSFMAgent",
  "arguments": {
    "answer": true,
    "evidence_models": ["ttm_96_28", "ttm_512_96"]
  },
  "outputs": ["final_answer"],
  "deterministic": {
    "name": false,
    "action": true,
    "arguments": true,
    "outputs": true
  }
}

Pattern 3: Finish without Arguments (FMSA Agent)

{
  "name": "finish",
  "action": "Finish",
  "arguments": "",
  "outputs": ""
}

Best Practices:

3.2 Other Special Steps

Self-Ask: Agent asks itself a clarifying question

Agent-Ask: Agent requests information from another agent

4. Best Practices

4.1 Consistency and Standards

  1. Agent Naming
    • Use predefined agent names consistently
    • Common agents: IoTAgent, FMSRAgent, TSFMAgent, WOAgent
    • In MCP context: Use standard MCP server names
  2. Action Naming
    • Use consistent tool/function names across scenarios
    • Document available tools and their signatures
    • Examples: sensors, history, get_work_orders, Equipment, DateRange
  3. Variable Naming
    • Use descriptive variable names for outputs
    • Examples: equipment, date_range, events, sensor_data, forecast_results
    • Avoid generic names like result, output, data unless context is clear
  4. Step Naming
    • Use descriptive step names that indicate purpose
    • Good: create_equipment, filter_anomalies, calculate_statistics
    • Avoid: step1, step2, step3 (unless no better name exists)

4.2 Determinism Marking

When to mark as deterministic:

Example:

{
  "name": "load_work_orders",
  "action": "pickle.load",
  "deterministic": {
    "name": false,        // Step name can vary
    "action": false,      // pickle.load is not a standard action
    "arguments": false,   // File path may vary
    "outputs": false      // Content depends on file
  }
}

4.3 Multi-Agent Scenarios

For complex queries requiring multiple agents:

  1. Set type to “multiagent” 
    "type": "multiagent"
  2. Include all agents in planning_steps
    • Show agent handoffs explicitly
    • Document inter-agent communication
  3. Use Agent-Ask for coordination
    • When one agent needs information from another
    • Document the request-response pattern
  4. Consider execution order
    • Some agents may need to complete before others start
    • Use execution_links to enforce ordering

Example:

"planning_steps": [
  {
    "agent": "IoTAgent",
    "instruction": "Retrieve sensor data for the equipment"
  },
  {
    "agent": "TSFMAgent",
    "instruction": "Analyze sensor data for anomalies"
  },
  {
    "agent": "WOAgent",
    "instruction": "Recommend work orders based on detected anomalies"
  }
]

4.4 Handling Non-Deterministic Scenarios

For utterances with multiple valid answers:

  1. Set deterministic to false 
    "deterministic": false
  2. Provide representative output
    • Show one valid example in final_out
    • Note that other valid outputs exist
  3. Define acceptance criteria
    • In characteristic_form, describe what makes an answer valid
    • List required elements
    • Specify constraints
  4. Document alternatives
    • Use possible_alternatives to show other valid approaches
    • Helps evaluators understand acceptable variations

Example:

"characteristic_form": "The answer should contain one or more valid work order recommendations with rationale. The recommendations should be relevant to addressing anomalies in equipment CWC04009.",
"deterministic": false,
"final_out": {
  "recommendations": ["M005", "M006", "OP004"]
},
"final_out_description": [
  "Recommended work orders based on anomaly analysis",
  "Other valid recommendations may exist depending on prioritization criteria"
]

4.5 Cross-Validation Checks

Before finalizing ground truth, validate:

  1. Planning Steps Validation
    • All agents are from predefined list
    • Instructions are clear and semantic
    • Steps form logical sequence
  2. Execution Steps Validation
    • All step names are unique
    • All actions are valid tool/function names
    • All argument references point to existing outputs
    • Deterministic flags are correctly set
  3. Execution Links Validation
    • All source and target names exist in execution_steps
    • No cycles in the DAG
    • All steps (except first) have incoming links
    • All steps (except last) have outgoing links
  4. Output Validation
    • final_out matches characteristic_form description
    • final_out_description is clear and complete
    • For deterministic queries, exact expected output is specified
  5. Consistency Validation
    • type matches primary agent in planning/execution
    • category aligns with utterance characteristics
    • deterministic flag matches output predictability

4.6 Quality Assurance Checklist

Before submitting ground truth, verify:

Structural Completeness

Semantic Correctness

Planning Quality

Execution Quality

DAG Validity

Output Quality

Consistency

4.7 Common Pitfalls and How to Avoid Them

8.1 Inconsistent Finish Steps

Problem: Different agents use different Finish patterns

Solution: Standardize Finish usage

8.2 Ambiguous Variable References

Problem: Unclear how outputs are referenced in arguments

Solution: Use consistent reference syntax

8.3 Incomplete Determinism Marking

Problem: Deterministic flags not set at all levels

Solution: Always include complete deterministic dict

"deterministic": {
  "name": false,
  "action": true/false,
  "arguments": true/false,
  "outputs": true/false
}

8.4 Circular Dependencies

Problem: Execution links create cycles

Solution: Validate DAG structure

8.5 Missing Intermediate Steps

Problem: Jumping from data retrieval to final output without processing

Solution: Include all necessary transformations

8.6 Vague Characteristic Forms

Problem: Expected output not clearly described

Solution: Be specific and detailed

4.8 Advanced Scenarios

For complex ground truth scenarios:

Conditional Execution

For scenarios with conditional logic:

{
  "name": "conditional_step",
  "action": "if_then_else",
  "agent": "Agent",
  "arguments": {
    "condition": "anomaly_detected == true",
    "then_action": "recommend_maintenance",
    "else_action": "continue_monitoring"
  },
  "outputs": ["action_taken"],
  "deterministic": {
    "name": false,
    "action": false,
    "arguments": false,
    "outputs": false
  }
}

Parallel Execution Paths

For scenarios with parallel processing:

"execution_links": [
  {"source": "get_data", "target": "analyze_path1"},
  {"source": "get_data", "target": "analyze_path2"},
  {"source": "analyze_path1", "target": "merge_results"},
  {"source": "analyze_path2", "target": "merge_results"}
]

5. Common Patterns and Templates

This section presents real-world patterns from AssetOpsBench ground truth scenarios, demonstrating practical implementations used in industrial asset management.

5.1 Simple Knowledge Query Pattern (ID 103)

Use Case: Single-agent query for domain knowledge

Scenario: “List all failure modes of asset Wind Turbine.”

Pattern Characteristics:

{
  "planning_steps": [
    {
      "agent": "FMSRAgent",
      "instruction": "List all failure modes of asset Wind Turbine."
    }
  ],
  "execution_steps": [
    {
      "name": "get_failure_modes",
      "action": "Get Failure Modes",
      "agent": "FMSRAgent",
      "arguments": "Wind Turbine",
      "outputs": "The failure modes of asset Wind Turbine: ['Blade Failure', 'Gearbox Failure', 'Generator Failure', 'Bearing Failure', 'Electrical System Failure', 'Control System Failure', 'Foundation Failure', 'Tower Failure', 'Hub Failure', 'Pitch System Failure ', 'Yaw System Failure ', 'Hydraulic System Failure']",
      "deterministic": {
        "name": false,
        "action": true,
        "arguments": true,
        "outputs": false
      }
    },
    {
      "name": "finish",
      "action": "Finish",
      "arguments": "",
      "outputs": ""
    }
  ],
  "execution_links": [
    {"source": "get_failure_modes", "target": "finish"}
  ]
}

5.2 Multi-Step Data Processing Pattern (ID 400)

Use Case: Create objects, retrieve data, process, and generate output

Scenario: “Get the work order of equipment CWC04013 for year 2017.”

Pattern Characteristics:

{
  "planning_steps": [
    {
      "agent": "WOAgent",
      "instruction": "Create an Equipment instance using the provided equipment_id CWC04013."
    },
    {
      "agent": "WOAgent",
      "instruction": "Create a DateRange object spanning 2017-01-01 to 2017-12-31."
    },
    {
      "agent": "WOAgent",
      "instruction": "Retrieve the work orders file for the specified equipment and date range."
    },
    {
      "agent": "WOAgent",
      "instruction": "Load the work orders data from the retrieved file using pickle."
    },
    {
      "agent": "WOAgent",
      "instruction": "Generate a structured JSON file from the work orders using an LLM."
    }
  ],
  "execution_steps": [
    {
      "name": "create_equipment",
      "action": "Equipment",
      "agent": "WOAgent",
      "arguments": {"equipment_id": "CWC04013"},
      "outputs": ["equipment"],
      "deterministic": {"name": false, "action": true, "arguments": true, "outputs": true}
    },
    {
      "name": "create_date_range",
      "action": "DateRange",
      "agent": "WOAgent",
      "arguments": {"start_date": "2017-01-01", "end_date": "2017-12-31"},
      "outputs": ["date_range"],
      "deterministic": {"name": false, "action": true, "arguments": true, "outputs": true}
    },
    {
      "name": "get_work_orders_file",
      "action": "get_work_orders",
      "agent": "WOAgent",
      "arguments": {"equipment": "equipment", "date_range": "date_range"},
      "outputs": ["file"],
      "deterministic": {"name": false, "action": true, "arguments": true, "outputs": true}
    },
    {
      "name": "load_work_orders",
      "action": "pickle.load",
      "agent": "WOAgent",
      "arguments": {"file": "file"},
      "outputs": ["work_orders"],
      "deterministic": {"name": false, "action": false, "arguments": false, "outputs": false}
    },
    {
      "name": "print_work_orders",
      "action": "print",
      "agent": "WOAgent",
      "arguments": {"obj": "work_orders"},
      "deterministic": {"name": false, "action": false, "arguments": false, "outputs": false}
    },
    {
      "name": "generate_json",
      "action": "LLM.generate_json",
      "agent": "WOAgent",
      "arguments": {"data": "work_orders"},
      "outputs": ["json_file"],
      "deterministic": {"name": false, "action": false, "arguments": false, "outputs": false}
    }
  ],
  "execution_links": [
    {"source": "create_equipment", "target": "get_work_orders_file"},
    {"source": "create_date_range", "target": "get_work_orders_file"},
    {"source": "get_work_orders_file", "target": "load_work_orders"},
    {"source": "load_work_orders", "target": "print_work_orders"},
    {"source": "load_work_orders", "target": "generate_json"}
  ]
}

5.3 Multi-Agent Coordination Pattern (ID 601)

Use Case: Simple multi-agent coordination for knowledge retrieval

Scenario: “List all failure modes of asset Chiller 6 at MAIN site.”

Pattern Characteristics:

{
  "planning_steps": [
    {
      "agent": "IoTAgent",
      "instruction": "Get basic description/identifiers for Chiller 6 at MAIN site (asset verification)."
    },
    {
      "agent": "FMSRAgent",
      "instruction": "List all failure modes for the verified asset (Chiller 6 at MAIN)."
    }
  ],
  "execution_steps": [
    {
      "name": "fmsr_list",
      "agent": "FMSRAgent",
      "action": "get_failure_modes",
      "arguments": {"asset_name": "Chiller 6", "site_name": "MAIN"},
      "outputs": {
        "status": "success",
        "failure_modes": [
          "Compressor Overheating: Failed due to Normal wear, overheating",
          "Heat Exchangers: Fans: Degraded motor or worn bearing due to Normal use",
          "Evaporator Water side fouling",
          "Condenser Water side fouling",
          "Condenser Improper water side flow rate",
          "Purge Unit Excessive purge",
          "Refrigerant Operated Control Valve Failed spring"
        ]
      },
      "deterministic": {"name": false, "action": false, "arguments": false, "outputs": false}
    },
    {
      "name": "finalize",
      "agent": "FMSRAgent",
      "action": "Finish",
      "arguments": {"validation_checks": ["len(failure_modes)==7"]},
      "outputs": {"final_state": "completed"},
      "deterministic": {"name": false, "action": false, "arguments": false, "outputs": false}
    }
  ],
  "execution_links": [
    {"source": "fmsr_list", "target": "finalize"}
  ],
  "final_output": {
    "asset": "Chiller 6",
    "site": "MAIN",
    "parameter": "Failure Modes",
    "failure_modes": [
      "Compressor Overheating: Failed due to Normal wear, overheating",
      "Heat Exchangers: Fans: Degraded motor or worn bearing due to Normal use",
      "Evaporator Water side fouling",
      "Condenser Water side fouling",
      "Condenser Improper water side flow rate",
      "Purge Unit Excessive purge",
      "Refrigerant Operated Control Valve Failed spring"
    ]
  }
}

5.4 Complex Multi-Agent with Error Recovery Pattern (ID 502)

Use Case: Multi-agent workflow with retry logic and error handling

Scenario: “What is the forecast for ‘Chiller 9 Condenser Water Flow’ in the week of 2020-04-27 based on data from the MAIN site?”

Pattern Characteristics:

Key Pattern Elements:

  1. Sensor Location: IoTAgent locates the specific sensor
  2. Data Retrieval with Retry: Initial failure due to incorrect asset identifier, followed by successful retry
  3. Model Selection: TSFMAgent enumerates available models and selects appropriate checkpoint
  4. Forecasting: Run time-series forecasting with validated arguments
  5. Validation: Confirm artifacts and provide comprehensive summary
{
  "planning_steps": [
    {
      "agent": "IoTAgent",
      "instruction": "Locate 'Condenser Water Flow' sensor for 'Chiller 9' at 'MAIN'."
    },
    {
      "agent": "IoTAgent",
      "instruction": "Retrieve windowed history (2020-04-27 → 2020-05-03) and save CSV."
    },
    {
      "agent": "TSFMAgent",
      "instruction": "List available pretrained checkpoints and select 'ttm_96_28'."
    },
    {
      "agent": "TSFMAgent",
      "instruction": "Run forecast on the CSV and save JSON to declared path."
    },
    {
      "agent": "TSFMAgent",
      "instruction": "Validate artifacts and finalize summary."
    }
  ],
  "execution_steps": [
    {
      "name": "sensors",
      "agent": "IoTAgent",
      "action": "LocateSensor",
      "arguments": {
        "site_name": "MAIN",
        "assetnum": "Chiller 9",
        "sensor_name_filter": "Condenser Water Flow"
      },
      "outputs": {
        "status": "success",
        "site_name": "MAIN",
        "assetnum": "Chiller 9",
        "sensor_id": "chiller9_condenser_water_flow",
        "sensor_name": "Condenser Water Flow",
        "file_path": "/tmp/cbmdir/chiller9_MAIN_sensors.json",
        "message": "Located target sensor."
      },
      "deterministic": {"name": false, "action": false, "arguments": false, "outputs": false}
    },
    {
      "name": "history",
      "agent": "IoTAgent",
      "action": "DownloadSeries_Attempt1",
      "arguments": {
        "site_name": "MAIN",
        "assetnum": "9",
        "metric": "Condenser Water Flow",
        "start": "2020-04-27T00:00:00Z",
        "final": "2020-05-03T23:59:59Z",
        "save_series_csv": true,
        "save_path": "data/raw/iot/main/chiller_9_condenser_water_flow_2020-04-27_to_2020-05-03.csv"
      },
      "outputs": {
        "status": "failure",
        "error": "assetnum '9' not found (expected 'Chiller 9').",
        "csv_path": null,
        "message": "History retrieval failed due to asset identifier mismatch."
      },
      "deterministic": {"name": false, "action": false, "arguments": false, "outputs": false}
    },
    {
      "name": "history",
      "agent": "IoTAgent",
      "action": "DownloadSeries_ReattemptWithFixedAssetnum",
      "arguments": {
        "site_name": "MAIN",
        "assetnum": "Chiller 9",
        "metric": "Condenser Water Flow",
        "start": "2020-04-27T00:00:00Z",
        "final": "2020-05-03T23:59:59Z",
        "save_series_csv": true,
        "save_path": "data/raw/iot/main/chiller_9_condenser_water_flow_2020-04-27_to_2020-05-03.csv"
      },
      "outputs": {
        "status": "success",
        "rows": 581,
        "sampling_hint": "irregular/partial (expected ~672 @ 15min)",
        "csv_path": "data/raw/iot/main/chiller_9_condenser_water_flow_2020-04-27_to_2020-05-03.csv",
        "message": "Series retrieved and saved to CSV."
      },
      "deterministic": {"name": false, "action": false, "arguments": false, "outputs": false}
    },
    {
      "name": "jsonreader",
      "agent": "IoTAgent",
      "action": "DirectFileRead_Attempt",
      "arguments": {
        "file_name": "chiller9_condenser_water_flow_MAIN_2020-04-27.json"
      },
      "outputs": {
        "status": "failure",
        "error": "[Errno 2] No such file or directory: 'chiller9_condenser_water_flow_MAIN_2020-04-27.json'",
        "message": "Direct JSON read failed; proceed with CSV path from history step."
      },
      "deterministic": {"name": false, "action": false, "arguments": false, "outputs": false}
    },
    {
      "name": "tsfm_list_models",
      "agent": "TSFMAgent",
      "action": "EnumeratePretrainedCheckpoints",
      "arguments": {},
      "outputs": {
        "status": "success",
        "models": [
          {"model_id": "ttm_96_28", "model_checkpoint": "ttm_96_28", "model_description": "Pretrained forecasting model with context length 96"},
          {"model_id": "ttm_512_96", "model_checkpoint": "ttm_512_96", "model_description": "Pretrained forecasting model with context length 512"},
          {"model_id": "ttm_energy_96_28", "model_checkpoint": "ttm_96_28", "model_description": "Energy-tuned model, context length 96"},
          {"model_id": "ttm_energy_512_96", "model_checkpoint": "ttm_512_96", "model_description": "Energy-tuned model, context length 512"}
        ],
        "selected": "ttm_96_28",
        "message": "Selected checkpoint 'ttm_96_28' for forecasting."
      },
      "deterministic": {"name": false, "action": false, "arguments": false, "outputs": false}
    },
    {
      "name": "tsfm_forecast",
      "agent": "TSFMAgent",
      "action": "RunForecast_ttm_96_28_WithCorrectArgs",
      "arguments": {
        "input_csv": "data/raw/iot/main/chiller_9_condenser_water_flow_2020-04-27_to_2020-05-03.csv",
        "timestamp_col": "timestamp",
        "value_col": "value",
        "window_start": "2020-04-27T00:00:00Z",
        "window_end": "2020-05-03T23:59:59Z",
        "checkpoint": "ttm_96_28",
        "save_path": "data/derived/forecast/chiller_9/condenser_water_flow/forecast_2020-04-27_to_2020-05-03_ttm_96_28.json"
      },
      "outputs": {
        "status": "success",
        "model_checkpoint": "ttm_96_28",
        "input_read": true,
        "input_rows": 581,
        "forecast_span": {"start": "2020-04-27T00:00:00Z", "end": "2020-05-03T23:59:59Z"},
        "forecast_file": "data/derived/forecast/chiller_9/condenser_water_flow/forecast_2020-04-27_to_2020-05-03_ttm_96_28.json",
        "message": "Forecast generated and saved (arguments validated; prior unpacking/validation errors resolved)."
      },
      "deterministic": {"name": false, "action": false, "arguments": false, "outputs": false}
    },
    {
      "name": "finalize",
      "agent": "TSFMAgent",
      "action": "Finish",
      "arguments": {
        "validation_checks": [
          "exists:data/raw/iot/main/chiller_9_condenser_water_flow_2020-04-27_to_2020-05-03.csv",
          "exists:data/derived/forecast/chiller_9/condenser_water_flow/forecast_2020-04-27_to_2020-05-03_ttm_96_28.json"
        ],
        "summary_request": "Confirm variable binding, input read, forecast span, row counts, and artifact paths. Note prior failure remediation (assetnum correction; correct TSFM input schema)."
      },
      "outputs": {
        "final_state": "completed",
        "summary": "Forecast for Chiller 9 'Condenser Water Flow' (2020-04-27 → 2020-05-03) generated with checkpoint ttm_96_28. Input CSV (581 rows) read; forecast saved to declared path. Earlier issues (assetnum mismatch, missing JSON file, TSFM arg validation) were identified and resolved.",
        "forecast_file": "data/derived/forecast/chiller_9/condenser_water_flow/forecast_2020-04-27_to_2020-05-03_ttm_96_28.json"
      },
      "deterministic": {"name": false, "action": false, "arguments": false, "outputs": false}
    }
  ],
  "execution_links": [
    {"source": "sensors", "target": "history"},
    {"source": "history", "target": "history"},
    {"source": "history", "target": "jsonreader"},
    {"source": "jsonreader", "target": "tsfm_list_models"},
    {"source": "tsfm_list_models", "target": "tsfm_forecast"},
    {"source": "tsfm_forecast", "target": "finalize"}
  ]
}

Key Lessons from ID 502:

6. Evaluation Using Ground Truth

6.1 Evaluation Metrics

Ground truth enables multiple evaluation dimensions:

  1. Planning Accuracy
    • Compare agent’s planning steps to ground truth planning_steps
    • Use semantic similarity (not exact match)
    • Measure: Planning step coverage, agent selection accuracy
  2. Execution Accuracy
    • Compare agent’s execution DAG to ground truth execution DAG
    • Check: Tool selection, argument correctness, execution order
    • Measure: DAG similarity, tool usage accuracy
  3. Output Correctness
    • Compare agent’s final output to ground truth final_out
    • For deterministic: Exact match or validation rules
    • For non-deterministic: Semantic similarity, constraint satisfaction
    • Measure: Output accuracy, format compliance
  4. Efficiency
    • Compare number of steps taken vs. ground truth
    • Identify unnecessary steps or missing optimizations
    • Measure: Step count ratio, execution time

6.2 Scoring Functions

DAG Comparison:

Output Comparison:

Planning Comparison:

7. Paradigm-Specific Best Practices

7.1 Multi-Agent Ground Truth Best Practices

When creating ground truth for multi-agent systems:

Agent Selection

Agent Coordination

Domain-Specific Actions

Example Multi-Agent Pattern:

{
  "type": "multiagent",
  "planning_steps": [
    {
      "agent": "IoTAgent",
      "instruction": "Retrieve sensor data for equipment"
    },
    {
      "agent": "TSFMAgent",
      "instruction": "Analyze sensor data for anomalies"
    },
    {
      "agent": "WOAgent",
      "instruction": "Recommend work orders based on anomalies"
    }
  ],
  "execution_steps": [
    {
      "name": "get_sensor_data",
      "action": "sensors",
      "agent": "IoTAgent",
      "arguments": {"site_name": "MAIN", "assetnum": "Chiller 6"}
    },
    {
      "name": "detect_anomalies",
      "action": "detect_anomalies",
      "agent": "TSFMAgent",
      "arguments": {"data": "sensor_data"}
    },
    {
      "name": "recommend_wo",
      "action": "recommend_work_order",
      "agent": "WOAgent",
      "arguments": {"anomalies": "detected_anomalies"}
    }
  ]
}

7.2 Multi-MCP Ground Truth Best Practices

When creating ground truth for MCP (Model Context Protocol) systems:

MCP Server Types

Use standard MCP server types:

MCP Tool Naming

Use MCP protocol tool naming conventions:

{
  "action": "filesystem/read_file",      // Server/tool format
  "action": "database/query",
  "action": "web-search/search",
  "action": "api-client/post"
}

Or simplified format:

{
  "action": "read_file",                 // Tool name only
  "agent": "filesystem"                  // Server specified separately
}

MCP Resource References

Handle MCP resources with proper URIs:

{
  "arguments": {
    "resource_uri": "file:///path/to/data.csv",
    "resource_type": "text/csv"
  }
}

{
  "arguments": {
    "resource_uri": "db://localhost/sensors",
    "query": "SELECT * FROM readings WHERE date > '2024-01-01'"
  }
}

MCP Tool Schemas

Reference MCP tool schemas in execution steps:

{
  "name": "read_config",
  "action": "read_file",
  "agent": "filesystem",
  "arguments": {
    "path": "/config/app.json"
  },
  "outputs": ["file_content"],
  "deterministic": {
    "name": false,
    "action": true,
    "arguments": true,
    "outputs": true
  }
}

MCP Server Coordination

For multi-MCP scenarios:

{
  "type": "multiagent",  // Or "multi-mcp"
  "planning_steps": [
    {
      "agent": "filesystem",
      "instruction": "Read data file"
    },
    {
      "agent": "database",
      "instruction": "Query reference data"
    },
    {
      "agent": "api-client",
      "instruction": "Send results to external API"
    }
  ]
}

Example Multi-MCP Pattern:

{
  "id": 600,
  "text": "Read customer data from CSV, enrich with database info, and send to CRM API.",
  "type": "multiagent",
  "category": "Complex Query",
  "deterministic": true,
  "planning_steps": [
    {
      "agent": "filesystem",
      "instruction": "Read customer CSV file"
    },
    {
      "agent": "database",
      "instruction": "Query customer details from database"
    },
    {
      "agent": "api-client",
      "instruction": "POST enriched data to CRM API"
    }
  ],
  "execution_steps": [
    {
      "name": "read_csv",
      "action": "read_file",
      "agent": "filesystem",
      "arguments": {
        "path": "/data/customers.csv"
      },
      "outputs": ["csv_content"],
      "deterministic": {
        "name": false,
        "action": true,
        "arguments": true,
        "outputs": true
      }
    },
    {
      "name": "parse_csv",
      "action": "parse",
      "agent": "filesystem",
      "arguments": {
        "content": "csv_content",
        "format": "csv"
      },
      "outputs": ["customer_ids"],
      "deterministic": {
        "name": false,
        "action": true,
        "arguments": true,
        "outputs": true
      }
    },
    {
      "name": "query_details",
      "action": "query",
      "agent": "database",
      "arguments": {
        "sql": "SELECT * FROM customers WHERE id IN (@customer_ids)",
        "params": {"customer_ids": "customer_ids"}
      },
      "outputs": ["customer_details"],
      "deterministic": {
        "name": false,
        "action": true,
        "arguments": true,
        "outputs": true
      }
    },
    {
      "name": "send_to_crm",
      "action": "post",
      "agent": "api-client",
      "arguments": {
        "url": "https://crm.example.com/api/customers",
        "body": "customer_details",
        "headers": {"Content-Type": "application/json"}
      },
      "outputs": ["api_response"],
      "deterministic": {
        "name": false,
        "action": true,
        "arguments": true,
        "outputs": false
      }
    }
  ],
  "execution_links": [
    {"source": "read_csv", "target": "parse_csv"},
    {"source": "parse_csv", "target": "query_details"},
    {"source": "query_details", "target": "send_to_crm"}
  ]
}

7.3 Hybrid Approach Best Practices

When combining multi-agent and multi-MCP:

Clear Separation of Concerns

Consistent Naming

Layered Execution

{
  "execution_steps": [
    {
      "name": "agent_decides",
      "action": "analyze_requirements",
      "agent": "PlannerAgent",
      "outputs": ["required_files"]
    },
    {
      "name": "mcp_retrieves",
      "action": "read_file",
      "agent": "filesystem",
      "arguments": {"paths": "required_files"},
      "outputs": ["file_contents"]
    },
    {
      "name": "agent_processes",
      "action": "process_data",
      "agent": "ProcessorAgent",
      "arguments": {"data": "file_contents"},
      "outputs": ["results"]
    }
  ]
}

7.4 Choosing Between Paradigms

Use Multi-Agent When:

Use Multi-MCP When:

Use Hybrid When:

8. Using Ground Truth for Agent Evaluation

8.1 Three-Level Evaluation Framework

Ground truth enables comprehensive evaluation at three levels:

Level Focus Metrics Weight
Outcome Final result correctness Task completeness (y₁), Data retrieval (y₂), Result verification (y₃) 50%
Process Execution trajectory Action sequence, DAG structure, Step completeness 30%
Planning Strategic decomposition ROUGE scores (planning alignment) 20%

8.2 Evaluation Workflow

Step 1: Collect Agent Trace → Step 2: Load Ground Truth → Step 3: Multi-Level Comparison

8.3 Rubric-Based Scoring (Outcome Level)

Three Core Metrics:

  1. Task Completeness (y₁): Compare agent output against characteristic_form using LLM-as-Judge (0-1 score)
  2. Data Retrieval Accuracy (y₂): Verify correct parameters (equipment IDs, date ranges, sensors)
  3. Result Verification (y₃):
    • Deterministic: Exact match with final_out
    • Non-deterministic: Semantic similarity

Combined Score: Rubric_Score = (y₁ + y₂ + y₃) / 3

Example (Scenario 421):

8.4 Execution Alignment (Process Level)

Comparison Methods:

Example:

8.5 Planning Alignment (Strategic Level)

Use ROUGE metrics to compare planning strategies:

Example: GT planning vs Agent planning → ROUGE-L: 0.65

8.6 Hybrid Scoring Formula

Final_Score = 0.5 × Rubric_Score + 0.3 × Execution_Score + 0.2 × Planning_Score

Complete Example (Scenario 421):

{
  "final_score": 0.78,
  "rubric_score": 0.87,
  "execution_score": 0.68,
  "planning_score": 0.65,
  "failure_analysis": {
    "missing_steps": ["load_events", "dedupe"],
    "output_issues": "Missing 2 of 5 expected codes"
  }
}

8.7 Batch Evaluation

Aggregate Metrics Across Benchmark:

AssetOpsBench Example: GPT-4.1 achieved 0.72 avg score across 141 scenarios (70% completion rate)

8.8 Automated Failure Detection

Ground truth enables automatic identification of:

8.9 Evaluation Best Practices

Scenario Type Approach
Deterministic Exact matching, strict validation, focus on correctness
Non-Deterministic Semantic similarity, allow trajectory variation, focus on outcome quality
Multi-Agent Track coordination, evaluate inter-agent communication, verify specialization
Multi-MCP Verify server selection, check tool parameters, validate resource URIs

9. Illustrated Examples from AssetOpsBench

Example 1: Simple Query (Scenario 3)

Utterance: “What assets can be found at the MAIN site?”

{
  "id": 3,
  "type": "IoT",
  "text": "What assets can be found at the MAIN site?",
  "category": "Knowledge Query",
  "deterministic": true,
  "characteristic_form": "The expected response should be the return value from querying the assets at the MAIN site. The response should be a reference to a file containing the list of assets",
  "group": "retrospective",
  "entity": "Site",
  "note": "Source: IoT data operations; Deterministic query with single correct answer; Category: Knowledge Query",
  "planning_steps": [
    {"agent": "IoTAgent", "instruction": "list assets for site MAIN"}
  ],
  "execution_steps": [
    {
      "name": "step1",
      "action": "assets",
      "agent": "IoTAgent",
      "arguments": {"site_name": "MAIN"},
      "outputs": []
    },
    {
      "name": "finish",
      "action": "Finish",
      "agent": "IoTAgent",
      "argument": "The assets at the MAIN site are: CQPA AHU 1, CQPA AHU 2B, Chiller 4, Chiller 6, Chiller 9, Chiller 3."
    }
  ],
  "execution_links": [{"source": "step1", "target": "finish"}]
}

Key Features: Linear DAG (2 steps), fully deterministic, single agent. Includes complete meta information: text, characteristic_form, group (retrospective), entity (Site), and note fields.

Example 2: Complex Diagnostic (Scenario 421)

Utterance: “Review anomalies/alerts for Chiller 9 (CWC04009) in May-June 2020 and suggest work orders.”

{
  "id": 421,
  "type": "Workorder",
  "deterministic": false,
  "planning_steps": [
    {"agent": "WOAgent", "instruction": "Create Equipment(CWC04009) and DateRange(2020-05-01 to 2020-06-30)"},
    {"agent": "WOAgent", "instruction": "Retrieve and filter events to ALERT and ANOMALY"},
    {"agent": "WOAgent", "instruction": "Get KPI-based and alert-based recommendations in parallel"},
    {"agent": "WOAgent", "instruction": "Combine and deduplicate recommendations"}
  ],
  "execution_steps": [
    {"name": "create_equipment", "action": "Equipment", "arguments": {"equipment_id": "CWC04009"}, "outputs": ["equipment"]},
    {"name": "create_date_range", "action": "DateRange", "arguments": {"start_date": "2020-05-01", "end_date": "2020-06-30"}, "outputs": ["date_range"]},
    {"name": "get_events", "action": "get_events", "arguments": {"equipment": "equipment", "date_range": "date_range"}, "outputs": ["events_pickle_path"]},
    {"name": "load_events", "action": "pickle_load", "arguments": {"file_path": "events_pickle_path"}, "outputs": ["events"]},
    {"name": "filter_anomaly_alert", "action": "filter", "arguments": {"obj": "events", "condition": "event.event_category in {'ALERT','ANOMALY'}"}, "outputs": ["anomaly_alert_events"]},
    {"name": "get_top_kpi_recommendations", "action": "recommend_from_top_kpi_anomalies", "arguments": {"events": "anomaly_alert_events", "top_k": 3}, "outputs": ["kpi_recommendations"]},
    {"name": "alert_recommendation", "action": "recommend_work_order_from_alert", "arguments": {"rule_id": "RUL0018"}, "outputs": ["alert_rec"]},
    {"name": "combine_and_dedupe", "action": "dedupe_by_key", "arguments": {"items": "[alert_rec] + kpi_recommendations", "key": "rec.primary_code"}, "outputs": ["final_result"]},
    {"name": "print_final", "action": "print", "arguments": {"obj": "final_result"}}
  ],
  "execution_links": [
    {"source": "create_equipment", "target": "get_events"},
    {"source": "create_date_range", "target": "get_events"},
    {"source": "get_events", "target": "load_events"},
    {"source": "load_events", "target": "filter_anomaly_alert"},
    {"source": "filter_anomaly_alert", "target": "get_top_kpi_recommendations"},
    {"source": "filter_anomaly_alert", "target": "alert_recommendation"},
    {"source": "get_top_kpi_recommendations", "target": "combine_and_dedupe"},
    {"source": "alert_recommendation", "target": "combine_and_dedupe"},
    {"source": "combine_and_dedupe", "target": "print_final"}
  ],
  "final_out": {"CWC04009": ["M005", "M006", "OP004", "M013", "OP002"]}
}

Key Features: Parallel branches (KPI + alert paths converge at dedupe), non-deterministic filtering/reasoning, 9 steps with 8 dependencies, ~3-4 hours to create.

DAG Pattern:

create_equipment ──┐
                   ├─> get_events ─> load_events ─> filter ─┬─> kpi_rec ──┐
create_date_range ─┘                                         │              ├─> dedupe ─> print
                                                              └─> alert_rec ─┘

Comparison:

Aspect Example 1 Example 2
Steps 2 9
Dependencies 1 8
Deterministic Fully Partially
DAG Pattern Linear Parallel + Convergence
Creation Time 30 min 3-4 hours

10. Two Paradigms: Multi-Agent vs Multi-MCP

10.1 Unified Ground Truth Structure

Key Insight: Both paradigms use the same ground truth structure. The only difference is semantic interpretation:

Field Multi-Agent Interpretation Multi-MCP Interpretation
type Agent type (IoTAgent, WOAgent) MCP server type (filesystem, database)
agent Agent name MCP server name
action Agent-specific function MCP tool name
planning_steps Agent orchestration plan MCP server coordination plan
execution_steps Agent function calls MCP tool invocations

10.2 Understanding the Two Approaches

Ground truth creation supports two distinct but structurally similar paradigms:

Multi-Agent Systems

Concept: Multiple specialized AI agents, each with domain expertise, collaborate to solve complex tasks.

Characteristics:

Ground Truth Representation:

{
  "type": "Workorder",           // Primary agent type
  "planning_steps": [
    {
      "agent": "WOAgent",         // Specific agent name
      "instruction": "Create Equipment instance"
    }
  ],
  "execution_steps": [
    {
      "name": "create_equipment",
      "action": "Equipment",      // Agent-specific action
      "agent": "WOAgent",         // Agent executing this step
      "arguments": {...}
    }
  ]
}

Multi-MCP Systems

Concept: Multiple Model Context Protocol (MCP) servers provide standardized tools and resources that LLMs can use to accomplish tasks.

Characteristics:

Ground Truth Representation:

{
  "type": "filesystem",          // Primary MCP server type
  "planning_steps": [
    {
      "agent": "filesystem",      // MCP server name
      "instruction": "Read configuration file"
    }
  ],
  "execution_steps": [
    {
      "name": "read_config",
      "action": "read_file",      // MCP tool name
      "agent": "filesystem",      // MCP server executing this step
      "arguments": {
        "path": "/config/app.json"
      }
    }
  ]
}

10.3 When to Use Each Paradigm

Use Multi-Agent Ground Truth When:

Use Multi-MCP Ground Truth When:

Use Hybrid Approach When:

10.4 Conversion Between Paradigms

Ground truth can be adapted between paradigms by reinterpreting fields:

Multi-Agent → Multi-MCP:

// Multi-Agent
{
  "type": "IoTAgent",
  "action": "get_sensor_data"
}

// Equivalent Multi-MCP
{
  "type": "database",
  "action": "query"
}

Multi-MCP → Multi-Agent:

// Multi-MCP
{
  "type": "filesystem",
  "action": "read_file"
}

// Equivalent Multi-Agent
{
  "type": "FileAgent",
  "action": "read"
}

10.5 Examples of Both Paradigms

Example 1: Multi-Agent Ground Truth

Scenario: Retrieve work orders for equipment

{
  "id": 400,
  "text": "Get the work order of equipment CWC04013 for year 2017.",
  "type": "Workorder",
  "category": "Data Query",
  "deterministic": true,
  "planning_steps": [
    {
      "agent": "WOAgent",
      "instruction": "Create Equipment instance for CWC04013"
    },
    {
      "agent": "WOAgent",
      "instruction": "Retrieve work orders for 2017"
    }
  ],
  "execution_steps": [
    {
      "name": "create_equipment",
      "action": "Equipment",
      "agent": "WOAgent",
      "arguments": {"equipment_id": "CWC04013"}
    },
    {
      "name": "get_work_orders",
      "action": "get_work_orders",
      "agent": "WOAgent",
      "arguments": {
        "equipment": "equipment",
        "date_range": "date_range"
      }
    }
  ]
}

Example 2: Multi-MCP Ground Truth

Scenario: Read and analyze configuration file

{
  "id": 501,
  "text": "Read the application configuration and validate required fields.",
  "type": "filesystem",
  "category": "Data Query",
  "deterministic": true,
  "planning_steps": [
    {
      "agent": "filesystem",
      "instruction": "Read configuration file from disk"
    },
    {
      "agent": "json-parser",
      "instruction": "Parse JSON configuration"
    },
    {
      "agent": "validator",
      "instruction": "Validate required fields"
    }
  ],
  "execution_steps": [
    {
      "name": "read_config",
      "action": "read_file",
      "agent": "filesystem",
      "arguments": {
        "path": "/config/app.json"
      },
      "outputs": ["config_content"]
    },
    {
      "name": "parse_json",
      "action": "parse",
      "agent": "json-parser",
      "arguments": {
        "content": "config_content"
      },
      "outputs": ["config_object"]
    },
    {
      "name": "validate",
      "action": "validate_schema",
      "agent": "validator",
      "arguments": {
        "data": "config_object",
        "required_fields": ["api_key", "endpoint", "timeout"]
      },
      "outputs": ["validation_result"]
    }
  ]
}

Example 3: Hybrid Ground Truth

Scenario: IoT agent uses filesystem MCP server

{
  "id": 502,
  "text": "Load sensor data from CSV file and detect anomalies.",
  "type": "multiagent",
  "category": "Complex Query",
  "deterministic": false,
  "planning_steps": [
    {
      "agent": "filesystem",
      "instruction": "Read sensor data CSV file"
    },
    {
      "agent": "IoTAgent",
      "instruction": "Parse and validate sensor data"
    },
    {
      "agent": "AnomalyDetectionAgent",
      "instruction": "Detect anomalies in sensor readings"
    }
  ],
  "execution_steps": [
    {
      "name": "read_csv",
      "action": "read_file",
      "agent": "filesystem",
      "arguments": {"path": "/data/sensors.csv"},
      "outputs": ["csv_content"]
    },
    {
      "name": "parse_data",
      "action": "parse_sensor_data",
      "agent": "IoTAgent",
      "arguments": {"csv_data": "csv_content"},
      "outputs": ["sensor_readings"]
    },
    {
      "name": "detect_anomalies",
      "action": "detect",
      "agent": "AnomalyDetectionAgent",
      "arguments": {"data": "sensor_readings"},
      "outputs": ["anomalies"]
    }
  ]
}

10.6 Choosing the Right Paradigm for Your Ground Truth

Decision Framework:

  1. What are you evaluating?
    • AI agent reasoning and coordination → Multi-Agent
    • LLM tool-using capabilities → Multi-MCP
    • Both → Hybrid
  2. What is your system architecture?
    • Specialized agents with domain logic → Multi-Agent
    • MCP servers with standardized tools → Multi-MCP
    • Agents using MCP tools → Hybrid
  3. What is your evaluation focus?
    • Agent selection and orchestration → Multi-Agent
    • Tool selection and parameter passing → Multi-MCP
    • End-to-end task completion → Either/Both

Recommendation: Start with the paradigm that matches your system architecture, then adapt as needed. The ground truth structure is flexible enough to support both.

11. Summary

Core Requirements for High-Quality Ground Truth

  1. Clear Understanding: Thoroughly understand utterance intent and domain context
  2. Structured Planning: Break down problems into logical planning_steps
  3. Detailed Execution: Specify exact tools, arguments, and data flow in execution_steps
  4. Valid DAG: Ensure execution_links form proper directed acyclic graph
  5. Accurate Determinism: Mark determinism correctly at all levels
  6. Complete Output: Provide final_out or final_out_description with validation criteria
  7. Thorough Validation: Verify completeness, correctness, and testability

Key Principles

Document Organization

Main Sections (1-12): Core concepts, workflow, structure, best practices, examples, value proposition References (Section 13): Citations and related work Appendices (A-E): Detailed technical content

Getting Started

  1. Review Section 2 (High-Level Workflow) for the 5-phase process
  2. Study Section 11 (Illustrated Examples) for concrete patterns
  3. Use Appendix E (Quick Reference Checklist) during creation
  4. Consult Section 4 (Ground Truth Structure) for field definitions
  5. Reference Section 10 (Paradigm-Specific Practices) for multi-agent vs. multi-MCP guidance

Value Proposition

Ground truth trajectories enable:

ROI: 564 hours to create 141 scenarios → 1000+ hours saved in evaluation (AssetOpsBench experience)

By following this best practice, you can create ground truth data that enables systematic, reproducible evaluation of AI agent systems across diverse domains and use cases.

12. References

Primary References

  1. Patel, D., Lin, S., Rayfield, J., Zhou, N., Vaculin, R., Martinez, N., O’donncha, F., & Kalagnanam, J. (2025). “AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance.” arXiv preprint arXiv:2506.03828.
    • URL: https://arxiv.org/abs/2506.03828
    • Key Contribution: 141 ground truth scenarios, evaluation metrics (rubric-based + ROUGE), architectural comparison
  2. AssetOpsBench Utterance Dataset. IBM Research (2025).
    • URL: https://huggingface.co/datasets/ibm-research/AssetOpsBench
    • Contains: 152 task utterances for industrial asset operations and maintenance scenarios
    • Platform: HuggingFace Datasets
  3. AssetOpsBench Ground Truth Dataset (2026 - IBM Internal).
    • Repository: AssetOpsBenchGroundTruth/
    • Contains: 152 validated scenarios across IoT, Workorder, TSFM, and FMSR domains
  4. Lin, C. Y. (2004). “ROUGE: A Package for Automatic Evaluation of Summaries.” Proceedings of the ACL Workshop on Text Summarization Branches Out.
    • Used for planning and execution alignment scoring in ground truth evaluation

Technical Specifications

  1. Model Context Protocol (MCP) Specification. Anthropic.
    • URL: https://modelcontextprotocol.io/
    • Framework for standardized tool interfaces in multi-MCP systems

Appendix A: Ground Truth JSON Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["id", "text", "type", "category", "deterministic", "characteristic_form", "planning_steps", "execution_steps", "execution_links"],
  "properties": {
    "id": {"type": "integer"},
    "uuid": {"type": "string", "format": "uuid"},
    "text": {"type": "string"},
    "type": {"type": "string"},
    "category": {"type": "string"},
    "deterministic": {"type": "boolean"},
    "characteristic_form": {"type": "string"},
    "planning_steps": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["agent", "instruction"],
        "properties": {
          "agent": {"type": "string"},
          "instruction": {"type": "string"}
        }
      }
    },
    "execution_steps": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["name", "action", "agent"],
        "properties": {
          "name": {"type": "string"},
          "action": {"type": "string"},
          "agent": {"type": "string"},
          "arguments": {"type": ["object", "string"]},
          "outputs": {"type": "array"},
          "deterministic": {
            "type": "object",
            "properties": {
              "name": {"type": "boolean"},
              "action": {"type": "boolean"},
              "arguments": {"type": "boolean"},
              "outputs": {"type": "boolean"}
            }
          }
        }
      }
    },
    "execution_links": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["source", "target"],
        "properties": {
          "source": {"type": "string"},
          "target": {"type": "string"}
        }
      }
    },
    "final_out": {"type": ["object", "null"]},
    "final_out_description": {
      "type": "array",
      "items": {"type": "string"}
    }
  }
}

Appendix B: Common Failure Patterns

C.1 Tool Selection Errors

Pattern: Agent uses wrong tool for the task

// Ground Truth
{"action": "get_work_orders"}

// Agent Execution
{"action": "get_events"}  // Wrong tool

Detection: Compare execution_steps[].action fields Fix: Improve tool descriptions, add examples

C.2 Parameter Errors

Pattern: Agent uses incorrect arguments

// Ground Truth
{"equipment_id": "CWC04013"}

// Agent Execution
{"equipment_id": "CWC04009"}  // Wrong equipment

Detection: Compare execution_steps[].arguments fields Fix: Validate parameter extraction, add constraints

C.3 Missing Steps

Pattern: Agent skips required operations

// Ground Truth: 5 steps
create_equipment → create_date_range → get_events → load_events → process

// Agent: 3 steps (missing load_events, process)
create_equipment → create_date_range → get_events

Detection: Compare DAG node counts and structure Fix: Improve planning, add step-by-step guidance

C.4 Incorrect Sequencing

Pattern: Agent executes steps in wrong order

// Ground Truth
A → B → C

// Agent Execution
A → C → B  // Wrong order, may cause dependency errors

Detection: Compare execution_links topology Fix: Enforce dependency constraints, validate prerequisites

C.5 Premature Termination

Pattern: Agent stops before completing workflow

// Ground Truth
... → final_step → Finish

// Agent Execution
... → partial_step  // Stops early, no Finish

Detection: Check for Finish action and final_out presence Fix: Add completion criteria, validate output requirements

Appendix D: Domain-Specific Examples

D.1 Healthcare Diagnosis

{
  "id": 701,
  "text": "Review patient symptoms and lab results to recommend diagnostic tests.",
  "type": "Healthcare",
  "category": "Diagnostic Recommendation",
  "planning_steps": [
    {"agent": "DiagnosticAgent", "instruction": "Retrieve patient symptoms and history"},
    {"agent": "DiagnosticAgent", "instruction": "Analyze lab results"},
    {"agent": "DiagnosticAgent", "instruction": "Recommend diagnostic tests based on findings"}
  ]
}

D.2 Financial Analysis

{
  "id": 801,
  "text": "Analyze Q4 revenue trends and forecast Q1 performance.",
  "type": "Finance",
  "category": "Forecasting",
  "planning_steps": [
    {"agent": "FinanceAgent", "instruction": "Retrieve Q4 revenue data"},
    {"agent": "FinanceAgent", "instruction": "Calculate trends and growth rates"},
    {"agent": "FinanceAgent", "instruction": "Generate Q1 forecast using time series model"}
  ]
}

D.3 Customer Support

{
  "id": 901,
  "text": "Resolve customer complaint about delayed shipment.",
  "type": "CustomerService",
  "category": "Issue Resolution",
  "planning_steps": [
    {"agent": "SupportAgent", "instruction": "Retrieve order and shipment details"},
    {"agent": "SupportAgent", "instruction": "Identify delay cause"},
    {"agent": "SupportAgent", "instruction": "Propose resolution and compensation"}
  ]
}
This site is open source. Improve this page.