AssetOpsBench

Utterance Design Guideline for Domain-Agnostic Applications

Purpose

This guideline helps domain experts and developers create well-structured, consistent utterances across diverse domains. Utterances are natural language queries or commands that users make to interact with AI-powered systems.

Why This Guideline: Well-designed utterances enable:

Domain-Agnostic Design: This guideline is designed to handle various domains through a consistent abstraction pattern. The same patterns apply across:

A Good Set of Utterances for a Given Domain Should:

For Example, in Industrial Asset Management Contexts, This Translates To:

Core Problem Groups

To design effective utterances across any domain, we define three universal problem categories that span the complete lifecycle of data-driven decision-making:

1. retrospective (Knowledge Extraction)

Definition
Understanding historical or current state through data retrieval and analysis.

Universal Coverage

Focus Question
What has happened or what is known?

Cross-Domain Examples

2. predictive

Definition
Estimating future behavior or states under status quo conditions (no interventions).

Universal Coverage

Focus Question
What is likely to happen next?

Cross-Domain Examples

3. prescriptive

Definition
Determining optimal interventions, actions, or strategies under operational constraints.

Universal Coverage

Focus Question
What should be done and when?

Cross-Domain Examples

Domain Abstraction Pattern

To create domain-agnostic utterances that can be adapted across applications, use a consistent placeholder system:

Core Placeholders

Placeholder Description Examples
{ENTITY} Primary object of analysis Patient, Account, Equipment, Vehicle, Student
{ENTITY_CLASS} Category or type Diagnosis, Portfolio, HVAC, Fleet, Course
{LOCATION} Physical or logical location Hospital, Branch, Site, Route, Campus
{METRIC} Measurement or KPI Heart Rate, Balance, Temperature, Speed, Grade
{EVENT_TYPE} Type of occurrence Admission, Transaction, Alert, Delay, Enrollment
{EVENT_NAME} Specific event label Readmission, Withdrawal, Failure, Accident, Dropout
{TIME_RANGE} Temporal scope “Last month”, “2020-06-01 to 2020-06-30”, “Next quarter”
{WINDOW} Time horizon “Within 14 days”, “Next maintenance window”, “By end of semester”
{ACTION} Intervention or decision Treatment, Investment, Maintenance, Reroute, Tutoring
{CONDITION} Trigger or state “When anomaly detected”, “If balance < threshold”, “When grade drops”
{OUTPUT_FORM} Response format List, Table, Chart, File, Summary

Usage Example

Generic Template:
“Retrieve {METRIC} for {ENTITY} at {LOCATION} over {TIME_RANGE}”

Domain Adaptations:

Utterance Schema Structure

Each utterance should be documented with the following attributes in this order:

Required Fields

  1. id (integer): Unique identifier for the utterance
  2. text (string): The actual utterance text in natural language
  3. type (string): The MCP server, AI agent, service, or system component that processes this utterance
    • In MCP (Model Context Protocol) systems: Identifies which MCP server handles the request (e.g., “filesystem”, “database”, “web-search”)
    • In multi-agent systems: Identifies which agent handles the request (e.g., “IoT”, “FMSA”, “TSFM”, “Workorder”, “multiagent”)
    • In single-agent systems: Can represent the domain or service type (e.g., “Healthcare”, “Finance”, “CustomerService”)
    • In microservices: Can indicate the service endpoint (e.g., “PatientAPI”, “BillingService”, “InventoryService”)
  4. category (string): Classification of the utterance type (see Category Definitions section)
  5. deterministic (boolean): Whether the utterance has a single correct answer (true) or allows multiple valid responses (false)
  6. characteristic_form (string): Description of the expected response format and content
  7. group (string or array): Problem category - RETROSPECTIVE, PREDICTIVE, PRESCRIPTIVE, or combination (e.g., [“retrospective”, “predictive”])
  8. entity (string or array): Primary entity/entities the utterance addresses (e.g., “Chiller”, “Patient”, “Account”, “Vehicle”)
  9. note (string): A flexible field for any information useful to explain or contextualize the utterance, including source, owner/contributor, omitted details, design rationale, implementation notes, or any other relevant metadata

Field Descriptions

type: Specifies which MCP server, AI agent, service, or system component processes this utterance:

group: Indicates which of the three core problem categories the utterance belongs to:

entity: Specifies the primary physical subject(s) of the utterance - the tangible things being analyzed, monitored, or acted upon:

note: A flexible, required field for storing any information useful to explain or contextualize the utterance. Common uses include:

The field is intentionally flexible to accommodate diverse documentation needs across different domains and use cases.

Examples

Basic Example:

{
  "id": 1,
  "text": "What IoT sites are available?",
  "type": "IoT",
  "category": "Knowledge Query",
  "deterministic": true,
  "characteristic_form": "The expected response should be the return value of all sites, either as text or as a reference to a file",
  "group": "retrospective",
  "entity": "Site",
  "note": "Source: Initial domain analysis; Owner: Domain SME Team; Basic inventory query pattern"
}

Predictive Example:

{
  "id": 416,
  "text": "When an anomaly happens for equipment CWC04009, can you recommend top three work orders?",
  "type": "Workorder",
  "category": "Decision Support",
  "deterministic": false,
  "characteristic_form": "It gives a list of work order with a primary failure code.",
  "group": "prescriptive",
  "entity": "Chiller",
  "note": "Source: Operations team request; Owner: Maintenance SME; Supports proactive maintenance workflow"
}

Multi-Group Example:

{
  "id": 420,
  "text": "Review the performance of chiller 9 for June 2020 and track any anomalies or operation violations as alerts.",
  "type": "Workorder",
  "category": "Decision Support",
  "deterministic": false,
  "characteristic_form": "Review operational performance and detect anomalies or alerts for Chiller 9 during June 2020.",
  "group": ["retrospective", "predictive"],
  "entity": "Chiller",
  "note": "Source: Operations dashboard requirements; Owner: Analytics Team; Multi-category query combining historical review with anomaly detection"
}

Example with Note:

{
  "id": 409,
  "text": "Get the daily count of alert, anomaly and work order event for May 2020.",
  "type": "Workorder",
  "category": "Knowledge Query",
  "deterministic": false,
  "characteristic_form": "There are 26 days with records. Depending on the LLM used, the result could be daily total event summary or daily summary for each event type.",
  "group": "retrospective",
  "entity": ["Alert", "Anomaly", "WorkOrder"],
  "note": "We have both work order business data object and work order event as a group type in the event file."
}

Category Definitions (Suggested)

The following categories are suggested classifications that have proven useful across multiple domains. You may adapt, combine, or create new categories based on your specific domain needs.

Information Retrieval

Requests for accessing existing information from knowledge bases or databases.

Data Extraction

Requests for retrieving specific data points, measurements, or records over time.

Analysis & Inference

Requests for AI-powered analysis, predictions, or pattern recognition on provided data.

Model Customization

Requests for adapting or fine-tuning AI models to specific contexts or data.

Anomaly & Exception Detection

Requests for identifying unusual patterns, outliers, or deviations from normal behavior.

Recommendation & Optimization

Requests for actionable recommendations, strategic guidance, or optimization suggestions.

Future State Prediction

Requests for forecasting future states, events, or probabilities.

Multi-Step Orchestration

Complex requests requiring coordination of multiple operations in sequence.

Note: These categories are not mutually exclusive. Some utterances may span multiple categories. The key is to choose the primary category that best describes the utterance’s main intent. For domain-specific category examples, see the case studies below.

Deterministic vs. Non-Deterministic Utterances

Deterministic (deterministic: true)

Non-Deterministic (deterministic: false)

Characteristic Form Guidelines

The characteristic_form field describes the expected response structure and validation criteria.

For Deterministic Utterances

  1. Exact Expected Output: Specify the precise data or format
  2. Validation Criteria: How to verify correctness
  3. Data Format: Specify file types, data structures

For Non-Deterministic Utterances

  1. Response Constraints: What must be included
  2. Quality Criteria: What makes a good response
  3. Acceptable Variations: Note when multiple approaches are valid

General Guidelines

Domain-Agnostic Utterance Templates

Template 1: Historical Data Retrieval

"Retrieve {METRIC} for {ENTITY} at {LOCATION} from {TIME_RANGE}"

Template 2: Knowledge Lookup

"List all {ATTRIBUTE} of {ENTITY_CLASS}"

Template 3: Predictive Analysis

"Forecast {METRIC} for {ENTITY} for {FUTURE_RANGE} based on data from {PAST_RANGE}"

Template 4: Anomaly Detection

"Is there any anomaly detected in {ENTITY}'s {METRIC} in {TIME_RANGE} at {LOCATION}?"

Template 5: Recommendation

"When {CONDITION} happens for {ENTITY}, recommend {ACTION_TYPE}"

Template 6: Multi-Factor Decision Support

"Review {DATA_SOURCES} for {ENTITY} during {PERIOD} and {recommend/analyze/prioritize} {ACTION}"

Utterance Set Completeness

Definition

Completeness refers to the degree to which a set of utterances covers the full scope of requirements for a specific domain. A complete utterance set should address all critical operational scenarios, decision points, and information needs within the domain.

Coverage Dimensions

Operational Lifecycle Coverage

Problem Category Balance

Entity/System Coverage

Stakeholder Perspective Coverage

Achieving Completeness

To ensure comprehensive coverage, consider mapping domain requirements to the three core categories (Retrospective, Predictive, Prescriptive) and identifying critical operational scenarios. Create a coverage matrix to highlight gaps, then iteratively refine the utterance set by starting with high-priority scenarios, testing with domain experts, and expanding based on actual usage patterns and user feedback.

Utterance Specificity and Incremental Development

Collaborative Development with Subject Matter Experts

Subject Matter Experts (SMEs) are domain specialists with deep knowledge of specific equipment, processes, or operational contexts. Rather than requiring one SME to create all utterances at once, the guideline supports incremental, collaborative development.

Suggested Approach:

Prioritization: Focus on critical operations, high-frequency needs, and high-impact decisions first. Quality and relevance matter more than quantity.

Validation Checklist

Before finalizing an utterance, verify:

Summary

Effective domain-agnostic utterance design requires:

Framework Fundamentals

  1. Universal categorization into Retrospective, Predictive, or Prescriptive
  2. Consistent abstraction using domain-agnostic placeholders
  3. Proper classification by query type
  4. Accurate deterministic flag based on answer uniqueness
  5. Detailed characteristic form describing expected responses

Completeness and Coverage

  1. Comprehensive coverage across operational lifecycle and stakeholder needs
  2. Balanced distribution across problem categories
  3. Gap analysis using coverage matrices
  4. Iterative refinement based on feedback

Collaborative Development

  1. SME-driven creation leveraging domain experts
  2. Incremental development allowing phased rollout
  3. Distributed ownership with clear responsibilities
  4. Prioritization focusing on critical scenarios
  5. Quality over quantity ensuring practical utility

Practical Implementation

  1. Domain-specific adaptation while maintaining consistency
  2. Validation criteria ensuring accuracy
  3. Documentation standards including ownership and dependencies
  4. Continuous improvement through monitoring and feedback

Getting Started

For New Domains:

  1. Identify 3-5 critical entity types or operational scenarios
  2. Engage relevant SMEs for each area
  3. Create 5-10 core utterances per SME covering high-priority needs
  4. Build coverage matrix to track completeness
  5. Iterate and expand based on user feedback

For Existing Systems:

  1. Analyze current query patterns and user needs
  2. Map existing queries to the three core categories
  3. Identify gaps using the completeness assessment methods
  4. Prioritize gap-filling based on business impact
  5. Engage SMEs to create missing utterances incrementally

Case Studies

For detailed examples of applying this guideline to specific domains, see:

1. Industrial Asset Management Case Study

File: case_study_industrial_asset_management.md Domain: Building automation and HVAC systems Focus: Chiller and AHU equipment monitoring, predictive maintenance Dataset: 152 utterances evolved incrementally with SME input Status: Comprehensive production example showing real-world development process

Key Highlights:

2. Wind Turbine Operations Case Study

File: case_study_wind_turbine.md Domain: Renewable energy and wind power generation Focus: Wind turbine monitoring, power forecasting, maintenance optimization Dataset: 30 utterances designed complete from the start Status: Complete demonstration example showing balanced framework application

Key Highlights:

Comparison: The two case studies complement each other - Industrial Asset Management shows organic growth in a real project, while Wind Turbine demonstrates planned, balanced design from inception. Both validate the guideline’s cross-domain applicability.

This site is open source. Improve this page.