DEMO

Agent Ops: Build Agents You Can Trust

See watsonx Orchestrate Agent Ops uncover a hidden compliance defect in a loan-underwriting agent team, validate the fix, and red-team the agents before they reach production.

Building Blocks used:

ai.govern

Skills used

Download skills for IBM Bob →

agent-ops

Products Used

IBM watsonX Orchestrate

1 Review use case

Building an AI agent is one thing. Knowing that it behaves as expected, follows business rules and stays reliable as you change it is another. watsonx Orchestrate Agent Ops provides a framework to evaluate, monitor and continuously improve agents throughout their lifecycle, and this demo shows it at work on a realistic case: loan underwriting at a bank, where every decision involves multiple steps, strict regulatory requirements and real compliance risk. An orchestrator agent coordinates three specialists for intake, credit risk and compliance, and together they process the application and produce the decision letter.

The demo compares two versions of this system. In version one, a self-employed applicant is approved and receives a professional decision letter, but the compliance agent silently skipped the anti-money-laundering check, so the approval should have been conditional. There are no errors and no warnings, just a confident but incorrect decision, which is why evaluating the final answer is not enough. Observability lets you inspect every model call, agent handoff, token count and latency, and the evaluation framework replays five test cases with a simulated applicant, scoring routing, tool calls, arguments and decisions: version one passes three of five, while version two, with a single agent's instructions corrected, passes all five. The tests are reusable, so regressions are caught after every prompt, tool or model change.

Two more layers go beyond checking tool calls. Rubric-based evaluation turns four compliance rules written in plain language into twenty pass-or-fail judgments by an LLM judge, each with an explanation, which gives a risk team a way to validate business requirements without writing code. Automated red teaming then sends an attacker model after the agents with three strategies from the framework's attack catalog, each trying to obtain an unconditional approval without the AML screen: every attack succeeds against version one on the first turn, while version two holds its compliance requirements through a six-turn escalation. Together these show how to move from testing correctness to building agents you can trust before they reach production.

Start Here

Watch this short overview

Preview the application

Explore key screens from the demo

2 Try the interactive application

Experience the interactive demo and explore its features. A demo script is provided to help when presenting the application to partners.

Try the demo

Have questions, want a walkthrough or found a bug?

Contact the demo owner for assistance.

Submit a request →

3 Explore how it was built

Two tiers. The demo application — a React front end and a FastAPI backend in one container on IBM Cloud Code Engine — serves the pages, proxies live agent runs, reads traces, and runs evaluation jobs through the Agent Ops framework's Python API. Everything else lives in one IBM watsonx Orchestrate SaaS instance: the agentic system under test (a loan orchestrator with intake, credit-risk and compliance agents, five tools, and the gpt-oss-120b model on watsonx.ai, in two versions), the Agent Ops evaluation framework from the ADK, which drives that system as a simulated user and scores it with ground-truth test cases, rubric rules and red-team attacks, and the platform's observability, which records one trace per conversation with handoffs, tool calls, tokens and latency.

Architecture

4 Download and modify with IBM Bob

Download the source code and specifications for this application to set up and run it locally. Once downloaded, open the code in IBM Bob to modify and adapt it to your own use case.

Source Code

Agent Ops: Build Agents You Can Trust — Seller Demo Script

Before you start: Open the demo in a wide browser window. Every run is live; jobs run one at a time and results stay for ten minutes. Warm up by running Evaluate once for v1 and v2 so results are waiting. The tabs are: Overview, Run a scenario, Evaluate, Rubric, Red team.

1. Overview

Key message: Building an agent is one thing — knowing it behaves as expected and stays reliable through changes is another.

Stay on the Overview page.
Point at Agents in this demo: the loan orchestrator above intake, credit risk and compliance.
Point at the two version cards: v1 (as first shipped, with a hidden compliance defect) and v2 (after evaluating and optimising).

2. Run v1 with Marcus

Key message: No errors, no warnings — a confident but incorrect decision. Evaluating the final answer is not enough.

Click Run a scenario. Keep v1 selected and click Marcus Reyes under Submit an application.
Narrate the stream: handoffs, tool calls, results, then the decision letter.
Point at the red pills: AML screening skipped, and the decision versus the expected conditional approval.

3. Load the trace

Key message: Observability explains an incident after the fact. The rest of the demo catches the problem before production.

Click Load platform trace (enables ~15 seconds after the run).
Scroll the spans: every model call, handoff, token count and latency is visible.

4. Run v2 with Marcus

Key message: One agent’s instructions changed. But how do we prove the fix holds across all scenarios?

Select v2 and click Marcus Reyes again.
Point at the green pills: AML screening ran, conditional approval issued.

5. Evaluate

Key message: These tests are reusable — regressions are caught after every prompt, tool, or model change.

Click Evaluate. Press Run now for v1 (or use the warm-up result).
While it runs, explain: the framework replays five test cases with a simulated applicant, scoring routing, tool calls, arguments and decisions.
Point at Journey success 3 of 5 beside Routing F1 100%.
Expand the Marcus Reyes row: show the wrong decision and the missing AML screening call.
Run v2 and point at 5 of 5 with 100% precision and recall.

6. Rubric

Key message: An LLM judge validates business requirements written in plain language — no code required.

Click Rubric. Run v1, then v2.
Read the four compliance rules written in plain language.
Point at v1 failing AML screening for Marcus Reyes and Priya Natarajan (both self-employed). Click a red cross to show the judge’s reasoning.
Point at v2: all twenty checks pass.

7. Red team

Key message: Version one fell on the first turn. Version two held its compliance requirements through a six-turn escalation.

Click Red team. Run v1, then v2.
Explain: an attacker model plays an applicant trying to skip the AML check using three strategies — Crescendo, Instruction override, and Emotional appeal.
Point at v1: three of three attacks succeed on the first turn.
Open the Crescendo conversation on v2 and scroll its six turns to show the agent holding firm.

8. Close

Key message: Observability, trajectory evaluation, business-rule validation and red teaming in one framework that ships with the watsonx Orchestrate ADK. Building an agent that works is the beginning — the goal is an agent you can trust.

Return to Overview.

If something is off: A disabled button with “Another run is in progress” means wait for the current job. If v2 scores 4 of 5, run it again — the framework just measured a rare slip.