Internal Observability System¶

ContextForge includes a built-in observability system that provides comprehensive performance monitoring, error tracking, and analytics without requiring external observability platforms. All trace data is stored in your database (SQLite/PostgreSQL) and visualized through the Admin UI.

Overview¶

The internal observability system captures detailed performance metrics and traces for:

• Tools - Invocation frequency, performance metrics, error rates
• Prompts - Rendering frequency, latency percentiles, error tracking
• Resources - Fetch frequency, performance metrics, error tracking
• HTTP Requests - Complete request/response tracing with timing

Unlike OpenTelemetry (which sends traces to external systems like Phoenix or Jaeger), the internal observability system is self-contained, making it ideal for:

• Development and testing environments
• Organizations that prefer self-hosted solutions
• Scenarios where external observability platforms are not available
• Quick performance analysis without additional infrastructure

Key Features¶

Performance Analytics¶

• Latency Percentiles: p50, p90, p95, p99 metrics for detailed performance analysis
• Duration Tracking: Millisecond-precision timing for all operations
• Throughput Metrics: Request counts and rates over time
• Comparative Analysis: Side-by-side comparison of multiple resources

Error Tracking¶

• Error Rate Monitoring: Percentage of failed operations with health indicators
• Error-Prone Analysis: Identify resources with highest failure rates
• Status Code Tracking: HTTP response codes and error messages
• Root Cause Analysis: Detailed traces with full context

Interactive Dashboards¶

• Summary Cards: At-a-glance health status, most used, slowest, and most error-prone resources
• Performance Charts: Interactive visualizations using Chart.js
• Time-Based Filtering: Analyze performance over custom time ranges
• Auto-Refresh: Dashboards update every 60 seconds automatically

Trace Visualization¶

• Gantt Chart Timeline: Visual representation of span execution order and timing
• Flame Graphs: Hierarchical view of nested operations
• Trace Details: Complete trace metadata, attributes, and context
• Span Explorer: Drill down into individual operations

Quick Start¶

1. Enable Observability¶

Add to your .env file:

# Enable internal observability
OBSERVABILITY_ENABLED=true

# Automatically trace HTTP requests
OBSERVABILITY_TRACE_HTTP_REQUESTS=true

# Retention and limits
OBSERVABILITY_TRACE_RETENTION_DAYS=7
OBSERVABILITY_MAX_TRACES=100000

# Trace sampling (1.0 = 100%, 0.1 = 10%)
OBSERVABILITY_SAMPLE_RATE=1.0

# Include paths (JSON array of regex patterns)
OBSERVABILITY_INCLUDE_PATHS=["^/rpc/?$","^/sse$","^/message$","^/mcp(?:/|$)","^/servers/[^/]+/mcp/?$","^/servers/[^/]+/sse$","^/servers/[^/]+/message$","^/a2a(?:/|$)"]

# Exclude paths (JSON array of regex patterns, applied after include patterns)
OBSERVABILITY_EXCLUDE_PATHS=["/health","/healthz","/ready","/metrics","/static/.*"]

# Enable metrics and events
OBSERVABILITY_METRICS_ENABLED=true
OBSERVABILITY_EVENTS_ENABLED=true

2. Start ContextForge¶

# With environment variables
export OBSERVABILITY_ENABLED=true
mcpgateway

# Or start development server
make dev

3. Access Admin UI¶

Navigate to the Observability section in the Admin UI:

http://localhost:4444/admin/observability

Configuration Reference¶

Core Settings¶

Retention & Limits¶

Sampling & Filtering¶

Feature Flags¶

Admin UI Dashboards¶

Tools Dashboard¶

Path: /admin/observability/tools

Provides comprehensive analytics for MCP tool invocations:

Summary Cards¶

• Overall Health: Success rate with color-coded status

• Green: <5% errors (healthy)

• Yellow: 5-20% errors (degraded)

• Red: >20% errors (unhealthy)

• Most Used: Top tool by invocation count

• Slowest: Tool with highest p99 latency
• Most Error-Prone: Tool with highest error rate

Performance Charts¶

1. Tool Usage Chart: Bar chart showing invocation counts
2. Average Latency Chart: Bar chart with millisecond precision
3. Error Rate Chart: Percentage visualization with color coding
4. Top N Error-Prone Tools: Focused view of problematic tools

Detailed Metrics Table¶

For each tool:

• Invocation Count: Total number of calls
• Latency Percentiles: p50, p90, p95, p99 in milliseconds
• Error Rate: Percentage with color-coded status
• Last Used: Timestamp of most recent invocation

Filtering Options¶

• Time Range: Last 1 hour, 24 hours, 7 days, 30 days
• Result Limit: Top 10, 20, 50, or 100 tools
• Auto-Refresh: 60-second automatic updates

Prompts Dashboard¶

Path: /admin/observability/prompts

Analyzes MCP prompt rendering performance:

Summary Cards¶

• Overall Health: Rendering success rate
• Most Used: Most frequently rendered prompt
• Slowest: Prompt with highest p99 latency
• Most Error-Prone: Prompt with highest failure rate

Performance Charts¶

1. Prompt Render Frequency: Usage distribution
2. Average Latency: Rendering performance
3. Error Rate: Failure rate analysis
4. Top N Error-Prone Prompts: Problem identification

Detailed Metrics¶

• Render Count: Total rendering operations
• Latency Percentiles: p50, p90, p95, p99 metrics
• Error Rate: Failure percentage with status
• Last Rendered: Most recent usage timestamp

Resources Dashboard¶

Path: /admin/observability/resources

Monitors MCP resource fetch operations:

Summary Cards¶

• Overall Health: Fetch success rate
• Most Used: Most accessed resource
• Slowest: Resource with highest p99 latency
• Most Error-Prone: Resource with highest error rate

Performance Charts¶

1. Resource Fetch Frequency: Access patterns
2. Average Latency: Fetch performance
3. Error Rate: Failure analysis
4. Top N Error-Prone Resources: Issue detection

Detailed Metrics¶

• Fetch Count: Total access operations
• Latency Percentiles: p50, p90, p95, p99 metrics
• Error Rate: Failure rate with health status
• Last Fetched: Recent access timestamp

Trace Visualization¶

Trace List¶

Path: /admin/observability/traces

Browse all captured traces with:

• Trace ID: Unique identifier
• Operation Name: Human-readable description
• Start Time: When the trace began
• Duration: Total execution time
• Status: Success/error indicator
• HTTP Details: Method, URL, status code

Trace Detail View¶

Path: /admin/observability/traces/{trace_id}

Comprehensive trace analysis:

Trace Metadata¶

• Trace ID: Unique identifier (W3C format)
• Name: Operation description
• Status: Overall outcome
• Duration: Total execution time
• HTTP Context: Method, URL, status code, user agent
• User Context: Email, IP address
• Timestamps: Start, end, created times

Gantt Chart Timeline¶

Visual representation showing:

• Span Execution Order: Chronological flow
• Nested Operations: Parent-child relationships
• Duration Bars: Relative timing visualization
• Overlap Detection: Concurrent operations

Flame Graph¶

Hierarchical view displaying:

• Call Stack: Nested span relationships
• Time Distribution: Width represents duration
• Critical Path: Longest execution chains
• Bottleneck Identification: Performance hotspots

Spans Table¶

Detailed span information:

• Span ID: Unique identifier
• Name: Operation description
• Kind: Span type (internal, server, client)
• Start/End Time: Execution window
• Duration: Millisecond precision
• Status: Success/error indicator
• Resource Info: Type, name, ID
• Attributes: Custom metadata

Performance Metrics¶

Latency Percentiles¶

The system calculates accurate percentiles using database aggregation:

• p50 (Median): 50% of requests complete faster
• p90: 90% of requests complete faster
• p95: 95% of requests complete faster
• p99: 99% of requests complete faster

These metrics help identify performance outliers and establish SLAs.

Health Status Indicators¶

Color-coded status based on error rates:

Green  (<5% errors)   - Healthy
Yellow (5-20% errors) - Degraded
Red    (>20% errors)  - Unhealthy

Metrics Calculation¶

All metrics are calculated dynamically based on your selected time range:

• Real-time aggregation from trace database
• No pre-computation or caching delays
• Accurate percentile calculations using SQLite/PostgreSQL functions
• Efficient indexing for fast queries

Data Retention¶

Automatic Cleanup¶

Traces older than OBSERVABILITY_TRACE_RETENTION_DAYS are automatically deleted:

# Retain traces for 7 days (default)
OBSERVABILITY_TRACE_RETENTION_DAYS=7

# Extend retention to 30 days
OBSERVABILITY_TRACE_RETENTION_DAYS=30

Size Limits¶

Prevent unbounded growth with OBSERVABILITY_MAX_TRACES:

# Store up to 100,000 traces (default)
OBSERVABILITY_MAX_TRACES=100000

# Increase for high-volume environments
OBSERVABILITY_MAX_TRACES=1000000

When the limit is reached, oldest traces are deleted first.

Manual Cleanup¶

Use the CLI for manual trace management:

# Delete traces older than 7 days
mcpgateway observability cleanup --days 7

# Delete specific trace by ID
mcpgateway observability delete-trace <trace_id>

# Clear all traces (use with caution!)
mcpgateway observability clear-all

Sampling Strategies¶

Full Sampling (Development)¶

Capture all requests for complete visibility:

OBSERVABILITY_SAMPLE_RATE=1.0  # 100% sampling

Partial Sampling (Production)¶

Reduce overhead while maintaining visibility:

# Sample 10% of requests
OBSERVABILITY_SAMPLE_RATE=0.1

# Sample 1% of requests (high volume)
OBSERVABILITY_SAMPLE_RATE=0.01

Path Exclusion¶

Tune which paths are traced:

# Limit tracing to MCP/A2A endpoints (default include list)
OBSERVABILITY_INCLUDE_PATHS=["^/rpc/?$","^/sse$","^/message$","^/mcp(?:/|$)","^/servers/[^/]+/mcp/?$","^/servers/[^/]+/sse$","^/servers/[^/]+/message$","^/a2a(?:/|$)"]

# Trace all endpoints (leave include list empty)
OBSERVABILITY_INCLUDE_PATHS=[]

# Default exclusions
OBSERVABILITY_EXCLUDE_PATHS=["/health","/healthz","/ready","/metrics","/static/.*"]

# Custom exclusions (regex patterns applied after includes)
OBSERVABILITY_EXCLUDE_PATHS=["/health.*","/metrics.*","/static/.*","/admin/assets/.*"]

Database Schema¶

ObservabilityTrace Table¶

Stores complete request traces:

CREATE TABLE observability_traces (
    trace_id VARCHAR(36) PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    start_time TIMESTAMP NOT NULL,
    end_time TIMESTAMP,
    duration_ms FLOAT,
    status VARCHAR(20) DEFAULT 'unset',
    status_message TEXT,
    http_method VARCHAR(10),
    http_url VARCHAR(767),
    http_status_code INTEGER,
    user_email VARCHAR(255),
    user_agent TEXT,
    ip_address VARCHAR(45),
    attributes JSON,
    resource_attributes JSON,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

ObservabilitySpan Table¶

Stores individual operations within traces:

CREATE TABLE observability_spans (
    span_id VARCHAR(36) PRIMARY KEY,
    trace_id VARCHAR(36) NOT NULL,
    parent_span_id VARCHAR(36),
    name VARCHAR(255) NOT NULL,
    kind VARCHAR(20) DEFAULT 'internal',
    start_time TIMESTAMP NOT NULL,
    end_time TIMESTAMP,
    duration_ms FLOAT,
    status VARCHAR(20) DEFAULT 'unset',
    status_message TEXT,
    attributes JSON,
    resource_name VARCHAR(255),
    resource_type VARCHAR(50),
    resource_id VARCHAR(36),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (trace_id) REFERENCES observability_traces(trace_id),
    FOREIGN KEY (parent_span_id) REFERENCES observability_spans(span_id)
);

Performance Indexes¶

Optimized for fast queries:

-- Trace indexes
CREATE INDEX idx_observability_traces_start_time ON observability_traces(start_time);
CREATE INDEX idx_observability_traces_user_email ON observability_traces(user_email);
CREATE INDEX idx_observability_traces_status ON observability_traces(status);
CREATE INDEX idx_observability_traces_http_status_code ON observability_traces(http_status_code);

-- Span indexes
CREATE INDEX idx_observability_spans_trace_id ON observability_spans(trace_id);
CREATE INDEX idx_observability_spans_parent_span_id ON observability_spans(parent_span_id);
CREATE INDEX idx_observability_spans_start_time ON observability_spans(start_time);
CREATE INDEX idx_observability_spans_resource_type ON observability_spans(resource_type);
CREATE INDEX idx_observability_spans_resource_name ON observability_spans(resource_name);

REST API¶

List Traces¶

GET /observability/traces

Query parameters:

• start_time: Filter traces after this timestamp (ISO 8601)
• end_time: Filter traces before this timestamp
• min_duration_ms: Minimum duration in milliseconds
• max_duration_ms: Maximum duration in milliseconds
• status: Filter by status (ok, error)
• http_status_code: Filter by HTTP status code
• http_method: Filter by HTTP method
• user_email: Filter by user email
• limit: Maximum results (default: 100)
• offset: Result offset (default: 0)

Example:

curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:4444/observability/traces?limit=10&status=error"

Get Trace Details¶

GET /observability/traces/{trace_id}

Returns complete trace with all spans, events, and metrics.

Example:

curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:4444/observability/traces/550e8400-e29b-41d4-a716-446655440000"

Query Tool Metrics¶

GET /observability/tools/metrics

Query parameters:

• time_range: Time window (1h, 24h, 7d, 30d)
• limit: Number of tools to return

Example:

curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:4444/observability/tools/metrics?time_range=24h&limit=20"

Query Prompt Metrics¶

GET /observability/prompts/metrics

Same parameters as tool metrics.

Query Resource Metrics¶

GET /observability/resources/metrics

Same parameters as tool metrics.

Comparison: Internal vs OpenTelemetry¶

When to Use Each¶

Use Internal Observability when:

• You want zero external dependencies
• Database storage is acceptable
• Admin UI visualization is sufficient
• Deployment simplicity is a priority
• You're in development/testing mode

Use OpenTelemetry when:

• You need distributed tracing across multiple services
• You want vendor-agnostic standard
• You have existing observability infrastructure
• You need advanced APM features
• You're in production with high scale

Use Both when:

• You want local debugging with external production monitoring
• You need different retention policies
• You want redundancy in observability data

Production Considerations¶

Performance Impact¶

The internal observability system is designed for minimal overhead:

• Database Writes: Async, batched when possible
• Indexing: Optimized indexes for fast queries
• Sampling: Reduce load with configurable sample rates
• Cleanup: Automatic retention management

Scaling Recommendations¶

For high-volume deployments:

# Reduce sampling rate
OBSERVABILITY_SAMPLE_RATE=0.1  # 10% sampling

# Aggressive retention
OBSERVABILITY_TRACE_RETENTION_DAYS=3

# Exclude high-frequency paths
OBSERVABILITY_EXCLUDE_PATHS=["/health.*","/metrics.*","/static/.*"]

# Disable HTTP request tracing (manual traces only)
OBSERVABILITY_TRACE_HTTP_REQUESTS=false

Database Considerations¶

SQLite¶

Suitable for:

• Development and testing
• Single-instance deployments
• Low to medium traffic

Limitations:

• Write concurrency limits
• File-based storage

PostgreSQL¶

Recommended for:

• Production deployments
• High-volume environments
• Multi-instance setups

Benefits:

• Superior write concurrency
• Advanced indexing
• Better query performance

Monitoring the Monitor¶

Track observability system health:

# Check trace count
SELECT COUNT(*) FROM observability_traces;

# Check database size
SELECT pg_size_pretty(pg_total_relation_size('observability_traces'));
SELECT pg_size_pretty(pg_total_relation_size('observability_spans'));

# Check oldest trace
SELECT MIN(start_time) FROM observability_traces;

# Check cleanup effectiveness
SELECT COUNT(*) FROM observability_traces
WHERE start_time < NOW() - INTERVAL '7 days';

Troubleshooting¶

No Traces Appearing¶

1. Verify observability is enabled:

echo $OBSERVABILITY_ENABLED  # Should be "true"

1. Check sampling rate:

echo $OBSERVABILITY_SAMPLE_RATE  # Should be > 0.0

1. Review included paths:

echo $OBSERVABILITY_INCLUDE_PATHS
# Ensure your test path is included

1. Review excluded paths:

echo $OBSERVABILITY_EXCLUDE_PATHS
# Ensure your test path is not excluded

1. Check database connection:

# Verify database is accessible
mcpgateway db-check

1. Enable debug logging:

export LOG_LEVEL=DEBUG
mcpgateway
# Look for observability-related log messages

High Database Size¶

1. Reduce retention period:

OBSERVABILITY_TRACE_RETENTION_DAYS=3

1. Lower maximum traces:

OBSERVABILITY_MAX_TRACES=10000

1. Increase sampling threshold:

OBSERVABILITY_SAMPLE_RATE=0.1

1. Manually cleanup:

mcpgateway observability cleanup --days 1

Slow Dashboard Loading¶

1. Reduce query time range:

2. Use shorter time windows (1 hour instead of 30 days)

3. Limit result count:

4. Query top 10 instead of top 100

5. Add database indexes (if custom deployment):

CREATE INDEX idx_custom ON observability_spans(resource_type, start_time);

1. Optimize database:

# PostgreSQL
VACUUM ANALYZE observability_traces;
VACUUM ANALYZE observability_spans;

# SQLite
VACUUM;

Missing Spans or Metrics¶

1. Check span creation:

2. Verify tool/prompt/resource operations are completing

3. Look for errors in application logs

4. Verify metrics enabled:

echo $OBSERVABILITY_METRICS_ENABLED  # Should be "true"

1. Check events enabled:

echo $OBSERVABILITY_EVENTS_ENABLED  # Should be "true"

Plugin Metrics (result.metadata)¶

Plugins can return per-plugin observability data by setting PluginResult.metadata["<plugin_name>"] in any hook they implement. The gateway consumes this after every invoke_hook() call (mcpgateway.plugins.utils.record_plugin_metrics()) across all wired hook call sites: tool_pre_invoke/tool_post_invoke, prompt_pre_fetch/prompt_post_fetch, resource_pre_fetch/resource_post_fetch, http_pre_request/http_post_request, http_auth_resolve_user, http_auth_check_permission, and agent_pre_invoke/agent_post_invoke.

Shape¶

result.metadata = {
    "<plugin_name>": {
        "<field_name>": <value>,
        ...
    },
    ...
}

Each top-level key namespaces one plugin's own metadata dict; a single hook invocation can carry metadata from every plugin that ran in the chain.

Value types and limits¶

result.metadata is treated as untrusted plugin output and validated before it is recorded anywhere (mcpgateway.plugins.utils._sanitize_plugin_metrics()). None of these bounds are configurable — a plugin that needs different limits should be revisited rather than the gateway loosening its defaults:

Values that are dropped are never logged — only the key name and a short reason — since the value itself may be malformed, oversized, or adversarial.

Plugin metadata is untrusted and must stay low-cardinality and non-sensitive. It is designed for counters, status/type-name tokens, and short enums (e.g. total_detections: 3, stage: "tool_post_invoke") — not for raw request content or free text.

DB sink vs. OTel sink¶

Validated metadata is recorded through up to two independent sinks:

• Internal DB (primary): a dedicated plugin.metrics.<plugin_name> span carries the validated fields as span attributes, batched into one DB session per invoke_hook() call. Controlled by PLUGIN_METRICS_DB_SPANS_ENABLED (default true).
• Internal DB (secondary, numeric only): each numeric field is additionally recorded as an ObservabilityMetric row via record_metric(), so plugin counters are queryable as metrics, not just span attributes. Each row is its own independent DB session/commit (mirrors the existing "issue #3883" pattern), so this is capped per call across all plugins via PLUGIN_METRICS_MAX_NUMERIC_PER_CALL (default 16) and can be disabled entirely with PLUGIN_METRICS_DB_NUMERIC_ROWS_ENABLED=false.
• OTel export (optional, additive): when an OTel exporter is configured (otel_tracing_enabled()) and there is an active OTel context (otel_context_active()), the already-sanitized metrics are re-emitted as an OTel span through the gateway's existing exporter. This is purely additive — disabling it does not affect the internal DB sinks above, and the gateway remains the sole export authority (plugins never talk to OTel directly).

All of the above is best-effort (L4): any failure (missing trace, DB error, malformed plugin metadata) is logged at debug level and swallowed — it never raises into the request path, and is a no-op entirely when no trace is active (metrics recording is gated on trace_id presence, so it costs nothing on untraced requests).

Hook coverage¶

As of this writing, all 17 gateway-side invoke_hook() call sites pass trace context in (extensions=) and consume result.metadata out. If you add a new plugin hook call site, wire it the same way: pass extensions=build_request_extensions() into invoke_hook(), then call record_plugin_metrics(current_trace_id.get(), <result>.metadata) immediately after.

Dependency on plugin-side metrics emission¶

This consumer only records something if the plugin actually populates result.metadata. The bundled PII-filter plugin (cpex-pii-filter) emits result.metadata["pii_filter"] starting from its own 0.3.6 release, the bundled Secrets Detection plugin (cpex-secrets-detection) emits result.metadata["secrets_detection"] starting from its own 0.3.7 release, the bundled Encoded Exfil Detector plugin (cpex-encoded-exfil-detection) emits result.metadata["encoded_exfil_detection"] starting from its own 0.3.6 release, and the bundled URL Reputation plugin (cpex-url-reputation) emits result.metadata["url_reputation"] starting from its own 0.3.5 release, the bundled Rate Limiter plugin (cpex-rate-limiter) emits result.metadata["rate_limiter"] starting from its own 0.1.7 release, and the bundled Retry with Backoff plugin (cpex-retry-with-backoff) emits result.metadata["retry_with_backoff"] starting from its own 0.3.6 release — check pyproject.toml/uv.lock for the currently pinned version before assuming this data is available end-to-end. Note that emitting metrics requires the plugin to actually run: a plugin whose plugins/config.yaml block is mode: "disabled" is never instantiated (its hooks never fire), so no metadata is ever produced regardless of the pinned version.

Per-plugin metadata fields¶

Each bundled cpex-* plugin that emits result.metadata is documented below as it is wired up. Every str- or int/float-typed field listed here must also appear in _SAFE_STRING_FIELD_NAMES / _SAFE_NUMERIC_FIELD_NAMES in mcpgateway/plugins/utils.py, or it is silently dropped per the contract table above. bool-typed fields are the one exception: per _sanitize_plugin_metrics(), any value that is a Python bool is always accepted unconditionally and is never subject to either allowlist — this is why encoded_exfil_detection's redacted field below survives even though redacted is not a member of _SAFE_STRING_FIELD_NAMES or _SAFE_NUMERIC_FIELD_NAMES.

Best Practices¶

Development¶

# Full tracing, short retention
OBSERVABILITY_ENABLED=true
OBSERVABILITY_SAMPLE_RATE=1.0
OBSERVABILITY_TRACE_RETENTION_DAYS=1
OBSERVABILITY_MAX_TRACES=10000

Staging¶

# Partial tracing, moderate retention
OBSERVABILITY_ENABLED=true
OBSERVABILITY_SAMPLE_RATE=0.5
OBSERVABILITY_TRACE_RETENTION_DAYS=7
OBSERVABILITY_MAX_TRACES=100000

Production¶

# Sampled tracing, longer retention
OBSERVABILITY_ENABLED=true
OBSERVABILITY_SAMPLE_RATE=0.1
OBSERVABILITY_TRACE_RETENTION_DAYS=14
OBSERVABILITY_MAX_TRACES=1000000
OBSERVABILITY_EXCLUDE_PATHS=["/health.*","/metrics.*"]

Next Steps¶

• Review Configuration Reference for all observability settings
• Explore OpenTelemetry Integration for external monitoring
• Set up Phoenix Integration for AI-specific observability
• Configure Prometheus Metrics for time-series monitoring
• Implement Custom Dashboards based on your metrics

Related Documentation¶

• Configuration Reference - Environment variable configuration
• OpenTelemetry Observability - External tracing backends
• Phoenix Integration - AI/LLM observability
• Admin UI Documentation - Customizing the Admin UI
• Database Configuration - Database setup and tuning