📊 Metadata Tracking & Audit Trails¶

MCP Gateway provides comprehensive metadata tracking for all entities (Tools, Resources, Prompts, Servers, Gateways) to enable enterprise-grade audit trails, compliance monitoring, and operational troubleshooting.

🎯 Overview¶

Every entity in MCP Gateway now includes detailed metadata about: - Who created or modified the entity - When the operation occurred - From where (IP address, user agent) - How it was created (UI, API, bulk import, federation) - Source tracking for federated entities and bulk operations

📊 Metadata Fields¶

All entities include the following metadata fields:

🖥️ Viewing Metadata¶

Admin UI¶

Metadata is displayed in the detail view modals for all entity types:

1. Navigate to any entity list (Tools, Resources, Prompts, Servers, Gateways)
2. Click "View" on any entity
3. Scroll down to the "Metadata" section

Example metadata display:

┌─ Metadata ──────────────────────────────────────┐
│ Created By:      admin                           │
│ Created At:      1/15/2024, 10:30:00 AM        │
│ Created From:    192.168.1.100                  │
│ Created Via:     ui                              │
│ Last Modified By: alice                          │
│ Last Modified At: 1/16/2024, 2:22:00 PM        │
│ Version:         3                               │
│ Import Batch:    N/A                             │
└─────────────────────────────────────────────────┘

API Responses¶

All entity read endpoints include metadata fields in JSON responses:

# Get tool with metadata
curl -H "Authorization: Bearer $TOKEN" \
     http://localhost:4444/tools/abc123

{
  "id": "abc123",
  "name": "example_tool",
  "description": "Example tool",
  "createdBy": "admin",
  "createdAt": "2024-01-15T10:30:00Z",
  "createdFromIp": "192.168.1.100",
  "createdVia": "ui",
  "createdUserAgent": "Mozilla/5.0...",
  "modifiedBy": "alice",
  "modifiedAt": "2024-01-16T14:22:00Z",
  "version": 3,
  "importBatchId": null,
  "federationSource": null,
  ...
}

🔍 Metadata by Source Type¶

Manual Creation (UI/API)¶

• created_via: "ui" or "api"
• created_by: Authenticated username
• created_from_ip: Client IP address
• federation_source: null
• import_batch_id: null

Bulk Import Operations¶

• created_via: "import"
• import_batch_id: UUID linking related imports
• created_by: User who initiated import
• federation_source: null

Federation (MCP Server Discovery)¶

• created_via: "federation"
• federation_source: Source gateway name
• created_by: User who registered the gateway
• import_batch_id: null

Legacy Entities (Pre-Metadata)¶

• All metadata fields: null
• UI displays: "Legacy Entity", "Pre-metadata"
• version: 1 (automatically assigned)

🛡️ Authentication Compatibility¶

Metadata tracking works seamlessly across all authentication modes:

With Authentication (AUTH_REQUIRED=true)¶

# Example: User "admin" creates a tool
{
  "createdBy": "admin",
  "createdVia": "api",
  "createdFromIp": "192.168.1.100"
}

Without Authentication (AUTH_REQUIRED=false)¶

# Example: Anonymous creation
{
  "createdBy": "anonymous",
  "createdVia": "api",
  "createdFromIp": "192.168.1.100"
}

JWT vs Basic Authentication¶

• JWT Authentication: Extracts username from token payload (username or sub field)
• Basic Authentication: Uses provided username directly
• Both formats handled gracefully by the extract_username() utility

🔄 Version Tracking¶

Each entity maintains a version number that increments on modifications:

# Initial creation
POST /tools -> version: 1

# First update
PUT /tools/123 -> version: 2

# Second update
PUT /tools/123 -> version: 3

Version tracking helps identify: - Configuration drift between environments - Change frequency for troubleshooting - Rollback points for recovery scenarios

📈 Use Cases¶

Security Auditing¶

• Track who created/modified sensitive configurations
• Identify unauthorized changes by IP address
• Monitor bulk import operations for compliance

Operational Troubleshooting¶

• Trace entity origins during incident response
• Identify batch operations that may have caused issues
• Understand federation dependencies between gateways

Compliance Reporting¶

• Generate audit reports for regulatory requirements
• Track change management processes
• Demonstrate access controls and change attribution

Development & Testing¶

• Identify test vs production entities
• Track deployment-specific configurations
• Monitor cross-environment migrations

🔧 Configuration¶

No Additional Setup Required¶

Metadata tracking is automatically enabled for all new installations and upgrades:

• Database migration runs automatically on startup
• Existing entities show graceful fallbacks for missing metadata
• No environment variables needed - uses existing AUTH_REQUIRED setting

Proxy Support¶

Metadata capture automatically handles reverse proxy scenarios:

# Respects X-Forwarded-For headers
X-Forwarded-For: 203.0.113.1, 192.168.1.1, 127.0.0.1
# Records: created_from_ip = "203.0.113.1" (original client)

Privacy Considerations¶

The system captures IP addresses and user agents for audit purposes:

• IP addresses: Consider GDPR/privacy implications for EU deployments
• User agents: May contain personally identifiable information
• Data retention: Define policies for metadata archival
• Access control: Metadata follows same permissions as parent entity

🚀 Migration Guide¶

Upgrading Existing Deployments¶

1. Automatic Migration

   # Migration runs automatically on startup
   # Or run manually:
   alembic upgrade head
   

2. Verify Migration

3. Check admin UI - all entities show metadata sections
4. API responses include new metadata fields

5. Legacy entities display gracefully

6. No Downtime Required

7. All metadata columns are nullable
8. Existing functionality unmodified
9. Gradual adoption of metadata features

Metadata Backfill (Optional)¶

For enhanced audit trails, optionally backfill known metadata:

-- Backfill system-created entities
UPDATE tools SET
    created_by = 'system',
    created_via = 'migration',
    version = 1
WHERE created_by IS NULL;

-- Similar for other entity tables
UPDATE gateways SET created_by = 'system', created_via = 'migration', version = 1 WHERE created_by IS NULL;
UPDATE servers SET created_by = 'system', created_via = 'migration', version = 1 WHERE created_by IS NULL;
UPDATE prompts SET created_by = 'system', created_via = 'migration', version = 1 WHERE created_by IS NULL;
UPDATE resources SET created_by = 'system', created_via = 'migration', version = 1 WHERE created_by IS NULL;

🔮 Future Enhancements¶

Enhanced Audit Features¶

• Change history tracking - Before/after state comparison
• Metadata-based filtering - Search entities by creator, date, source
• Audit log export - Generate compliance reports
• Custom metadata fields - User-defined entity attributes

Cross-Gateway Features¶

• Metadata synchronization across federated gateways
• Trust scoring based on metadata quality
• Provenance tracking for complex federation scenarios

Analytics Integration¶

• Usage pattern analysis from metadata
• Creator activity dashboards
• Import/export trend monitoring

📋 API Examples¶

Creating Entities with Metadata¶

Metadata is captured automatically - no additional parameters needed:

# Create tool - metadata captured automatically
curl -X POST http://localhost:4444/tools \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "example_tool",
    "url": "http://example.com/api",
    "integration_type": "REST",
    "request_type": "GET"
  }'

# Response includes metadata
{
  "id": "abc123",
  "name": "example_tool",
  "createdBy": "admin",
  "createdAt": "2024-01-15T10:30:00Z",
  "createdVia": "api",
  "version": 1,
  ...
}

Filtering by Metadata (Future)¶

# Future enhancement - filter by creator
GET /tools?created_by=admin

# Filter by creation method
GET /tools?created_via=federation

# Filter by date range
GET /tools?created_after=2024-01-01&created_before=2024-01-31

❓ FAQ¶

Q: Will this affect existing deployments?¶

A: No breaking changes. Existing entities show graceful fallbacks, all APIs work unmodified.

Q: What happens if authentication is disabled?¶

A: Metadata still works - created_by will be "anonymous" instead of a username.

Q: How much storage does metadata require?¶

A: Minimal - approximately 13 additional nullable text columns per entity.

Q: Can I disable metadata tracking?¶

A: Not currently - metadata is core to the audit system. All fields are optional and backwards compatible.

Q: How do I export metadata for compliance?¶

A: Use the standard export functionality - metadata is included in all entity exports.

This comprehensive metadata system provides enterprise-grade audit capabilities while maintaining full backwards compatibility and operational simplicity.