Gateway Tuning Guide¶

This page collects practical levers for squeezing the most performance, reliability, and observability out of MCP Gateway-no matter where you run the container (Code Engine, Kubernetes, Docker Compose, Nomad, etc.).

TL;DR

1. Tune the runtime environment via .env and configure mcpgateway to use PostgreSQL and Redis.
2. Adjust Gunicorn workers & time-outs in gunicorn.conf.py.
3. Right-size CPU/RAM for the container or spin up more instances (with shared Redis state) and change the database settings (ex: connection limits).
4. Benchmark with hey (or your favourite load-generator) before & after. See also: performance testing guide

1 - Environment variables (.env)¶

Redis Connection Pool Tuning¶

When using CACHE_TYPE=redis, tune the connection pool for your workload:

High-concurrency production settings:

REDIS_MAX_CONNECTIONS=100
REDIS_SOCKET_TIMEOUT=1.0
REDIS_SOCKET_CONNECT_TIMEOUT=1.0
REDIS_HEALTH_CHECK_INTERVAL=15

Tip Any change here requires rebuilding or restarting the container if you pass the file with --env-file.

2 - Gunicorn settings (gunicorn.conf.py)¶

Edit the file before building the image, then redeploy.

2b - Uvicorn Performance Extras¶

MCP Gateway uses uvicorn[standard] which includes high-performance components that are automatically detected and used:

Automatic Detection¶

When Gunicorn spawns Uvicorn workers, these components are automatically detected:

# Verify extras are installed
pip list | grep -E "uvloop|httptools|websockets|watchfiles"

# Expected output (Linux/macOS):
# httptools    0.6.x
# uvloop       0.21.x
# websockets   15.x.x
# watchfiles   1.x.x

Platform Notes¶

• Linux/macOS: Full performance benefits (uvloop + httptools)
• Windows: httptools provides benefits; uvloop unavailable (graceful fallback to asyncio)

Performance Impact¶

Combined improvements from uvloop and httptools:

Note: These optimizations are transparent - no code or configuration changes needed.

2c - Granian (Alternative HTTP Server)¶

MCP Gateway supports two HTTP servers: - Gunicorn + Uvicorn (default) - Battle-tested, mature, excellent stability - Granian (alternative) - Rust-based, native HTTP/2, lower memory

Usage¶

# Local development
make serve                    # Gunicorn + Uvicorn (default)
make serve-granian            # Granian (alternative)
make serve-granian-http2      # Granian with HTTP/2 + TLS

# Container with Gunicorn (default)
make container-run
make container-run-gunicorn-ssl

# Container with Granian (alternative)
make container-run-granian
make container-run-granian-ssl

# Docker Compose (default uses Gunicorn)
docker compose up

Switching HTTP Servers¶

The HTTP_SERVER environment variable controls which server to use:

# Docker/Podman - use Gunicorn (default)
docker run mcpgateway/mcpgateway

# Docker/Podman - use Granian
docker run -e HTTP_SERVER=granian mcpgateway/mcpgateway

# Docker Compose - set in environment section
environment:
  - HTTP_SERVER=gunicorn  # default
  # - HTTP_SERVER=granian # alternative

Configuration¶

Performance tuning profiles:

# High-throughput (fewer workers, more threads per worker)
GRANIAN_WORKERS=4 GRANIAN_RUNTIME_THREADS=4 make serve

# High-concurrency (more workers, max backpressure)
GRANIAN_WORKERS=16 GRANIAN_BACKPRESSURE=1024 GRANIAN_BACKLOG=4096 make serve

# Memory-constrained (fewer workers)
GRANIAN_WORKERS=2 make serve

# Force HTTP/1 only (avoids HTTP/2 overhead)
GRANIAN_HTTP=1 make serve

Notes: - On Python 3.12+, the Rust task implementation is unavailable; asyncio is used automatically - uvloop provides best performance on Linux/macOS - Increase GRANIAN_BACKLOG and GRANIAN_BACKPRESSURE for high-concurrency workloads

Backpressure for Overload Protection¶

Granian's native backpressure prevents unbounded request queuing during overload. When the server reaches capacity, excess requests receive immediate 503 responses instead of waiting in a queue (which can cause memory exhaustion or cascading timeouts).

How it works:

Incoming Request
       │
       ▼
┌──────────────────────────────────┐
│  Granian Worker (1 of N)         │
│                                  │
│  current_requests < BACKPRESSURE?│
│      │                           │
│      ├── YES → Process request   │
│      │                           │
│      └── NO  → Immediate 503     │
│               (no queuing)       │
└──────────────────────────────────┘

Capacity calculation:

Total capacity = GRANIAN_WORKERS × GRANIAN_BACKPRESSURE

Example with recommended settings:
  Workers: 16
  Backpressure: 64
  Total: 16 × 64 = 1024 concurrent requests

  Requests 1-1024: Processed normally
  Request 1025+: Immediate 503 Service Unavailable

Recommended production settings:

# docker-compose.yml or Kubernetes
environment:
  - HTTP_SERVER=granian
  - GRANIAN_WORKERS=16
  - GRANIAN_BACKLOG=4096        # OS socket queue for pending connections
  - GRANIAN_BACKPRESSURE=64     # Per-worker limit (16×64=1024 total)

Benefits over unbounded queuing:

When to Use Granian¶

Performance Comparison¶

Note: Always benchmark with your specific workload before switching servers.

Real-World Performance (Database-Bound Workload)¶

Under load testing with 2500 concurrent users against PostgreSQL:

Key Finding: When the database is the bottleneck, both servers achieve similar throughput. The main differences are:

• Memory: Gunicorn uses 32% less RAM (fork-based model with copy-on-write)
• CPU: Granian uses 8% less CPU (more efficient HTTP parsing in Rust)
• Stability: Granian handles overload gracefully (backpressure), Gunicorn queues indefinitely

Recommendation:

3 - Container resources¶

Always test with your workload; JSON-RPC payload size and backend model latency change the equation.

To change your database connection settings, see the respective documentation for your selected database or managed service. For example, when using IBM Cloud Databases for PostgreSQL - you can raise the maximum number of connections.

4 - Performance testing¶

4.1 Tooling: hey¶

Install one of:

brew install hey            # macOS
sudo apt install hey         # Debian/Ubuntu
# or build from source
go install github.com/rakyll/hey@latest  # $GOPATH/bin must be in PATH

4.2 Sample load-test script (tests/hey.sh)¶

#!/usr/bin/env bash
# Run 10 000 requests with 200 concurrent workers.
JWT="$(cat jwt.txt)"   # <- place a valid token here
hey -n 10000 -c 200 \
    -m POST \
    -T application/json \
    -H "Authorization: Bearer ${JWT}" \
    -D tests/hey/payload.json \
    http://localhost:4444/rpc

Payload (tests/hey/payload.json)

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "convert_time",
  "params": {
    "source_timezone": "Europe/Berlin",
    "target_timezone": "Europe/Dublin",
    "time": "09:00"
  }
}

4.3 Reading the output¶

hey prints latency distribution, requests/second, and error counts. Focus on:

• 99^th percentile latency - adjust timeout if it clips.
• Errors - 5xx under load often mean too few workers or DB connections.
• Throughput (RPS) - compare before/after tuning.

4.4 Common bottlenecks & fixes¶

5 - Logging & observability¶

• Set loglevel = "debug" in gunicorn.conf.py during tests; revert to info in prod.
• Forward stdout/stderr from the container to your platform's log stack (e.g. kubectl logs, docker logs).
• Expose /metrics via a Prometheus exporter (planned) for request timing & queue depth; track enablement in the project roadmap.

6 - MCP Session Pool Tuning¶

The MCP session pool maintains persistent connections to upstream MCP servers, providing 10-20x latency improvement for repeated tool calls from the same user.

MCP_SESSION_POOL_ENABLED=true

When to Enable Pooling¶

Configuration Variables¶

Recommended Production Settings¶

# Baseline settings for authenticated deployments (low-to-moderate traffic)
MCP_SESSION_POOL_ENABLED=true
MCP_SESSION_POOL_MAX_PER_KEY=10
MCP_SESSION_POOL_TTL=300
MCP_SESSION_POOL_TRANSPORT_TIMEOUT=30

# High-concurrency settings (1000+ concurrent users)
MCP_SESSION_POOL_ENABLED=true
MCP_SESSION_POOL_MAX_PER_KEY=200          # 50-200 for high concurrency
MCP_SESSION_POOL_TTL=300
MCP_SESSION_POOL_TRANSPORT_TIMEOUT=30
MCP_SESSION_POOL_ACQUIRE_TIMEOUT=60       # Longer timeout under load

# Ensure identity headers are present
ENABLE_HEADER_PASSTHROUGH=true
DEFAULT_PASSTHROUGH_HEADERS="Authorization,X-Tenant-Id,X-User-Id,X-API-Key"

Session Isolation¶

Sessions are isolated by a composite key: (URL, identity_hash, transport_type). Identity is derived from authentication headers (Authorization, X-Tenant-ID, X-User-ID, X-API-Key, Cookie).

Key security considerations:

1. Anonymous Pooling: When no identity headers are present, identity collapses to "anonymous" and all such requests share sessions. This is safe only if upstream MCP servers are stateless.

2. Shared Credentials: With OAuth Client Credentials or static API keys, all users share the same identity hash. Only safe if the upstream MCP server has no per-user state.

3. Header Passthrough: If gateway auth is disabled (AUTH_REQUIRED=false), enable header passthrough to preserve user identity:

   ENABLE_HEADER_PASSTHROUGH=true
   DEFAULT_PASSTHROUGH_HEADERS="Authorization,X-Tenant-Id,X-User-Id"
   

Long-Running Tools¶

The transport timeout applies to all HTTP operations, not just connection establishment. For tools that take longer than 30 seconds:

# Increase for long-running tools
MCP_SESSION_POOL_TRANSPORT_TIMEOUT=120

Health Check Timeout Trade-offs¶

Pool staleness checks use MCP_SESSION_POOL_TRANSPORT_TIMEOUT (default 30s) for session acquisition. When MCP_SESSION_POOL_EXPLICIT_HEALTH_RPC=true, the explicit RPC call uses HEALTH_CHECK_TIMEOUT (default 5s).

Behavior summary:

Trade-off: The 30s transport timeout allows long-running tools to complete but means unhealthy sessions may take longer to detect. If you need faster failure detection:

# Stricter health checks (5s timeout for explicit RPC)
MCP_SESSION_POOL_EXPLICIT_HEALTH_RPC=true
HEALTH_CHECK_TIMEOUT=5

# Or reduce transport timeout (affects all operations)
MCP_SESSION_POOL_TRANSPORT_TIMEOUT=10

Circuit Breaker Behavior¶

The circuit breaker is keyed by URL only (not per-identity). After MCP_SESSION_POOL_CIRCUIT_BREAKER_THRESHOLD consecutive session creation failures for a URL, the circuit opens and all requests fail fast for MCP_SESSION_POOL_CIRCUIT_BREAKER_RESET seconds.

Note: Only session creation failures (connection refused, SSL errors) trip the circuit. Tool call failures do not affect the circuit breaker.

Monitoring¶

Monitor pool performance via the metrics endpoint:

curl -u admin:changeme http://localhost:4444/admin/mcp-pool/metrics

Response includes: - total_sessions_created / total_sessions_reused: Pool hit ratio - pool_hits / pool_misses: Cache effectiveness - active_sessions: Current utilization - circuit_breaker_states: Per-URL circuit status

Operational Checklist¶

Before enabling pooling in production:

• Confirm upstream MCP servers are stateless for any shared/anonymous access
• Verify identity headers are present and stable
• Validate tool call durations vs MCP_SESSION_POOL_TRANSPORT_TIMEOUT
• Ensure tracing headers are not relied upon in pooled sessions

After enabling pooling:

• Monitor pool metrics at /admin/mcp-pool/metrics
• Watch for increased tool timeouts or unexpected auth failures
• Verify correlation IDs in upstream logs (note: per-request headers are stripped from pooled sessions)

7 - Nginx Reverse Proxy Tuning¶

When deploying MCP Gateway behind nginx (as in the default docker-compose.yml), several optimizations can significantly improve performance under load.

Admin UI Caching¶

Admin pages use Jinja2 template rendering which is CPU-intensive under high concurrency. The default nginx configuration enables short-TTL caching with multi-tenant isolation:

Performance impact (4000 concurrent users):

Multi-Tenant Cache Safety¶

The cache key includes all authentication credentials to ensure user isolation:

proxy_cache_key "$scheme$request_method$host$request_uri$is_args$args$http_authorization$cookie_jwt_token$cookie_access_token";

This ensures:

• Bearer token auth ($http_authorization): API clients get isolated caches
• Primary session cookie ($cookie_jwt_token): Browser users get isolated caches
• Alternative auth cookie ($cookie_access_token): Fallback auth method also isolated

Verifying Cache Behavior¶

Check the X-Cache-Status header to verify caching is working:

curl -I http://localhost:8080/admin/ -b "jwt_token=..." | grep X-Cache
# X-Cache-Status: MISS  (first request)
# X-Cache-Status: HIT   (subsequent requests within 5s)
# X-Cache-Status: STALE (background refresh in progress)

Disabling Admin Caching¶

If you need real-time admin data or have concerns about caching, modify infra/nginx/nginx.conf:

location /admin {
    proxy_cache off;
    add_header Cache-Control "no-cache, no-store, must-revalidate" always;
    # ... rest of config
}

8 - High-Concurrency Production Tuning¶

This section covers comprehensive tuning for deployments handling 1000+ concurrent users. These settings have been tested under load with 6500 concurrent users.

8.1 Database Connection Pool (SQLAlchemy)¶

The gateway's internal connection pool manages connections between the application and PgBouncer (or PostgreSQL directly).

Example high-concurrency configuration:

# With PgBouncer (recommended)
DB_POOL_CLASS=queue
DB_POOL_PRE_PING=true
DB_POOL_SIZE=20
DB_MAX_OVERFLOW=10
DB_POOL_TIMEOUT=60
DB_POOL_RECYCLE=60    # Half of PgBouncer CLIENT_IDLE_TIMEOUT (120s)

Common errors and solutions:

8.2 PgBouncer Connection Pooler¶

PgBouncer multiplexes many application connections into fewer PostgreSQL connections, dramatically reducing database overhead.

Client-Side Settings (from gateway workers)¶

Server-Side Settings (to PostgreSQL)¶

Timeout Settings¶

Transaction Reset Settings¶

Example PgBouncer configuration (docker-compose.yml):

pgbouncer:
  image: edoburu/pgbouncer:latest
  environment:
    - DATABASE_URL=postgres://postgres:password@postgres:5432/mcp
    - POOL_MODE=transaction
    # Client limits
    - MAX_CLIENT_CONN=5000
    - DEFAULT_POOL_SIZE=600
    - MIN_POOL_SIZE=100
    - RESERVE_POOL_SIZE=150
    # Server limits
    - MAX_DB_CONNECTIONS=700
    - SERVER_LIFETIME=1800
    - SERVER_IDLE_TIMEOUT=600
    # Timeouts
    - QUERY_WAIT_TIMEOUT=60
    - CLIENT_IDLE_TIMEOUT=120
    - IDLE_TRANSACTION_TIMEOUT=60
    # Reset
    - SERVER_RESET_QUERY=DISCARD ALL
    - SERVER_RESET_QUERY_ALWAYS=1
  ulimits:
    nofile:
      soft: 65536
      hard: 65536

8.3 Container File Descriptor Limits (ulimits)¶

Each network connection requires a file descriptor. Containers default to 1024 soft limit, which is insufficient for high concurrency.

docker-compose.yml example:

services:
  pgbouncer:
    ulimits:
      nofile:
        soft: 65536
        hard: 65536

  postgres:
    ulimits:
      nofile:
        soft: 8192
        hard: 8192

  redis:
    ulimits:
      nofile:
        soft: 65536
        hard: 65536

  gateway:
    ulimits:
      nofile:
        soft: 65535
        hard: 65535

Verification:

# Check container limits
docker exec <container> cat /proc/1/limits | grep "open files"

# Count current open FDs
docker exec <container> ls /proc/1/fd | wc -l

Common error: accept() failed: No file descriptors available - Increase ulimits.nofile.

8.4 Host System Tuning (sysctl)¶

The Docker host kernel settings affect all containers. These must be set on the host, not in containers.

Apply temporarily:

sudo sysctl -w \
  net.core.somaxconn=65535 \
  net.core.netdev_max_backlog=65535 \
  net.ipv4.tcp_max_syn_backlog=65535 \
  net.ipv4.tcp_fin_timeout=15 \
  net.ipv4.tcp_tw_reuse=1 \
  net.ipv4.ip_local_port_range="1024 65535"

Apply permanently (/etc/sysctl.d/99-mcp-loadtest.conf):

# High-concurrency TCP tuning for MCP Gateway load testing
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65535
fs.file-max = 2097152

Then apply: sudo sysctl -p /etc/sysctl.d/99-mcp-loadtest.conf

8.5 HTTPX Client Pool (Outbound HTTP)¶

The gateway uses a shared HTTPX client pool for all outbound requests (federation, health checks, A2A, MCP tool calls).

Example:

HTTPX_MAX_CONNECTIONS=200
HTTPX_MAX_KEEPALIVE_CONNECTIONS=100
HTTPX_KEEPALIVE_EXPIRY=30.0
HTTPX_READ_TIMEOUT=120.0
HTTPX_POOL_TIMEOUT=10.0

8.6 Complete High-Concurrency Configuration¶

Here's a complete configuration for 3000-6500 concurrent users:

# docker-compose.yml gateway environment
environment:
  # Database pool (via PgBouncer)
  - DATABASE_URL=postgresql+psycopg://postgres:password@pgbouncer:6432/mcp
  - DB_POOL_CLASS=queue
  - DB_POOL_PRE_PING=true
  - DB_POOL_SIZE=20
  - DB_MAX_OVERFLOW=10
  - DB_POOL_TIMEOUT=60
  - DB_POOL_RECYCLE=60

  # MCP Session Pool
  - MCP_SESSION_POOL_ENABLED=true
  - MCP_SESSION_POOL_MAX_PER_KEY=200
  - MCP_SESSION_POOL_ACQUIRE_TIMEOUT=60

  # HTTPX Client Pool
  - HTTPX_MAX_CONNECTIONS=200
  - HTTPX_MAX_KEEPALIVE_CONNECTIONS=100
  - HTTPX_READ_TIMEOUT=120.0

  # Redis
  - REDIS_MAX_CONNECTIONS=150

  # Performance
  - LOG_LEVEL=ERROR
  - DISABLE_ACCESS_LOG=true
  - AUDIT_TRAIL_ENABLED=false

9 - Security tips while tuning¶

• Never commit real JWT_SECRET_KEY, DB passwords, or tokens-use .env.example as a template.
• Prefer platform secrets (K8s Secrets, Code Engine secrets) over baking creds into the image.
• If you enable gevent/eventlet, pin their versions and run bandit or trivy scans.