OAuth Troubleshooting Guide¶

This guide helps troubleshoot OAuth 2.0 authentication problems when registering MCP servers, particularly the "Invalid or expired state parameter" error.

Understanding OAuth State Parameter¶

The OAuth state parameter serves two critical purposes:

• CSRF Protection: Prevents cross-site request forgery attacks via HMAC signature
• Session Binding: Links the authorization request to the callback with embedded gateway ID and user context

How State Works in ContextForge¶

┌─────────┐     1. GET /oauth/authorize/{gateway_id}     ┌─────────────┐
│  User   │ ────────────────────────────────────────────>│ ContextForge │
└─────────┘                                              └──────┬──────┘
                                                                │
                                                                │ 2. Generate:
                                                                │    - PKCE code_verifier/challenge
                                                                │    - State (JSON + HMAC signature)
                                                                │    - Store in Redis/DB/Memory
                                                                v
┌─────────┐     3. Redirect to provider with             ┌─────────────┐
│  User   │ <────────────────────────────────────────────│ ContextForge │
└────┬────┘        state + code_challenge                └─────────────┘
     │
     │ 4. User authenticates at OAuth provider
     v
┌─────────────┐
│   OAuth     │
│  Provider   │
└──────┬──────┘
       │
       │ 5. Redirect to /oauth/callback?code=xxx&state=xxx
       v
┌─────────────┐     6. Validate state:                   ┌─────────────┐
│ ContextForge │ ────────────────────────────────────────>│ State Store │
│             │     - Check signature (HMAC-SHA256)      │             │
│             │     - Check expiration (5 min TTL)       │             │
│             │     - Check not already used             │             │
│             │     - Retrieve code_verifier             │             │
└──────┬──────┘                                          └─────────────┘
       │
       │ 7. Exchange code + code_verifier for tokens
       v
┌─────────────┐
│   OAuth     │
│  Provider   │
└─────────────┘

State Structure¶

The state parameter is a base64-encoded JSON payload with HMAC signature:

# From mcpgateway/services/oauth_manager.py:627
state_data = {
    "gateway_id": gateway_id,
    "app_user_email": app_user_email,
    "nonce": secrets.token_urlsafe(16),
    "timestamp": datetime.now(timezone.utc).isoformat()
}
# Signed with HMAC-SHA256 using AUTH_ENCRYPTION_SECRET

OAuth Endpoints¶

Note: The three /oauth/registered-clients* endpoints manage DCR client records that are stored globally, with no team association. Because there is no team scope to narrow into, these endpoints require an un-narrowed admin token — an admin API/legacy token carrying a token_teams narrowing claim (or a public-only admin token) receives 403 Forbidden with "OAuth client management requires un-narrowed admin access". Admin session tokens (e.g. the interactive Admin UI) resolve their teams from the database and cannot be narrowed, so this does not change UI behavior.

Error Messages and Their Causes¶

"Invalid or expired state parameter - possible replay attack"¶

Location: mcpgateway/services/oauth_manager.py:544

Causes:

• State not found in storage (Redis/DB/Memory)
• State already consumed (single-use protection)
• State expired (TTL exceeded)

"Invalid state signature - possible CSRF attack"¶

Location: mcpgateway/services/oauth_manager.py:562

Causes:

• State was tampered with
• AUTH_ENCRYPTION_SECRET changed between authorization and callback
• Corrupted state parameter in URL

"State parameter gateway mismatch"¶

Location: mcpgateway/services/oauth_manager.py:572

Causes:

• Callback received for different gateway than initiated
• Gateway ID changed in state parameter
• Misconfigured redirect URI

"State has expired for gateway {gateway_id}"¶

Location: mcpgateway/services/oauth_manager.py:746, 782, 818

Cause: OAuth flow took longer than 5 minutes (STATE_TTL_SECONDS = 300)

"State was already used for gateway {gateway_id} - possible replay attack"¶

Location: mcpgateway/services/oauth_manager.py:751, 789, 824

Cause: User refreshed callback page or callback was triggered twice

"State not found in [Redis|database|memory] for gateway {gateway_id}"¶

Location: mcpgateway/services/oauth_manager.py:727, 772, 809

Causes:

• Server restart (in-memory storage lost)
• Different worker handled callback (multi-worker with in-memory)
• State never stored (storage failure during authorization)

Common Causes and Fixes¶

1. Multi-Worker Deployment with In-Memory Storage¶

Symptom: Intermittent failures - sometimes works, sometimes doesn't.

Cause: In-memory storage doesn't share state across workers.

# From mcpgateway/services/oauth_manager.py:35-37
_oauth_states: Dict[str, Dict[str, Any]] = {}  # Per-process only when CACHE_TYPE=memory!

Diagnosis:

# Check your cache type configuration
grep CACHE_TYPE .env
# Default is "database" - which is safe for multi-worker

# If CACHE_TYPE=memory, check worker count
ps aux | grep -E "(gunicorn|uvicorn)" | grep -v grep
# Multiple workers + CACHE_TYPE=memory = problem

Fix:

# Option A: Use default database storage (recommended - already the default)
CACHE_TYPE=database

# Option B: Use Redis for better performance
CACHE_TYPE=redis
REDIS_URL=redis://localhost:6379

# Option C: Single worker (development only, if using memory)
# gunicorn --workers 1 ...

2. State TTL Expired (5 Minutes)¶

Symptom: Always fails if user takes too long to authenticate.

Cause: State expires after 300 seconds.

# From mcpgateway/services/oauth_manager.py:42
STATE_TTL_SECONDS = 300  # 5 minutes

Diagnosis:

• Check timestamps in debug logs
• Note time between "Stored OAuth state" and callback

Fix:

• Complete OAuth flow within 5 minutes
• For slow identity providers, consider increasing TTL (requires code change)

3. State Already Consumed (Replay Protection)¶

Symptom: First attempt works, refresh/retry fails.

Cause: State is single-use and deleted after validation.

# From mcpgateway/services/oauth_manager.py:724-725
# Get and delete state atomically (single-use)
state_json = await redis.getdel(state_key)

Diagnosis:

# Check logs for duplicate callbacks
grep "State not found" logs/mcpgateway.log
grep "already been used" logs/mcpgateway.log

Fix:

• Don't refresh the callback page
• Start a new OAuth flow if needed (click "Authorize" again in Admin UI)

4. Server Restart During OAuth Flow¶

Symptom: Always fails after server restart.

Cause: In-memory states are lost on restart.

Fix:

• Use Redis or database storage for persistence
• Restart servers during low-traffic periods

5. Gateway ID Mismatch¶

Symptom: State validation fails with "gateway mismatch" in logs.

Cause: The callback returned to a different gateway than expected.

# From mcpgateway/services/oauth_manager.py:572
raise OAuthError("State parameter gateway mismatch")

Diagnosis:

# Check for gateway mismatch errors
grep "gateway mismatch" logs/mcpgateway.log

Fix:

• Verify the OAuth redirect URI matches the gateway configuration
• Check that gateway IDs are consistent
• Ensure redirect_uri in OAuth provider matches {BASE_URL}/oauth/callback

6. Load Balancer Without Sticky Sessions¶

Symptom: Fails randomly in load-balanced environments.

Cause: Different backend handles callback than authorization.

Fix:

# Use distributed state storage
CACHE_TYPE=redis
REDIS_URL=redis://localhost:6379

Or configure sticky sessions in your load balancer (not recommended).

7. AUTH_ENCRYPTION_SECRET Changed¶

Symptom: "Invalid state signature" error after config change.

Cause: HMAC signature is validated using AUTH_ENCRYPTION_SECRET.

Fix:

• Don't change AUTH_ENCRYPTION_SECRET during active OAuth flows
• Clear all pending states after changing the secret
• Ensure consistent secret across all workers/instances

8. PKCE Code Verifier Lost¶

Symptom: Token exchange fails after successful callback.

Cause: PKCE code_verifier not found during token exchange (stored with state).

Fix: Same as state storage issues - ensure Redis/database storage is working.

Admin UI OAuth Flow¶

Starting OAuth Authorization¶

1. Navigate to Admin Panel > Gateways
2. Find gateway with auth_type = oauth
3. Click Authorize button (only visible for OAuth gateways)
4. Browser redirects to /oauth/authorize/{gateway_id}
5. User authenticates at OAuth provider
6. Callback redirects to success page
7. Click Fetch Tools to import MCP tools

Debugging Steps¶

Step 1: Enable Debug Logging¶

# .env
LOG_LEVEL=DEBUG

Step 2: Check State Storage Type¶

# Verify current configuration
grep -E "^CACHE_TYPE" .env

# Valid values:
# CACHE_TYPE=database  (default - recommended, supports multi-worker)
# CACHE_TYPE=redis     (best for high-traffic production)
# CACHE_TYPE=memory    (single-worker only - NOT recommended)
# CACHE_TYPE=none      (disabled - OAuth will not work)

Step 3: Monitor State Operations¶

Watch for these log messages:

# Successful state storage
grep "Stored OAuth state" logs/mcpgateway.log

# State validation
grep -E "(validated|not found|expired|already been used)" logs/mcpgateway.log

# Signature validation
grep -E "(Invalid state signature|CSRF)" logs/mcpgateway.log

Healthy flow:

DEBUG - Stored OAuth state in Redis for gateway abc123
DEBUG - Successfully validated OAuth state from Redis for gateway abc123
INFO - Completed OAuth flow for gateway abc123, user user@example.com

Problematic flow (example: CACHE_TYPE=memory with multiple workers):

DEBUG - Stored OAuth state in memory for gateway abc123
WARNING - State not found in memory for gateway abc123
ERROR - OAuth callback failed: Invalid or expired state parameter - possible replay attack

Problematic flow (example: state expired):

DEBUG - Stored OAuth state in database for gateway abc123
WARNING - State has expired for gateway abc123
ERROR - OAuth callback failed: Invalid or expired state parameter - possible replay attack

Step 4: Verify Redis Connectivity (if using Redis)¶

# Test Redis connection
redis-cli ping
# Should return: PONG

# Check OAuth states in Redis
redis-cli keys "oauth:state:*"

# Inspect a specific state (TTL and content)
redis-cli ttl "oauth:state:<gateway_id>:<state>"
redis-cli get "oauth:state:<gateway_id>:<state>"

Step 5: Check Database States (if using database)¶

-- SQLite: Check for stored states (oauth_states table)
SELECT id, gateway_id, substr(state, 1, 50) as state_preview,
       code_verifier IS NOT NULL as has_pkce,
       expires_at, used, created_at
FROM oauth_states
ORDER BY created_at DESC
LIMIT 10;

-- SQLite: Check for expired states (should be cleaned up automatically)
SELECT COUNT(*) as expired_count
FROM oauth_states
WHERE expires_at < datetime('now');

-- Check for used states (should be deleted after use)
SELECT COUNT(*) as used_count
FROM oauth_states
WHERE used = 1;

-- View stored tokens after successful OAuth
SELECT id, gateway_id, user_id, app_user_email,
       expires_at, created_at
FROM oauth_tokens
ORDER BY created_at DESC
LIMIT 10;

-- PostgreSQL alternatives:
-- Use NOW() instead of datetime('now')
-- Use true instead of 1 for boolean

Step 6: Trace a Single OAuth Flow¶

# Enable debug logging first
export LOG_LEVEL=DEBUG

# Start the gateway
make dev

# In another terminal, tail logs
tail -f logs/mcpgateway.log | grep -E "(OAuth|state|gateway)"

# Start OAuth flow in browser and watch the logs

Configuration Reference¶

Recommended Production Configuration¶

# .env

# Use Redis for distributed state storage (recommended)
CACHE_TYPE=redis
REDIS_URL=redis://localhost:6379

# Redis connection settings
REDIS_MAX_RETRIES=3
REDIS_RETRY_INTERVAL_MS=2000

# OAuth settings
OAUTH_REQUEST_TIMEOUT=30
OAUTH_MAX_RETRIES=3

# PKCE settings
OAUTH_DISCOVERY_ENABLED=true
OAUTH_PREFERRED_CODE_CHALLENGE_METHOD=S256

# Secret for signing state (CRITICAL - must be consistent across instances)
AUTH_ENCRYPTION_SECRET=your-secret-key-here

# Enable debug logging temporarily for troubleshooting
LOG_LEVEL=DEBUG

Dynamic Client Registration (DCR)¶

If using DCR (RFC 7591) for automatic client registration:

# Enable DCR
DCR_ENABLED=true
DCR_AUTO_REGISTER_ON_MISSING_CREDENTIALS=true

# Default scopes for registered clients
DCR_DEFAULT_SCOPES=["mcp:read"]

# Token endpoint auth method
DCR_TOKEN_ENDPOINT_AUTH_METHOD=client_secret_basic

# Metadata cache TTL
DCR_METADATA_CACHE_TTL=3600

State Storage Comparison¶

Environment Variables¶

Quick Fixes¶

Fix 1: Verify Database Storage is Active (Default)¶

# Database storage is the default - verify it's set correctly
grep CACHE_TYPE .env

# If not set or set incorrectly, ensure it's using database:
# CACHE_TYPE=database

# Run migrations to ensure oauth_states table exists
alembic upgrade head

# Restart ContextForge
make dev

Fix 2: Switch to Redis Storage (For High-Traffic Production)¶

# Install Redis if needed
# Ubuntu/Debian
sudo apt install redis-server
sudo systemctl start redis

# macOS
brew install redis
brew services start redis

# Update .env
sed -i 's/CACHE_TYPE=.*/CACHE_TYPE=redis/' .env
# Or add if not present:
echo "CACHE_TYPE=redis" >> .env
echo "REDIS_URL=redis://localhost:6379" >> .env

# Restart ContextForge
make dev

Fix 3: Single Worker Mode (Only if using CACHE_TYPE=memory)¶

# Only needed if you're using CACHE_TYPE=memory
# (Not recommended - switch to database or redis instead)
gunicorn mcpgateway.main:app --workers 1 --bind 0.0.0.0:4444

# Or using make
WORKERS=1 make serve

Fix 4: Clear Stale States¶

# Redis - clear all OAuth states
redis-cli keys "oauth:state:*" | xargs -r redis-cli del

# Database - clear expired and stale states
sqlite3 mcp.db "DELETE FROM oauth_states WHERE expires_at < datetime('now') OR used = 1;"

# PostgreSQL
psql -c "DELETE FROM oauth_states WHERE expires_at < NOW() OR used = true;"

Verifying the Fix¶

After applying fixes, verify OAuth works:

1. Clear any stale states (see Fix 4 above)

2. Restart the gateway:

   make dev
   

3. Test OAuth flow:

   • Go to Admin Panel > Gateways
   • Click "Authorize" on an OAuth-configured gateway
   • Complete authentication within 5 minutes
   • Verify success page shows "OAuth Authorization Successful"
   • Click "Fetch Tools" to verify token works

4. Check logs for success:

   grep "Successfully validated OAuth state" logs/mcpgateway.log
   grep "Completed OAuth flow" logs/mcpgateway.log
   

5. Verify tokens stored:

   SELECT * FROM oauth_tokens ORDER BY created_at DESC LIMIT 5;
   

Troubleshooting DCR Issues¶

If using Dynamic Client Registration:

# Check DCR is enabled
grep DCR .env

# Look for DCR-related logs
grep -E "(DCR|Dynamic Client)" logs/mcpgateway.log

# Check registered clients (requires an un-narrowed admin bearer token)
curl -s -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" http://localhost:4444/oauth/registered-clients | jq

# Common DCR errors:
# - "DCR failed" - AS doesn't support RFC 7591
# - "No issuer configured" - Gateway missing issuer URL
# - "Metadata discovery failed" - AS metadata endpoint unreachable

Token audience mismatch¶

Symptom¶

The gateway logs or returns an error message:

Refusing to forward OAuth token for gateway 'X': Token audience mismatch: token aud does not match expected resource or gateway URL. Fix oauth_config (resource/scopes/issuer) or the IdP token request.

Diagnosis steps¶

1. Check the affected user's learned audience: Query the oauth_tokens table for (gateway_id, app_user_email) and inspect learned_aud / learned_iss. This is the value the validator uses authoritatively for that user's tokens (precedence 2, below the admin-configured oauth_config.resource).
2. Check gateway configuration: Verify the oauth_config.resource field via GET /admin/gateways/{id} or the Admin UI. If it is set, it takes precedence over per-user learned values.
3. Inspect the token: Decode the JWT access token (e.g. via jwt.io) and check the aud (audience) claim. Compare against learned_aud or oauth_config.resource above.
4. Verify IdP RFC 8707 support: Providers like Authentik and ServiceNow do not honor the resource parameter — they typically return a default audience such as the client_id. Per-user auto-learning handles this automatically.

Common causes and fixes¶

• IdP returns aud=client_id (Authentik, ServiceNow) — expected behavior for non-RFC-8707 providers. Per-user auto-learning captures the actual aud on the user's OAuth callback and stores it as OAuthToken.learned_aud. If a user is still failing validation, verify they have completed at least one successful OAuth flow (the row must exist and learned_aud must be non-null).
• Multi-tenant Entra ID, wrong tenant — if the token is being issued for the wrong audience in a multi-tenant setup, set oauth_config.resource explicitly to your Application ID URI (e.g., api://{your-app-id}). See OAuth Resource Configuration.
• Stale learned value for one user after IdP migration — a user's learned_aud becomes stale if the IdP configuration changed since their last authentication. Fix: the user re-authenticates. The next OAuth callback overwrites their OAuthToken.learned_aud with the current audience — no admin action needed.
• Stale admin-configured oauth_config.resource — if the admin explicitly set the resource and now needs to clear it, edit the gateway in the Admin UI and blank the Resource field (or PUT {"oauth_config": {"resource": null}} via API).
• Salesforce with full gateway URL — Salesforce tokens use origin-level audiences. The gateway's origin-derivation fallback (_derive_resource_origin) handles this automatically for the outbound resource parameter; inbound validation uses the per-user learned value.

Advisory vs. authoritative behavior¶

Precedence for the expected audience (first match wins):

1. oauth_config.resource — admin-configured. Authoritative.
2. OAuthToken.learned_aud for THIS USER — per-user. Authoritative for this user.
3. Gateway URL origin — auto-derived fallback. Advisory by default.

The advisory fallback (#3) only applies to users with no learned value (first authentication) and no admin-configured resource. To make the auto-derived case blocking (strict mode), set:

OAUTH_REQUIRE_CONFIGURED_RESOURCE=true

Useful debug logs¶

Set LOG_LEVEL=DEBUG and grep for:

• mcpgateway.services.oauth_manager — Unverified JWT decode failed (indicates the token is opaque or malformed and no learned_aud can be extracted).
• mcpgateway.services.gateway_service — Refusing to forward OAuth token (the blocking-error path).
• mcpgateway.services.token_validation_service — the per-warning lines emitted before the blocking check.
• mcpgateway.services.token_storage_service — token storage and retrieval events.

Gateway vs. upstream validation responsibility¶

ContextForge's local audience check is either advisory (auto-derived fallback) or authoritative (explicit / learned / OAUTH_REQUIRE_CONFIGURED_RESOURCE=true). If ContextForge forwards the token but the upstream MCP server still returns 401 Unauthorized, the MCP server's own audience configuration doesn't match what the IdP provided. Two options:

1. Configure the MCP server to accept the audience provided by the IdP.
2. Configure the IdP (or set oauth_config.resource explicitly) so the minted token's aud matches what the MCP server expects.

External IdP Bearer Token Rejected on API/MCP¶

This section covers requests that present an access token issued by an external SSO provider (e.g. Keycloak, Entra ID) directly as a Bearer credential to API/MCP endpoints, per SSO: Machine-to-machine API auth with external IdP tokens. Match the symptom to the exact log line emitted by mcpgateway/utils/verify_credentials.py.

Issuer trailing-slash mismatches¶

issuer not trusted is also the symptom for an issuer string that differs only by a trailing slash (e.g. https://keycloak.example.com/realms/master vs https://keycloak.example.com/realms/master/). ContextForge normalizes issuers for comparison, but if you configured the provider's issuer manually, ensure it matches the iss claim in tokens issued by that realm/tenant exactly (modulo trailing slash).

Popup OAuth Flow Issues (React UI)¶

Popup Blocked by Browser¶

Symptom: OAuth popup doesn't open, or opens then immediately closes.

Cause: Browser popup blocker or user gesture requirement not met.

Fix: - Ensure OAuth is triggered by direct user action (click handler) - Check browser console for popup blocker warnings - Add site to browser's popup allowlist - Use window.open() synchronously in click handler (not in async callback)

postMessage Not Received¶

Symptom: Popup completes OAuth but parent window doesn't receive result.

Causes: 1. Parent window closed before callback 2. Event listener not registered before popup opens 3. event.source validation rejecting message 4. Cross-origin policy blocking message

Diagnosis:

// Add debug logging to message handler
window.addEventListener('message', (event) => {
  console.log('Received message:', event.data);
  console.log('Event source matches popup:', event.source === authWindow);
  console.log('Event origin:', event.origin);
});

Fix: - Register message listener BEFORE opening popup - Verify event.source === authWindow check is correct - Check browser console for CORS/CSP errors - Ensure parent window stays open during OAuth flow

Popup State Mismatch¶

Symptom: Callback fails with "Invalid state" even though popup flow worked.

Cause: State token doesn't have popup. prefix.

Diagnosis:

# Check state generation logs
grep "popup\." logs/mcpgateway.log
grep "Stored OAuth state" logs/mcpgateway.log

Fix: - Verify popup=true query parameter is included in authorize URL - Check oauth_manager.py:1033 generates prefixed state - Ensure state prefix isn't stripped during storage/retrieval

CSP Blocks Inline Script¶

Symptom: Popup shows blank page or "Content Security Policy" error.

Cause: Strict CSP without nonce support for inline scripts.

Diagnosis:

# Check browser console for CSP errors
# Look for: "Refused to execute inline script because it violates CSP"

Fix: - Verify CSP middleware generates nonces (mcpgateway/middleware/csp_middleware.py) - Check get_csp_nonce_from_request() returns valid nonce - Ensure callback script includes nonce attribute: <script nonce="..."> - Review CSP header: should include script-src 'self' 'nonce-...'

Popup Closes Before User Sees Result¶

Symptom: Popup closes immediately, user doesn't see success/error message.

Cause: window.close() executes before user can read the message.

Expected Behavior: - Popup posts message to parent and closes automatically - Parent window should display success/error notification - Popup fallback HTML is only shown if window.opener is missing

Fix: Handle notifications in parent window, not popup:

window.addEventListener('message', (event) => {
  if (event.source !== authWindow) return;

  const { type, status, error, errorDescription } = event.data;

  if (type === 'oauth_callback') {
    if (status === 'success') {
      showNotification('OAuth authorization successful', 'success');
    } else {
      showNotification(`OAuth failed: ${errorDescription}`, 'error');
    }
  }
});

Multiple Popups or Duplicate Messages¶

Symptom: Multiple OAuth popups open or parent receives duplicate messages.

Cause: - User clicks "Authorize" multiple times - Event listener registered multiple times - Popup reference not tracked

Fix:

let authWindow = null;

function startOAuth(gatewayId) {
  // Close existing popup if any
  if (authWindow && !authWindow.closed) {
    authWindow.close();
  }

  authWindow = window.open(
    `/oauth/authorize/${gatewayId}?popup=true`,
    'oauth-popup',
    'width=600,height=700'
  );
}

// Use once() to prevent duplicate listeners
window.addEventListener('message', handleOAuthMessage, { once: true });

Popup Opens But Shows Error Page¶

Symptom: Popup opens but immediately shows OAuth error page.

Causes: 1. Gateway not configured for OAuth 2. Missing OAuth configuration fields 3. Invalid redirect URI 4. Provider rejected authorization request

Diagnosis:

# Check gateway OAuth config
curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:4444/oauth/status/{gateway_id}

# Check logs for authorization errors
grep "OAuth provider returned error" logs/mcpgateway.log

Fix: - Verify gateway has auth_type: oauth and complete oauth_config - Check redirect URI matches provider configuration - Review provider's authorization requirements (scopes, audience, etc.)

Related Documentation¶

• OAuth Integration - Main OAuth setup guide
• OAuth Resource Configuration - Configuring the RFC 8707 resource / audience field, provider patterns, forcing a re-learn
• Configuration Reference - All environment variables
• Scaling Guide - Multi-worker and Redis setup
• Securing ContextForge - Security best practices