Session-Affinity: Candidate Solutions for #4557¶

Issue #4557 reports a severe multi-worker session-affinity regression: 180 RPS → 9 RPS on the 3 × 24 reference stack, with tools/call p99 pinned at the 30 s forward timeout. The #4674 reproducer revealed ~24× amplification: one request per second drove the per-user rate-limit counter up at 24×, suggesting every request was being processed by all 24 workers.

This doc lays out the candidate solutions, walks through where each one shines and breaks down, and recommends a starting point with a path for further improvement.

The Core Problem¶

A stateful MCP request carrying Mcp-Session-Id can land on any of N workers, but only one worker holds the live upstream session. The UpstreamSessionRegistry entry contains a live ClientSession with an open connection, which is not serializable and not movable. When the request lands on the wrong worker, the architecture has exactly three structural choices:

1. Route correctly upstream, so the request always lands on the right worker.
2. Externalize session ownership, so any worker can serve any session.
3. Forward across workers, accepting that the wrong worker may receive the request and pass it along.

The three approaches below walk through each option in turn.

What any candidate solution must preserve¶

• The #4205 upstream-session isolation invariant: one upstream session per downstream session, no cross-session state leakage.
• Multi-worker / multi-container deployment shape (SO_REUSEPORT, gunicorn --preload, nginx fronting multiple replicas).
• All authentication shapes that streamable_http_auth() validates: ContextForge JWT, virtual-server OAuth verifier (RFC 9728), and MCP_REQUIRE_AUTH=false public-only mode.
• Existing observability: structured logs, OTEL spans, the mcpgw:* Redis state surface that operators read.
• Graceful behaviour on worker failure (no cluster-wide outage when one worker dies).

Approach 1 — Sticky Load Balancing¶

Status: validated in experimental conditions. hash $http_authorization config-only variant measured at 352 RPS, 0% failures, p99 530 ms on a 3 single-worker-pod prototype with per-user JWTs. ~21× per-worker efficiency vs the current affinity-layer baseline. See empirical summary and caveats below.

Stop forwarding at the application layer. Route correctly at the LB layer.

Client → nginx hashes on $http_authorization → pod → /rpc executes locally. No forwarding.

   Client      Mcp-Session-Id: sess-abc
     │
     ▼
   nginx     hash $http_mcp_session_id consistent;   ◄── Layer-1: pin to container
     │
     ▼
   Container N  (deterministic from session id)
     │
     ▼
   Worker M     ◄── Layer-2: need intra-container stickiness too
     │             (disable SO_REUSEPORT, OR run 1 worker per container)
     ▼
   /rpc executes — session is always here, no forward needed, no Redis pub/sub

Bootstrap routing. The initialize request doesn't yet carry Mcp-Session-Id, so hashing on that header can't pin the bootstrap itself. Solved empirically by hashing on Authorization instead (see empirical summary).

Empirical summary

Net: for deployments that meet the scope above, Approach 1 ships as a config-only change when the LB hash key is Authorization rather than Mcp-Session-Id.

Caveats (apply to the auth-hash variant):

• Public-only mode (MCP_REQUIRE_AUTH=false) is not covered. Unauthenticated requests have no Authorization header, so all anonymous traffic hashes to the empty string and concentrates on a single backend. Public-only deployments need a separate stickiness key (a deliberate cookie, a request id, or another stable header). The auth-hash variant on its own is not sufficient for them.
• Token rotation strands sessions. A refreshed JWT is a different string and may hash to a different pod. This happens during normal token refresh, not just on failover, so clients must be able to detect a session error and re-initialize routinely.
• One worker per pod is structural, not optional. nginx can only pin to the pod; if the pod has multiple gunicorn workers sharing a socket via SO_REUSEPORT, the request still scatters inside the pod and the conclusion above doesn't hold.

When to pick: moving to one-worker-per-pod (or already there). Auth-hash variant is the recommended config.

Approach 2 — Coordinator-Worker Model¶

Status: paper design only. Significant architectural change (new process type, IPC layer, ~22h prototype estimated). Not implemented. See paper design below.

Move session ownership out of workers. A single coordinator process per replica owns all upstream MCP sessions; workers become stateless and proxy through the coordinator via cheap local IPC.

nginx → any worker → coordinator (per replica, owns sessions) → upstream MCP server.

                       ┌─────────────────────────────────┐
                       │   Coordinator (1 per replica)   │
                       │   owns UpstreamSessionRegistry  │
                       └──┬──────────┬──────────┬────────┘
                          ▲          ▲          ▲
              UDS / shared-mem / localhost gRPC
                          │          │          │
                       ┌──┴──┐    ┌──┴──┐    ┌──┴──┐
                       │ W1  │    │ W2  │    │ W24 │     ◄── workers stateless
                       └─────┘    └─────┘    └─────┘
                          ▲
   nginx ──► gunicorn socket (any worker takes the request)

Paper design. Full design (IPC framing, per-session locking, request-flow walk-through, failure-mode comparison, SSE / ADR-052 open question, env-gated coexistence, ~22h prototype estimate) in the coordinator-worker design doc.

When to pick: when cluster-wide session migration becomes a hard requirement (blue/green deploys, auto-scale without session loss, multi-region failover). Not justified today.

Approach 3 — Redis-Based Cross-Worker Forwarding¶

Status: in-flight hardening. Three PRs (#4981, #4987, #4997) implement the four invariants below. Validated end-to-end on integration branch fix/session-affinity-multiworker-forwarding (~390 RPS, 0% failures on the 3 × 24 reference stack). Production-ready when the PRs land.

Redis stores sid → owner_worker_id; the receiving worker forwards the payload to the owner over an IPC transport, and the response comes back the same way. The architecture has no delta from the gateway's current design: the Redis directory, per-worker channels, and dead-worker reclaim are all already in place. The #4557 regression came from invariants not being honoured, not from the architecture being wrong. The four invariants below are what the in-flight PRs are fixing.

Invariants any Approach-3 implementation must satisfy¶

These are non-negotiable properties of the design. The existing code violated several of them, which is what produced the regression in #4557.

• Unique per-worker WORKER_ID after fork. --preload captures the master's id at import; workers must recompute in post_fork. A shared WORKER_ID collapses every worker onto one Redis channel, the source of the 24× amplification in #4557.
• Exactly one subscriber per per-worker channel. Follows from invariant 1, but worth stating independently because operators can verify it directly: PUBSUB NUMSUB mcpgw:pool_http:{worker_id} and PUBSUB NUMSUB mcpgw:pool_rpc:{worker_id} must each return 1, not N. Anything > 1 means a WORKER_ID collision is amplifying forwards on that transport.
• Forwarded requests execute in the owner process. Network loopback to 127.0.0.1 hits the shared gunicorn socket, where SO_REUSEPORT scatters the call to a random worker that doesn't hold the bound upstream session. In-process dispatch (httpx.ASGITransport(app=app)) keeps execution on the correct worker.
• Forwarded requests preserve the original streamable_http_auth() context AND the internal endpoint accepts the trusted-internal forwarding contract. The originating worker already validated the inbound credentials (ContextForge JWT, virtual-server OAuth, or public-only mode). The owner must trust that decision rather than re-authenticate: OAuth bearers fail internal JWT verification; public-only requests have no token to verify at all. For public-only mode specifically, the inner endpoint must also accept the trusted-internal HMAC marker without depending on the bearer short-circuit for CSRF; otherwise public-only forwards fail 403 on CSRF rather than 401 on auth. Either failure manifests on the inner dispatch even though the edge auth was correct.

Transport choice is independent of these invariants; none of the sub-options below compensate for an invariant being violated.

Worker X reads mcpgw:pool_owner:{sid} from Redis to find the owner, then forwards the payload to that worker over the configured transport.

   Worker X
     │ Redis GET mcpgw:pool_owner:{sid} → "worker-7"
     ▼
   <transport>  ──── forward payload ────►  Worker 7  (owns the session)
                ◄─── response ────────────
     │
     ▼
   Worker X returns response to client

The transport options below all share this ownership lookup. They differ only in how the request/response payload travels between workers.

Sub-option 3a — Redis pub/sub over TCP (the baseline)¶

Baseline transport. Operationally simplest; ~1–2 ms per round-trip. Being hardened by #4981 / #4987 / #4997.

   Worker X  ──PUBLISH──►  Redis (TCP)  ──fanout──►  Worker 7 (SUBSCRIBE)
   Worker X  ◄────── response via another pub/sub channel ──────────

Sub-option 3d — Direct TCP per worker¶

Each worker binds an internal TCP port; forward directly to it. Works cross-container. 2–10× faster than 3a.

   Redis (directory):
     mcpgw:worker_addr:worker-7 → 10.0.0.5:5007

   Worker X ────── direct TCP ──────► Worker 7  (10.0.0.5:5007)

Sub-option 3e — ZeroMQ point-to-point messaging¶

ZMQ REQ/REP over ipc:// (UDS) or tcp://. Purpose-built point-to-point messaging. New dependency; bypasses ASGI.

   Worker X                                 Worker 7
     ┌──────────┐                          ┌──────────┐
     │ ZMQ REQ  │ ──── tcp://10.0.0.5:5007 ──► REP    │
     └──────────┘                          └──────────┘
              ◄── reply ────────────────────────

Approach 4 — Redis-Resident Sessions¶

Status: ruled out for ContextForge. Requires every upstream MCP server to support cross-connection session resumption, which most don't (rmcp / Python SDK tie state to the TCP connection). Not viable for a federating gateway over arbitrary third-party upstreams.

Externalise session state to Redis so any worker can serve any session. Workers re-open upstream connections on each request and resume via the stored session id.

                       ┌───────────────────────────────────────┐
                       │  Redis (data path, not just directory)│
                       │                                       │
                       │  mcpgw:session:{sid} →                │
                       │    { upstream_url, upstream_sid,      │
                       │      last_seq, capabilities, ... }    │
                       └───────────────────────────────────────┘
                                ▲          ▲          ▲
                                │          │          │
                            (read/write per request)
                                │          │          │
                       ┌────────┴───┐ ┌────┴───┐ ┌────┴───┐
                       │  Worker 1  │ │  W 2   │ │  W N   │
                       │ stateless  │ │        │ │        │
                       └─────┬──────┘ └────┬───┘ └────┬───┘
                             │              │           │
                       (each opens its own upstream connection
                        on each request and resumes via upstream_sid)
                             │              │           │
                             ▼              ▼           ▼
                       ┌─────────────────────────────────────┐
                       │   Upstream MCP server (rmcp etc.)   │
                       │   must support: resume by sid       │
                       └─────────────────────────────────────┘

Comparison Matrix¶

Side-by-side comparison of all 6 variants: latency, cross-container support, operational delta, code-change size, and pub/sub dependency.

Recommendation¶

Priority order (try in this sequence; fall through if the constraints don't fit):

1. Approach 3: Redis pub/sub forwarding (current architecture). Smallest architectural delta. In-flight PRs (#4981, #4987, #4997) already address the four invariants. The Redis directory, per-worker channels, and dead-worker reclaim are all in place. Try this first because no deployment-shape change is required and the work is already underway.

2. Approach 1: Sticky LB on Authorization. Empirically validated at 117 RPS/worker (~21× the current per-worker efficiency). Try this if you can move to one-worker-per-pod and accept the user-pinning trade-offs (heavy-user concentration, session loss on token refresh). Smallest code change of any non-trivial option.

3. Approach 2: Coordinator-Worker model. Paper design ready (~22h prototype estimated). Try this only if Approaches 1 and 3 both prove unworkable: for instance if cluster-wide session migration becomes a hard requirement, or if you're willing to invest in a separate Rust/PyO3 coordinator process.

4. Approach 4: Redis-resident sessions. Last resort. Only viable if every upstream MCP server supports cross-connection session resumption AND the per-call reconnect cost (50–500 ms) is acceptable. Most rmcp / Python SDK upstreams don't support this today.

Within Approach 3, the transport sub-options (3d / 3e) can be adopted incrementally as performance demands grow. See Approach 3 above. They don't change the priority order.

Action Plan¶

Based on the recommendation above, the execution plan is:

Immediate¶

Take Approach 3 — Redis-based cross-worker forwarding forward now. This means landing the hardening work for the current architecture and treating the four Approach 3 invariants as the implementation contract: unique per-worker WORKER_ID, exactly one subscriber per worker channel, in-process owner execution, and preserved streamable_http_auth() context through trusted-internal forwarding.

Try Next¶

Try Approach 1 — sticky load balancing on Authorization after Approach 3, if Redis forwarding still leaves a performance or operational gap. Before recommending it broadly, validate the public-only traffic case, token-rotation behaviour, and the one-worker-per-pod deployment trade-off.

Hold Off For Now¶

Hold off Approach 2 — coordinator-worker model unless cluster-wide session migration, blue/green deploys without session loss, or multi-region failover becomes a hard requirement.

Hold off Approach 4 — Redis-resident sessions because it depends on cross-connection session resumption support that arbitrary upstream MCP servers generally do not provide.