Local - lite load testing and monitoring¶

Quick Start Commands¶

Step-1: (Skip if you are using linux running docker engine)¶

Colima Setup : 10 CPU / 18GB¶

# Start colima with 10 cpu and 18gb memory
colima start --cpu 10 --memory 18

# Check with colima status
colima list
# TERMINAL OUTPUT -
PROFILE    STATUS     ARCH       CPUS    MEMORY    DISK      RUNTIME    ADDRESS
default    Running    aarch64    10      18GiB     100GiB    docker

Step-2: Application setup , load testing and monitoring setup.¶

# Create production docker image
make docker-prod

# Note: Refer docker-compose.override.lite.yml for cpu and memory allocation
# Run lite replicas (uses docker compose)
make compose-lite-down && make compose-lite-up
# Access the gateway via proxy - http://localhost:8080

# Run lite monitoring (Not running pgadmin and redis commander)
make monitoring-lite-down && make monitoring-lite-up
# Access monitoring - http://localhost:3000

# Start load test UI (Runs on local host)
make load-test-ui
# Access locust - http://localhost:8089 and run the test
    # Concurreny = 800
    # Ramp up = 100
    # Run Time - 10m
    # uncheck FastTestEchoUser & FastTestTimeUser

Note: The lite commands (make compose-lite-up/down and make monitoring-lite-up/down) apply the docker-compose.override.lite.yml override file for reduced resource limits.

Gateway Lite Service¶

HTTP Server Selection¶

Both use Gunicorn (battle-tested, lower memory than Granian)

• 24 workers per replica (matches CPU cores)
• Connection pooling via PgBouncer

Database Layer (PostgreSQL + PgBouncer)¶

No changes between versions - both tuned for 3000 concurrent users.

Cache (Redis)¶

Resource Allocation: Old vs New¶

Understanding Limits vs Reservations¶

• Limits = Hard ceiling (Docker throttles) - can exceed available CPU
• Reservations = Soft guarantee - must fit in available CPU
• Actual Usage = Real consumption (typically 30-50% of limits)

Safety: Limits can exceed 10 CPU because containers don't hit them simultaneously. Docker throttles (not crashes).

Safety Verdict ✅¶

Original compose-up: 12 CPU reserved (120% of 10 CPU) ❌ Won't schedule

Resource Breakdown (Verified Against docker-compose.override.lite.yml)¶

compose-lite-up (core only):

Nginx:       0.5 CPU / 256MB reservation
Gateway(2×): 4.0 CPU / 5GB reservation
PostgreSQL:  1.0 CPU / 2GB reservation
PgBouncer:   0.5 CPU / 128MB reservation
Redis:       0.75 CPU / 0.75GB reservation
─────────────────────────────────────
Total:       6.75 CPU / 8.125GB ✅

+ monitoring-lite (10 services):

prometheus:         0.25 CPU / 256MB
grafana:            0.25 CPU / 256MB
loki:               0.25 CPU / 256MB
tempo:              0.25 CPU / 256MB
promtail:           0.25 CPU / 128MB
cadvisor:           0.25 CPU / 128MB
postgres_exporter:  0.125 CPU / 64MB
redis_exporter:     0.125 CPU / 64MB
pgbouncer_exporter: 0.125 CPU / 64MB
nginx_exporter:     0.125 CPU / 64MB
─────────────────────────────────────
Additional:  2.0 CPU / 1.5GB
Grand Total: 8.75 CPU / 9.625GB ✅ (87.5% utilization, 1.25 CPU headroom)

make compose-lite-up : Services¶

Included:

• ✅ Nginx (caching proxy)
• ✅ Gateway (2 replicas)
• ✅ PostgreSQL 18
• ✅ PgBouncer (connection pooler)
• ✅ Redis (cache)

Use Case: Fast, lightweight local load testing without observability overhead

make monitoring-lite-up: Services¶

Included:

• ✅ Prometheus (metrics collection)
• ✅ Grafana (dashboards)
• ✅ Loki (log aggregation)
• ✅ Tempo (distributed tracing)
• ✅ Promtail (log shipper)
• ✅ postgres_exporter (DB metrics)
• ✅ redis_exporter (cache metrics)
• ✅ pgbouncer_exporter (pool metrics)
• ✅ nginx_exporter (proxy metrics)
• ✅ cAdvisor (container metrics)

Excluded (vs. monitoring profile):

• ❌ pgAdmin (0.5 CPU / 256MB) - Disabled for Colima
• ❌ redis_commander (0.25 CPU / 128MB) - Disabled for Colima

Resource Savings: 0.75 CPU / 384MB by excluding admin tools

Use Case: Full observability (metrics, logs, traces) without admin UIs for local load testing.

Original Profiles (Not for 10 CPU Colima/ Docker Engine)¶

Performance Characteristics & Actual Usage¶

Throughput Capacity¶

Note: Database (PostgreSQL max_connections=800) is the bottleneck, not CPU.

Load test summary(800 users, 100 ramp rate, 10minutes)¶

Overall Metrics:

• Total Requests: 266,151
• Total Failures: 2 (0.00%)
• Requests/sec (RPS): 444.86

Response Times (ms):

• Average: 25.60
• Min: 0.92
• Max: 30011.87
• Median (p50): 8.00
• p90: 26.00
• p95: 60.00
• p99: 220.00

Lite local load test¶

Monitoring-Lite vs Full Monitoring Comparison¶

When to use each:

• make monitoring-lite-up - Local load testing on Colima/Docker Desktop (same observability stack, just excludes admin UIs)
• make monitoring-up - Full monitoring including pgAdmin and redis_commander admin UIs (requires full docker-compose.yml)