Uvicorn Scaling Analysis — titan-killer.service

OPTION A (recommended) Stay at 1 worker, harden the single process

Why: current traffic does not justify the complexity bill of any multi-worker fix. titan-killer peak CPU is <15% per systemctl status. We are nowhere near saturation.

What to do:

• Add --loop uvloop --http httptools to ExecStart for +30% single-process throughput at zero risk.
• Push the heavy DB-bound handlers (zeus search, GA4, financial_validator) to async-native DB access via asyncpg so the one event loop stops blocking.
• Move long CPU-bound jobs (RAG embeddings, PDF generation) out of request path into the existing systemd timers or /usr/local/bin/task_queue. Many already are.
• Set explicit --limit-concurrency 200 and --backlog 4096 to fail fast instead of queueing forever.

Cost: 1 engineer-hour. Risk: near zero (tunables only).

OPTION B Gunicorn + UvicornWorker, 2 workers, with websocket rewrite

Prereq change: replace the connected_boards list in websocket.py with a Redis pub/sub fan-out:

# websocket.py rewrite (sketch)
import redis.asyncio as aioredis
REDIS = aioredis.from_url("redis://localhost:6379")

@router.websocket("/ws/dispatch")
async def dispatch_websocket(ws: WebSocket):
    await ws.accept()
    pubsub = REDIS.pubsub()
    await pubsub.subscribe("dispatch:events")
    try:
        async for msg in pubsub.listen():
            if msg["type"] == "message":
                await ws.send_text(msg["data"].decode())
    except WebSocketDisconnect:
        await pubsub.unsubscribe("dispatch:events")

@router.post("/api/dispatch/broadcast")
async def broadcast(event: dict):
    await REDIS.publish("dispatch:events", json.dumps(event))

ExecStart becomes: gunicorn app:app -k uvicorn.workers.UvicornWorker -w 2 -b 0.0.0.0:8765 --timeout 120. Redis is already installed on VM (confirm with redis-cli ping).

Cost: 4-6 engineer-hours (rewrite + test dispatch board under 2 workers). Risk: medium — websocket regression would be silently partial. Requires Playwright two-client test before ship.

OPTION C nginx sticky sessions on /ws/, 2 workers, no code change

nginx sits in front of uvicorn (port 80/443 → 8765). Add ip_hash on the upstream block so each client IP pins to a worker. Broadcasts still lose 50% of events across workers, but each WebSocket client continues to see ITS worker’s state.

This only works if broadcasts originate from inside the same worker that owns the websocket. In our case, POST /api/dispatch/broadcast comes from an HTTP client (dashboard) that may land on a different worker than the websocket client → still drops 50% of events.

Verdict: sticky sessions alone do NOT fix this. Need Redis (option B) anyway.

OPTION D Split websocket/SSE out into a dedicated single-worker service

Run two systemd units:

• titan-killer.service: uvicorn app:app, 2 workers, serves everything EXCEPT /ws/* and /api/stream/*
• titan-realtime.service: uvicorn app_realtime:app, 1 worker, serves only /ws/* + /api/stream/* on port 8766
• nginx routes /ws/ and /api/stream/ to :8766, everything else to :8765

Cost: 2-3 hours (new app_realtime.py factoring, nginx config, systemd unit). Risk: low — isolates the fragile part. No Redis needed. Broadcasts to :8766 still work in a single process. HTTP routes get 2x parallelism on :8765.

Gotcha: the POST /api/dispatch/broadcast HTTP route is in websocket.py, so it must also live on :8766. nginx config handles that.