Every prompt, checked — at 100k RPS
AI Security Gateway
A production AI security gateway I own end to end — backend, frontend, DevOps, cloud, and scaling — that fronts 500+ endpoints, enforcing pre/post-LLM security and scaling with async concurrency from 100 to 100k+ RPS.
Every LLM call an enterprise makes is an attack surface — a prompt injection, a leaked secret, a jailbroken model. The gateway is the one chokepoint that makes 500+ endpoints safe by default. And because it sits in the hot path of every call, it can never be the bottleneck: it had to hold the line at 100k+ RPS.
- throughput
- 100k+ RPS
- scaled from 100 via async
- endpoints fronted
- 500+
- unsafe behavior
- −99%
- pre + post-LLM
- ownership
- sole
- backend → cloud
the bar it had to clear
- One chokepoint for 500+ endpoints — every prompt and completion passes pre- and post-LLM security, or it does not ship.
- Security cannot cost latency the product will feel — hold the budget at 100k+ RPS.
- Defense in depth: SHA-256 auth, PII redaction, and prompt-injection mitigation on both the request and the response.
- Owned end to end — backend, frontend, DevOps, cloud, and scaling, all on one person.
how it actually works
hover or tab a step to see how it works — or trace a route above.
- 01
Untrusted by default
Every request authenticates against SHA-256 API keys before it touches a model — no key, no call.
- 02
The chokepoint, on purpose100k+ RPS
One async gateway fronts 500+ endpoints; centralizing pre/post-LLM enforcement is what makes them safe by default. Scaled with async concurrency from 100 to 100k+ RPS.
- 03
Redact before inference
PII is detected and redacted before a prompt ever reaches a model — line-rate, on every request.
- 04
Routed, not hardwired
Multi-provider routing with fallback via LiteLLM, behind one secured interface.
- 05
Check the response too−99% unsafe
Completions are screened on the way out — catching unsafe output even when the prompt looked clean. Drives a 99% reduction in unsafe behavior.
- 06
Safe by construction
A response that passed pre- and post-LLM security, returned at 100k+ RPS.
the problem
A fleet of LLM-powered features across 500+ endpoints each re-implemented (or skipped) security. The task was to centralize it into one gateway that is safe by default and fast enough to sit in the hot path of every call — which a naive synchronous proxy cannot do at 100k RPS.
the approach
- Every request authenticates (SHA-256) and passes pre-LLM checks — PII redaction and prompt-injection screening — before a model ever sees it.
- Responses pass post-LLM checks on the way out, so unsafe completions are caught even when the prompt looked clean.
- Scaled from 100 to 100k+ RPS with async concurrency (ASGI/Uvicorn), non-blocking I/O, and response caching + request dedup on the hot path.
- Multi-provider routing via LiteLLM with fallback behind one secured interface; every hop audited to PostgreSQL/Redis.
the calls that mattered
Every architecture is a set of trade-offs taken on purpose. These are the load-bearing ones.
Make the gateway the single chokepoint — safe by default, not opt-in.
Security that each endpoint re-implements is security that some endpoint forgets. Centralizing pre/post-LLM enforcement protects 500+ endpoints by construction, and policy changes in one place.
the trade-offThe gateway becomes a hard dependency in the hot path of every call — which is exactly why it had to be engineered for 100k+ RPS, not for convenience.
Check the response, not just the prompt.
A clean-looking prompt can still elicit an unsafe completion — jailbreaks, leaked data. Post-LLM screening catches what pre-LLM cannot, driving a 99% reduction in unsafe behavior.
the trade-offA second inspection pass per call — paid for by keeping it async and cached so it never blocks the fast path.
Scale with async concurrency, not more machines.
The gateway is I/O-bound — network to models and to checks. Async concurrency (ASGI, non-blocking I/O) took it from 100 to 100k+ RPS on the same footprint, where a synchronous worker-per-request model would have needed orders of magnitude more boxes.
the trade-offAsync code is harder to reason about and debug — worth it for a thousand-fold throughput gain in the hot path.
what shipped
The gateway fronts 500+ endpoints at 100k+ RPS with a 99% reduction in unsafe LLM behavior — SHA-256 auth, PII redaction, and prompt-injection mitigation on every request and response — owned end to end from backend to cloud deployment and scaled with async concurrency from an initial 100 RPS.
what’s next
- Per-tenant policy and rate limits so each customer tunes its own safety/latency trade-off.
- Streaming inspection so post-LLM checks run token-by-token instead of on the full completion.
- Policy-as-code (OPA) so security rules ship without a redeploy.