Inside the gateway: why one choke point

Every model request in PrivateMind passes through one Rust service before it reaches a GPU. Browser chat, embeddings, voice, agents: same door. That is a single point of failure, on purpose, and this post is the honest version of why.

The costs, stated up front

A choke point is exactly what it sounds like. If the gateway is down, nothing on the platform can talk to a model. Every new capability pays a toll there too: when we added audio transcription, the gateway grew a dedicated sub-router just to raise the body limit, because a five-minute WAV blows past the 10 MB cap we hold everywhere else. That change has nothing to do with routing. It lives in the gateway anyway, because everything does, and the team that owns the gateway reviews it. One service, one review queue, one team in the critical path. Plus an extra network hop on every request.

Those costs are real. Here is what they buy.

One auth path, and the budget check rides along

API keys are Bearer ACCESS_KEY_ID:SECRET. The gateway looks the key up in an in-process cache (bounded LRU with a TTL, Postgres on miss) and verifies the secret against a stored SHA-256 hash with a constant-time compare. The same cached row carries everything else the request needs to be judged:

authenticate(header) -> key      // hash compare, cache over Postgres
  reject if revoked
  reject if expired
  reject if spent >= budget      // this month's spend, reset lazily in SQL
check_model_access(key, model)   // per-key allowlist, "*" means all
resolve(model) -> canonical name + upstream URL
proxy

The budget detail is worth a sentence. Spend resets monthly, but there is no cron flipping counters at midnight. The lookup query compares the row's spend period against the current month and reads stale spend as zero; the next metered write lands the counter back at the right place. Lazy reset, no scheduled job to miss. That trick only works because there is exactly one reader and one writer of that counter, and both are this service.

Aliases: callers ask for a capability, not a checkpoint

Clients can request a model by role alias instead of by name. The gateway resolves the alias and rewrites the model field in the request body to the canonical name before proxying, because the upstream inference server only knows canonical names and would 404 on the alias. Swap the model behind an alias and every caller upgrades on their next request, no client release. The indirection costs one map lookup and is the cheapest migration tool we own.

Streaming without losing the bill

The interesting engineering problem is not proxying SSE. It is metering a response you are forwarding byte-for-byte and must not buffer. The gateway forces stream_options: {"include_usage": true} onto every streaming request so the inference server puts token counts on the final chunk. Then it forwards each chunk immediately, with one concession to accounting: a substring scan for "usage" on each chunk, so the thousands of token chunks that cannot carry usage skip JSON parsing entirely and only the final chunk gets decoded.

The response body is wrapped in a guard stream that fires the captured usage, latency, and status through a oneshot channel when the stream completes or the client disconnects. The previous implementation slept 500 ms after the response started and raced slow streams; the guard made completion the event, not the clock. Mid-stream disconnects still get billed for what was generated.

Usage writes never block the request path. They go through a bounded channel drained by a single task, which is the only writer to the usage and access tables. On shutdown the gateway stops accepting requests, then gives the drain task five seconds to flush; rows still held by in-flight streams past that are lost. We chose losing a final usage row over hanging SIGTERM. That is a tradeoff, and we would make it again.

The payoff: one table answers the auditor

Every successful completion writes one usage row; every request, including the rejected ones, writes one access row. The usage row looks roughly like this:

{ org, deployment, key_id, user_id, model, workspace,
  prompt_tokens, cached_prompt_tokens, completion_tokens,
  cost_usd, latency_ms, status_code }

"Who called what model, with how many tokens, and what did it cost" is one SELECT. Not a join across a dozen services, not a log-aggregation query with a regex in it. For a platform whose core promise is that prompts and completions never leave infrastructure we own, that auditability is not a feature on the roadmap. It is the product.

The decentralized alternative is N services each implementing auth, budgets, and metering with N levels of care, and an audit story assembled from all of them. Our opinion, and it is an opinion: at this size, the choke point is cheaper than the federation. The day that stops being true, the gateway is also the one place that already knows every caller, which makes it the right place to split from.