Field notes: running an AI gateway in front of every model call

The Frontier Wire editors — Fri, 18 Sep 2026 00:00:00 GMT

An AI gateway is a proxy that sits between your applications and the model providers they call. Every request goes through it, and in exchange it gives you one place to handle keys, routing, retries, budgets and logging. This piece is about what that looked like for a mid-sized engineering team that put one in front of everything, and which of the promised benefits materialised.

Why we did it

We had four services calling three providers with keys scattered through environment variables. A pricing change at one provider took a week to fully roll out because each team had hard-coded model names. When one provider had an outage, three of the four services simply failed. None of this was exotic; it was the natural state of a team that had adopted models quickly.

What the gateway gave us

One interface. Every service now speaks a single request format and names a model by an alias. Swapping the model behind an alias is a configuration change, not a deploy. This alone paid for the migration.

Fallbacks and retries. When a provider returns an error or times out, the gateway retries and then fails over to a second provider. Outages became latency blips rather than incidents.

Budgets by team. Each service authenticates with its own virtual key, and each key has a monthly spend limit and a rate limit. Finance can see who spent what without asking engineering.

A single log. Every request, its latency, token counts and cost land in one place. This turned out to be the feature we used most: it is where we found the two endpoints responsible for most of our spend.

The gateway did not make the models better. It made our use of them legible.

What it cost

Latency overhead was small: single-digit milliseconds per request once the gateway was deployed in the same region as our services. The larger costs were organisational. Someone has to own the gateway's configuration, and the routing rules become a thing that can be wrong. We had one incident where a fallback rule silently sent traffic to a model that was cheaper but noticeably worse, and nobody noticed for two days because the requests were succeeding.

That incident is why evaluation now sits beside the gateway. Routing changes are tested against a fixed set of real prompts before they ship.

Choosing one

The category is crowded and the feature lists look similar. Four things separated the options for us:

Criterion	What to check
Overhead	Measure p99 latency added at your real request rate, not the vendor's number.
Provider coverage	Every provider you use today and the two you are evaluating.
Governance	Per-key budgets, rate limits, and an audit log that survives a restart.
Deployment	Self-hosted in your own network, or managed. Data residency usually decides this.

We shortlisted three, ran a week of shadow traffic through each, and chose on overhead and governance. The exact product matters less than the discipline of testing it with your own traffic before you commit.

The Frontier Wire: Tools

Field notes: running an AI gateway in front of every model call

Why we did it

What the gateway gave us

What it cost

Choosing one