The Frontier Wire

The reasoning-model era has a cost problem

The Frontier Wire editors — Thu, 24 Sep 2026 00:00:00 GMT

TL;DR

Reasoning models spend tokens thinking before they answer, and those tokens are billed as output.

The result is a query whose cost varies by an order of magnitude depending on how hard the question is.

Buyers are responding with routing, budgets and evaluation, not by picking a single model.

For two years the price of a large language model call moved in one direction. Providers cut list prices, competition increased, and the standard advice was to wait a quarter and pay less. Reasoning models have broken that pattern. The same question, asked of a model that thinks first, can cost more than the entire conversation would have cost a year ago.

What changed

A reasoning model does not answer immediately. It generates an intermediate chain of tokens, checks its own work, sometimes backtracks, and only then produces the reply the user sees. Providers bill those intermediate tokens as output, whether or not they are shown. A question that produces a fifty-word answer might generate several thousand words of reasoning on the way.

That is the mechanism behind the cost problem. Output tokens are the expensive kind, and reasoning multiplies them. The per-token price may be similar to a non-reasoning model, but the number of tokens is not.

Why it is worth paying for

The trade is not a bad one. Research on test-time compute has shown that letting a smaller model think longer can match or beat a much larger model on hard problems, at lower total cost. For maths, code, multi-step planning and anything with a verifiable answer, the accuracy gain is real and often decisive.

The question is no longer which model is best. It is how much thinking each request deserves.

The difficulty is that the gain is uneven. Easy questions do not benefit from extended reasoning at all, and they still pay for it if the model is allowed to think. A support chatbot that routes every message to a reasoning model is paying a research-grade price for questions about opening hours.

What buyers are doing about it

Three patterns have emerged among teams running these models in production.

Routing by difficulty. A cheap classifier or a small model decides whether a request needs reasoning. Simple requests go to a fast model; hard ones get the expensive path. This is the single largest lever, and it is why the AI gateway category has grown so quickly.

Thinking budgets. Most providers now expose an effort setting that caps reasoning tokens. Teams tune it per use case, accepting a small accuracy loss for a large cost reduction on the middle of the distribution.

Evaluation before rollout. Because cost is now a function of the question, the only way to forecast a bill is to run a representative sample of real traffic through each candidate configuration and measure both accuracy and spend. Teams that skipped this step have reported monthly invoices several times their estimate.

What to watch

The economics will keep moving. Providers are working on models that decide for themselves how long to think, which would move routing inside the model. Hardware improvements continue to cut the cost of every token. But the structural change is permanent: capability now has a dial, and the dial has a price. Anyone buying model access needs to know where their dial is set.

What test-time compute actually means, and why it changed the scaling debate

The Frontier Wire editors — Mon, 21 Sep 2026 00:00:00 GMT

For most of the last decade, the recipe for a better language model was simple: more parameters, more data, more training compute. The scaling laws that described this relationship were remarkably reliable, and they drove the industry's spending. Test-time compute is the name for a second lever that was always available but rarely pulled: spend more computation when the model is answering, not only when it is being trained.

The basic idea

A trained model is a fixed function. Given a prompt, it produces a distribution over the next token, and sampling from that distribution once gives you one answer. Test-time compute means doing more than that single pass. The extra work can take several forms:

Longer generation. Let the model write out intermediate steps before committing to an answer. This is chain-of-thought, and it works because each step conditions the next.
Multiple samples. Generate several candidate answers and pick the most common one, or the one a verifier scores highest. This is self-consistency and best-of-N sampling.
Search. Treat reasoning as a tree, expand the promising branches, prune the rest. This borrows from game-playing systems.
Revision. Ask the model to critique and rewrite its own draft, possibly several times.

Each of these turns a fixed cost per query into a variable one. The model can spend a little or a lot, depending on how hard the problem is.

Why it changed the debate

The important finding, established in a series of papers through 2024, is that the extra inference compute is often a better buy than the same compute spent on a bigger model. For problems where the model has a reasonable chance of getting the answer right, sampling and verifying can push accuracy well past what a model several times larger achieves with a single pass.

Compute is fungible. What matters is where in the pipeline you spend it.

This mattered for three reasons. It gave smaller models a path to frontier-level results on specific tasks. It meant capability could be tuned per request rather than fixed at training time. And it shifted a chunk of the industry's cost from one-off training runs to ongoing inference, which is a very different business.

The catch

Test-time compute helps most when there is a way to tell a good answer from a bad one. Maths, code with tests, and puzzles with checkable solutions are ideal. Open-ended writing, judgment calls and questions with no verifiable answer benefit far less, because there is nothing for the search or the verifier to grip.

There is also a hard ceiling. If the model cannot produce a correct answer in any of its samples, no amount of selection will find one. Thinking longer sharpens what a model can already almost do; it does not teach it something new.

How to think about it as a buyer

Treat reasoning effort as a knob with a price. Turn it up for tasks with verifiable outputs and high stakes. Turn it down, or off, for tasks where a first draft is good enough. And measure: the only reliable way to know where the knob should sit for your traffic is to run your own examples through each setting and compare accuracy against cost. Our news desk has more on why that bill has become a product decision.

Field notes: running an AI gateway in front of every model call

The Frontier Wire editors — Fri, 18 Sep 2026 00:00:00 GMT

An AI gateway is a proxy that sits between your applications and the model providers they call. Every request goes through it, and in exchange it gives you one place to handle keys, routing, retries, budgets and logging. This piece is about what that looked like for a mid-sized engineering team that put one in front of everything, and which of the promised benefits materialised.

Why we did it

We had four services calling three providers with keys scattered through environment variables. A pricing change at one provider took a week to fully roll out because each team had hard-coded model names. When one provider had an outage, three of the four services simply failed. None of this was exotic; it was the natural state of a team that had adopted models quickly.

What the gateway gave us

One interface. Every service now speaks a single request format and names a model by an alias. Swapping the model behind an alias is a configuration change, not a deploy. This alone paid for the migration.

Fallbacks and retries. When a provider returns an error or times out, the gateway retries and then fails over to a second provider. Outages became latency blips rather than incidents.

Budgets by team. Each service authenticates with its own virtual key, and each key has a monthly spend limit and a rate limit. Finance can see who spent what without asking engineering.

A single log. Every request, its latency, token counts and cost land in one place. This turned out to be the feature we used most: it is where we found the two endpoints responsible for most of our spend.

The gateway did not make the models better. It made our use of them legible.

What it cost

Latency overhead was small: single-digit milliseconds per request once the gateway was deployed in the same region as our services. The larger costs were organisational. Someone has to own the gateway's configuration, and the routing rules become a thing that can be wrong. We had one incident where a fallback rule silently sent traffic to a model that was cheaper but noticeably worse, and nobody noticed for two days because the requests were succeeding.

That incident is why evaluation now sits beside the gateway. Routing changes are tested against a fixed set of real prompts before they ship.

Choosing one

The category is crowded and the feature lists look similar. Four things separated the options for us:

Criterion	What to check
Overhead	Measure p99 latency added at your real request rate, not the vendor's number.
Provider coverage	Every provider you use today and the two you are evaluating.
Governance	Per-key budgets, rate limits, and an audit log that survives a restart.
Deployment	Self-hosted in your own network, or managed. Data residency usually decides this.

We shortlisted three, ran a week of shadow traffic through each, and chose on overhead and governance. The exact product matters less than the discipline of testing it with your own traffic before you commit.

How to compare LLM evaluation frameworks without getting sold to

The Frontier Wire editors — Tue, 15 Sep 2026 00:00:00 GMT

Evaluation is the part of building with language models that everyone agrees is important and most teams do badly. The tooling has matured, but comparing frameworks is hard because their marketing converges on the same words: observability, quality, guardrails. This guide lays out the criteria that actually separate them, so you can score candidates yourself.

Start with what you are evaluating

There are three distinct jobs, and a framework can be good at one and poor at the others.

Offline evaluation of prompts and models against a fixed dataset, before anything ships.
Online evaluation of production traffic, sampling real requests and scoring them after the fact.
Agent evaluation, where the unit is a multi-step trajectory with tool calls rather than a single response.

Decide which of these you need in the next six months. A framework built for single-turn offline tests will not stretch to agents without a rewrite.

The seven criteria

Criterion	Why it matters	Question to ask
Evaluator types	Exact match, model-graded, human, and code-based checks each catch different failures.	Can I write a custom evaluator in my own language?
Dataset management	Test sets drift and grow; versioning them is how you compare runs.	Can I diff two dataset versions?
Trace depth	Agent failures live in the middle of a trajectory, not the final answer.	Does a trace show every tool call and its inputs?
Human review	Some judgments need a person; the workflow determines whether anyone does it.	Can a non-engineer score a queue of outputs?
CI integration	An evaluation that does not block a bad deploy is a dashboard.	Can a score regression fail a pull request?
Production sampling	Offline sets miss the long tail of real usage.	Can I sample live traffic into a dataset?
Cost and latency reporting	Quality is one axis; the other two decide what ships.	Are tokens and latency recorded next to scores?

How to run the comparison

Take one workflow you actually run, ideally one that has broken before. Build the same evaluation in each candidate. Time how long it took, count how many features you had to work around, and note who on the team could operate it without help. This takes a day per tool and is worth more than any demo.

A framework you can run in CI beats a prettier one you check by hand.

Red flags

Be cautious of frameworks that score only with a model grader and offer no way to calibrate it against human labels. Be cautious of anything that stores traces but cannot turn a trace into a test case. And be cautious of a vendor who cannot show a score going down; the whole point is catching regressions, and a tool that has never displayed one has not been used seriously.

If you need a wider view of the space, our surveys desk maintains a map of the current landscape.

Stop calling it hallucination

The Frontier Wire editors — Sat, 12 Sep 2026 00:00:00 GMT

Every article about language models eventually reaches the paragraph about hallucination: the model made something up, cited a paper that does not exist, invented a court case. The word has become the industry's shorthand for being wrong. It is a bad word, and it is doing damage.

The word describes a malfunction that is not happening

A hallucination is a perception with no external cause. Applied to a language model, it suggests a system that normally reports the truth and occasionally suffers a glitch. That is not the mechanism. A language model produces the continuation that is most probable given its training and the prompt. When the probable continuation matches reality, we call it a fact. When it does not, we call it a hallucination. The model did the same thing both times.

This matters because the word invites the wrong fix. If the problem is a glitch, you patch it. If the problem is that the system has no distinction between recall and plausible invention, you need to change what you ask of it and how you check it.

The model is not lying and it is not malfunctioning. It is answering a question it was never asked: what would a good answer look like?

What to say instead

Confabulation is closer, and has a respectable history in neuroscience, where it describes a person filling a memory gap with a plausible story and believing it. But it still locates the problem inside the model. The more useful framing is that the model is guessing, that guessing is what it is built to do, and that the interesting question is how confident the guess is and whether anyone checked.

That framing points at the real fixes: grounding the model in retrieved documents so the probable continuation is anchored to a source, asking for citations that can be verified mechanically, and evaluating outputs against references instead of trusting them. None of those are patches for a glitch. They are the ordinary engineering of an unreliable component.

Why the industry likes the word

It is a comfortable word. It suggests the models are almost right, that the errors are anomalies, and that a future version will not have them. It sells a roadmap. Being honest, saying the model guesses and sometimes guesses wrong, sells nothing and obliges the vendor to explain what the buyer should do about it.

Buyers should insist on the honest version. Ask what the model's error rate is on your task, not whether it hallucinates. Ask what the product does when the guess is wrong. And when a vendor tells you the next model hallucinates less, ask them to show the evaluation. Our comparisons desk has a guide to what that evaluation should contain.

The state of open-weight licensing, mapped

The Frontier Wire editors — Tue, 08 Sep 2026 00:00:00 GMT

"Open" has become the most overloaded word in AI. Model weights that anyone can download are described as open, and so are models released under terms that forbid certain uses, cap the size of the company that may use them, or reserve the right to revoke access. These are not the same thing, and the difference decides whether you can build a product on the model. This survey maps the licence families in use and what each permits.

The families

Family	Examples of terms	Commercial use	Restrictions on use	Open by the OSI definition
Permissive open source	Apache 2.0, MIT	Yes	None beyond attribution	Yes
Community licences	Vendor-written "community" terms	Yes, with conditions	Often a user-count threshold, naming rules, or a ban on training competing models	No
Responsible AI licences	RAIL and variants	Yes	A list of prohibited uses that travels with the model	No
Research-only	Non-commercial terms	No	Research and evaluation only	No
Custom terms of use	Vendor-specific	Varies	Varies; often includes acceptable-use policies that can change	No

Permissive open source

A minority of notable models ship under Apache 2.0 or MIT. These are the only licences in the table that meet the Open Source Definition: anyone may use, modify and redistribute the weights for any purpose. If you need certainty that your rights will not change, this is the family to build on.

Community licences

The most common terms on large, capable models are vendor-written licences that grant broad commercial use with specific carve-outs. The usual ones are a threshold on the licensee's monthly active users above which a separate licence is required, a requirement to display attribution, and a prohibition on using the model's outputs to improve other models. These licences are workable for most companies but are not open source, and the carve-outs are exactly where a legal review should focus.

Read the clause about training other models. It is the one most teams break without noticing.

Responsible AI licences

RAIL-style licences attach a use-based restriction list to the model: no use for surveillance, for generating disinformation, for certain medical claims, and so on. The intent is ethical, and the restrictions are ones most businesses would never approach. The practical problem is that use restrictions are incompatible with the open-source definition and are difficult for downstream projects to inherit cleanly.

Research-only and custom terms

Some releases are explicitly non-commercial. Others ship under terms of use that reference an acceptable-use policy hosted on the vendor's website, which the vendor may update. A model licensed this way can become unusable for your product without the weights changing at all.

What to do with this

Before building on any open-weight model, find the licence text, not the announcement, and answer four questions: Is commercial use allowed? Is there a size or user threshold? Can outputs be used to train other models? Can the terms change after download? The answers sort every model in this survey into one of the five families above, and that placement is more informative than any benchmark.

RAG vs. long context: choose by failure mode, not by benchmark

The Frontier Wire editors — Fri, 04 Sep 2026 00:00:00 GMT

Two years ago the choice was forced: context windows were small, so anything beyond a few pages had to be retrieved. Windows have since grown by orders of magnitude, and a reasonable person now asks whether retrieval-augmented generation (RAG) is worth the machinery. The honest answer is that the two approaches fail in different ways, and the decision should be made by asking which failure you can live with.

How each one fails

Long context fails quietly. Give a model a very long document and it will usually answer well, but attention is not uniform. Research on positional effects showed that information in the middle of a long context is used less reliably than information at the start or end, and the effect persists in newer models even as it shrinks. The model does not tell you it skimmed. It simply answers with slightly less of the document than you gave it.

RAG fails loudly, and earlier. If the retriever does not surface the right passage, the model cannot use it. The failure is visible in the retrieved set, which means it can be measured and fixed, but it also means a mediocre retriever caps the whole system regardless of how good the model is.

Long context hides its misses inside a fluent answer. Retrieval shows you its misses before the model speaks.

The other axes

Axis	Long context	RAG
Cost per query	Scales with document size; you pay for every token every time	Roughly constant; only the retrieved passages are billed
Latency	Grows with input length	Retrieval adds a step, but the model call stays small
Freshness	Requires re-sending the updated document	Update the index; the next query sees it
Corpus size	Bounded by the window	Effectively unbounded
Debuggability	Hard: the model saw everything	Easier: inspect what was retrieved
Setup effort	Minimal	An index, a chunking strategy, and an embedding model to maintain

A decision rule

Use long context when the corpus is small enough to fit comfortably, changes rarely, and you value simplicity over per-query cost. A contract, a codebase module, a single long report.

Use retrieval when the corpus is larger than a window, changes often, or per-query cost matters. A knowledge base, a ticket history, anything multi-tenant.

Use both when the stakes are high: retrieve widely, then let a long window read everything retrieved instead of only the top few passages. This is the configuration most serious production systems have converged on, and it turns the two failure modes into checks on each other.

What to measure

Whichever you choose, build an evaluation set of questions with known answers and known source passages. Score retrieval recall separately from answer accuracy. If accuracy is low but recall is high, the model is the problem. If recall is low, no model will save you. Our comparisons desk has more on what an evaluation setup should contain.

Reading a model card like an auditor

The Frontier Wire editors — Sat, 29 Aug 2026 00:00:00 GMT

The model card was proposed in 2018 as a short, standardised document describing what a model is for, how it was evaluated, and where it should not be used. The idea caught on. Nearly every model release now ships with one. The quality has not kept pace with the adoption, and a card can now be anything from a rigorous disclosure to a benchmark table with a logo on it. This is a guide to reading one critically.

What a good card contains

The original proposal listed sections that remain the right checklist: intended use and out-of-scope uses, the evaluation data and metrics, disaggregated results across relevant groups, ethical considerations, and caveats. A card that covers each of those, with specifics, is doing its job.

The questions to ask

Which benchmarks, and which versions? Benchmark names are not enough. Many have multiple versions and evaluation settings, and scores are not comparable across them. A card that names a benchmark without a version or a prompt format is reporting a number you cannot check.

Who ran the evaluation? Vendor-run evaluations are the norm, and not necessarily wrong, but the card should say so. Independent results, where they exist, belong beside them.

Is anything disaggregated? An aggregate score hides variation. A card that reports one number per benchmark and nothing by language, domain or demographic group has not done the part of the work that model cards were invented for.

What is missing? The most informative section of many cards is the one that is not there. Training data is the usual gap. Compute is another. The absence is itself a disclosure.

A card that reports only the benchmarks the model wins is not a model card. It is a press release with a table.

Red flags

Be wary of cards that describe intended use so broadly that nothing is out of scope. Be wary of cards whose evaluation section changed between the preview and the general release without a changelog. And be wary of comparisons against competitors whose numbers were taken from a different evaluation setup than the vendor's own.

Why it matters

Model cards are the document a buyer will be asked to produce when a regulator, a customer or an incident review asks what was known about the model at the time. Reading them like an auditor now is cheaper than explaining later why nobody did. For the same discipline applied to evaluation tools, see our guide to comparing evaluation frameworks.