How to compare LLM evaluation frameworks without getting sold to

The Frontier Wire editors — Tue, 15 Sep 2026 00:00:00 GMT

Evaluation is the part of building with language models that everyone agrees is important and most teams do badly. The tooling has matured, but comparing frameworks is hard because their marketing converges on the same words: observability, quality, guardrails. This guide lays out the criteria that actually separate them, so you can score candidates yourself.

Start with what you are evaluating

There are three distinct jobs, and a framework can be good at one and poor at the others.

Offline evaluation of prompts and models against a fixed dataset, before anything ships.
Online evaluation of production traffic, sampling real requests and scoring them after the fact.
Agent evaluation, where the unit is a multi-step trajectory with tool calls rather than a single response.

Decide which of these you need in the next six months. A framework built for single-turn offline tests will not stretch to agents without a rewrite.

The seven criteria

Criterion	Why it matters	Question to ask
Evaluator types	Exact match, model-graded, human, and code-based checks each catch different failures.	Can I write a custom evaluator in my own language?
Dataset management	Test sets drift and grow; versioning them is how you compare runs.	Can I diff two dataset versions?
Trace depth	Agent failures live in the middle of a trajectory, not the final answer.	Does a trace show every tool call and its inputs?
Human review	Some judgments need a person; the workflow determines whether anyone does it.	Can a non-engineer score a queue of outputs?
CI integration	An evaluation that does not block a bad deploy is a dashboard.	Can a score regression fail a pull request?
Production sampling	Offline sets miss the long tail of real usage.	Can I sample live traffic into a dataset?
Cost and latency reporting	Quality is one axis; the other two decide what ships.	Are tokens and latency recorded next to scores?

How to run the comparison

Take one workflow you actually run, ideally one that has broken before. Build the same evaluation in each candidate. Time how long it took, count how many features you had to work around, and note who on the team could operate it without help. This takes a day per tool and is worth more than any demo.

A framework you can run in CI beats a prettier one you check by hand.

Red flags

Be cautious of frameworks that score only with a model grader and offer no way to calibrate it against human labels. Be cautious of anything that stores traces but cannot turn a trace into a test case. And be cautious of a vendor who cannot show a score going down; the whole point is catching regressions, and a tool that has never displayed one has not been used seriously.

If you need a wider view of the space, our surveys desk maintains a map of the current landscape.

RAG vs. long context: choose by failure mode, not by benchmark

The Frontier Wire editors — Fri, 04 Sep 2026 00:00:00 GMT

Two years ago the choice was forced: context windows were small, so anything beyond a few pages had to be retrieved. Windows have since grown by orders of magnitude, and a reasonable person now asks whether retrieval-augmented generation (RAG) is worth the machinery. The honest answer is that the two approaches fail in different ways, and the decision should be made by asking which failure you can live with.

How each one fails

Long context fails quietly. Give a model a very long document and it will usually answer well, but attention is not uniform. Research on positional effects showed that information in the middle of a long context is used less reliably than information at the start or end, and the effect persists in newer models even as it shrinks. The model does not tell you it skimmed. It simply answers with slightly less of the document than you gave it.

RAG fails loudly, and earlier. If the retriever does not surface the right passage, the model cannot use it. The failure is visible in the retrieved set, which means it can be measured and fixed, but it also means a mediocre retriever caps the whole system regardless of how good the model is.

Long context hides its misses inside a fluent answer. Retrieval shows you its misses before the model speaks.

The other axes

Axis	Long context	RAG
Cost per query	Scales with document size; you pay for every token every time	Roughly constant; only the retrieved passages are billed
Latency	Grows with input length	Retrieval adds a step, but the model call stays small
Freshness	Requires re-sending the updated document	Update the index; the next query sees it
Corpus size	Bounded by the window	Effectively unbounded
Debuggability	Hard: the model saw everything	Easier: inspect what was retrieved
Setup effort	Minimal	An index, a chunking strategy, and an embedding model to maintain

A decision rule

Use long context when the corpus is small enough to fit comfortably, changes rarely, and you value simplicity over per-query cost. A contract, a codebase module, a single long report.

Use retrieval when the corpus is larger than a window, changes often, or per-query cost matters. A knowledge base, a ticket history, anything multi-tenant.

Use both when the stakes are high: retrieve widely, then let a long window read everything retrieved instead of only the top few passages. This is the configuration most serious production systems have converged on, and it turns the two failure modes into checks on each other.

What to measure

Whichever you choose, build an evaluation set of questions with known answers and known source passages. Score retrieval recall separately from answer accuracy. If accuracy is low but recall is high, the model is the problem. If recall is low, no model will save you. Our comparisons desk has more on what an evaluation setup should contain.

The Frontier Wire: Comparisons