I have a minor problem with local-LLM benchmark screenshots: they usually contain one large number, the unit tok/s, and exactly enough context to mislead everyone in the room.

A model can be fast at swallowing a long prompt and slow at writing an answer. It can be snappy for an interactive chat and terrible once several people hit the server at once. It can also look heroic in a synthetic loop that bears little resemblance to the thing you bought the machine for.

So no: “my model does 18 tokens per second” is not a benchmark result. It is the beginning of a benchmark result.

Inference has two jobs

For the decoder-only models most of us run locally, a request has two materially different phases:

  1. Prompt processing (also called prefill): the model consumes the input and builds its KV cache.
  2. Text generation (also called decode): the model produces the response one token at a time, extending that cache as it goes.

The official llama-bench tool treats them as separate tests: pp for prompt processing, tg for text generation, and pg for a combined prompt-plus-generation run. That is not tool-builder pedantry. It is the whole point. The same executable has separate knobs because the workloads are not interchangeable.

Prompt processing answers: how long before the answer can begin? Generation answers: how quickly does the answer arrive once it has begun? Mash those together and the result depends heavily on an arbitrary prompt length and output length chosen by whoever wanted the prettier number.

A benchmark that reports only one throughput figure is like reviewing a car with only its top speed. Technically a measurement. Practically a dodge.

The two scores I actually want

For an interactive local assistant, I want these reported separately:

  • Prompt-processing throughput at realistic input sizes.
  • Generation throughput at realistic output sizes.

Then I want one end-to-end run that states the exact shape of the request: prompt tokens, generated tokens, context size, batch size, backend, quantization, and repetitions. Without those, a number is trivia.

llama-bench makes the split explicit. A prompt-only test uses -p; a generation-only test uses -n; a combined test uses -pg. It also repeats tests and reports an average and standard deviation. That is a much healthier starting point than timing one lucky completion while the machine has nothing else to do.

Here is a deliberately boring baseline shape:

# Prompt ingestion at short, medium, and long inputs
llama-bench -m model.gguf -p 512,2048,8192 -n 0 -r 5

# Decode speed for a useful answer length
llama-bench -m model.gguf -p 0 -n 256 -r 5

# What a real request feels like
llama-bench -m model.gguf -pg 2048,256 -r 5

The commands are not magic. Their value is that they force the question to be specific.

Match the benchmark to the job

Different local workloads punish different failures.

Chat and coding assistance care about the gap before the first visible token and the cadence after it. A machine that decodes briskly but takes an age to ingest a long conversation feels broken, even if its headline generation score is great.

RAG, document analysis, and large system prompts lean hard on prompt processing. If the useful work starts with several thousand retrieved tokens, testing only a tiny prompt is self-deception with a stopwatch.

Batch extraction may care more about aggregate throughput than one user’s first-token latency. The llama.cpp project has a separate batched benchmark for this reason. It distinguishes prompt-processing time from generation work and lets you test multiple parallel sequences. A single-user loop cannot tell you how a server behaves under concurrent demand.

Long-running agents need all of the above plus repeatability. An agent that reads large tool results, produces a short action, and repeats is neither a pure chat workload nor a bulk-generation workload. Measure the loop shape you actually run.

This is why there is no universal “best local model speed.” There are only models and machines that are fast for a stated request distribution. Annoying answer? Yes. Still the honest one.

What a credible result includes

I would reject any local benchmark missing these fields:

  • model file and quantization;
  • inference engine and backend;
  • prompt and generation token counts;
  • context window and batch size;
  • number of repetitions and variance;
  • whether the result is prompt processing, decoding, or end-to-end;
  • concurrency, if it is a server claim.

There is one more trap. llama-bench documents that its measurements do not include tokenization or sampling. That is fine; it is a focused engine benchmark. But it means its figure is not the full wall-clock experience of an application request. HTTP handling, queueing, prompt formatting, tokenization, sampling, streaming, and contention all live outside that tidy number.

Do not misuse a precise measurement because it is conveniently incomplete.

A benchmark card I can trust

If I were publishing a result for a local setup, I would use a small card like this:

Model: <exact GGUF and quantization>
Engine/backend: <exact build and backend>
Context / batch: <values>
Repetitions: 5

Prompt processing: 2,048-token prompt, mean ± standard deviation
Generation: 256-token decode, mean ± standard deviation
End-to-end: 2,048-token prompt + 256-token response, wall-clock mean ± standard deviation
Concurrency: 1 (or explicitly stated server load)

That card is less exciting than a giant tok/s graphic. Good. It is also reusable, comparable, and hard to accidentally lie with.

I am not arguing against tokens per second. I am arguing against pretending there is only one kind. Prompt ingestion and token-by-token decoding are distinct jobs, and a useful benchmark gives each one a score.

Anything less is two benchmarks wearing a trench coat.

Sources