LLM inference speed: tokens per second is the wrong metric

Three metrics, three different questions

Those definitions are Artificial Analysis's, which is the closest thing this category has to a standard. Two details matter when you read anyone's numbers. They normalize to OpenAI tokens rather than each model's native tokenizer, precisely because a native-token count is not comparable across models. And they measure across a diverse 60-prompt set of varying lengths rather than a single benchmark prompt.

Why agents care about a different number than chat

A chat turn is one request producing one long answer. Its wall clock is roughly TTFT plus (output tokens / throughput), and since answers run to hundreds of tokens, throughput dominates.

A coding agent turn is nothing like that. It reads a file, thinks briefly, runs a command, reads the output, patches, runs tests. Each of those is a separate model call producing a short output, with tool execution in between. Its wall clock is roughly turn count times TTFT, plus tool time, plus a comparatively small amount of generation.

Provider A: TTFT 0.50s, throughput 450 tok/s
Provider B: TTFT 1.20s, throughput 700 tok/s

CHAT: one call, 800 output tokens
  A: 0.50 + 800/450 = 0.50 + 1.78 = 2.28 s
  B: 1.20 + 800/700 = 1.20 + 1.14 = 2.34 s
  -> Essentially tied. B's throughput almost erases its TTFT deficit.

AGENT: 20 calls, 120 output tokens each, 0.4s tool time between
  A: 20 x (0.50 + 120/450 + 0.4) = 20 x 1.167 = 23.3 s
  B: 20 x (1.20 + 120/700 + 0.4) = 20 x 1.771 = 35.4 s
  -> A is 34% faster. The TTFT gap was multiplied twenty times.

That is why a provider leaderboard sorted by tokens per second can point you at the wrong vendor for an agentic workload. Sort by time to first token instead, and weight it by how many turns your tasks actually take. Turn count is a property of the model rather than the provider, which is why our coding-model leaderboard prints steps and output tokens per task alongside the pass rate.

Who is actually fast, August 2026

These are the independently measured figures from the Artificial Analysis provider leaderboard, read in August 2026. They move constantly, so treat them as a snapshot and check the live board before quoting them.

Groq's own documentation publishes per-model figures rather than a single company number: 1,000 tokens per second on gpt-oss 20B and 500 on gpt-oss 120B. Publishing a number per model is more honest than publishing one for the company, and it is worth noting that Groq's speed advantage comes with the narrowest catalog in the category.

Speculative decoding and quantization, honestly

Speculative decoding

A small draft model proposes several tokens ahead, and the large model verifies them in a single forward pass. When the draft is right, you get several tokens for the price of one step. When it is wrong, the speculated tokens are discarded and you have burned compute for nothing. The net effect on throughput is real and workload dependent: highly predictable text (boilerplate, code with strong conventions, structured output) accepts a high fraction of drafts; genuinely novel reasoning accepts far less. Together's research lineage here is public, with Medusa, Sequoia, and SpecExec all published work that feeds its serving stack. The honest summary: it is free speed on predictable output and roughly neutral on unpredictable output, and it never changes the distribution of what the large model would have said.

Quantization

Storing weights at lower precision (FP8, INT8, NVFP4 and below) shrinks memory traffic, which is what actually bounds decode speed, so a quantized model streams faster and fits on smaller GPUs. The cost is quality, and the size of that cost is model and task specific rather than a fixed percentage. It shows up first on the tasks with the least slack: long-chain reasoning, precise arithmetic, and exact-format output. Note that providers do not always label it prominently; the Artificial Analysis board carries "GLM-5.2 NVFP4" as a distinct entry from GLM-5.2 for exactly this reason. If two providers quote very different speeds for what looks like the same model, check the precision before concluding one has better engineering.

Measure it yourself

Any published board is measured on someone else's prompts from someone else's network. Twenty lines gets you your own numbers on your own prompts.

import os, time, json, urllib.request

BASE = os.environ.get("BASE_URL", "https://api.openai.com/v1")
KEY  = os.environ["API_KEY"]
MODEL = os.environ.get("MODEL", "gpt-5.6-luna")

req = urllib.request.Request(
    f"{BASE}/chat/completions",
    headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"},
    data=json.dumps({
        "model": MODEL,
        "stream": True,
        "messages": [{"role": "user", "content": "Write a Python LRU cache. Code only."}],
    }).encode(),
)

sent = time.perf_counter()
first = None
tokens = 0

with urllib.request.urlopen(req) as resp:
    for raw in resp:
        line = raw.decode().strip()
        if not line.startswith("data: ") or line == "data: [DONE]":
            continue
        chunk = json.loads(line[6:])
        delta = chunk["choices"][0]["delta"].get("content")
        if not delta:
            continue
        if first is None:
            first = time.perf_counter()
        tokens += 1

done = time.perf_counter()
print(f"TTFT          {first - sent:.3f} s")
print(f"Output speed  {tokens / (done - first):.1f} chunks/s")
print(f"Total         {done - sent:.3f} s")

• Run it 20 times, report p50 and p95. A single sample tells you nothing; tail latency is what users notice.
• Run it from where your code runs. Measuring from a laptop in Dubai against a US-East endpoint measures the Atlantic, not the provider.
• Use your real prompt lengths. TTFT scales with input size, and an agent sending 25,000 tokens of repo context has a very different TTFT from a 20-token benchmark prompt.
• Count chunks, not tokens, unless you tokenize. Streaming deltas are not one token each. The script above is honest about measuring chunks; for true tok/s, read the usage field the provider returns at the end of the stream.
• Measure at the hour you actually work. Shared capacity is contended, and a 3am number is not your number.

Questions people ask

Keep reading

Continue through Inference providers with Free LLM API: every genuinely free option, with the catch for each.