Per-token APIs (Fireworks, Together, Baseten Model APIs, DeepInfra, Groq) are the default: zero idle cost, zero ops, and a fixed catalog. Serverless GPU platforms (Modal, RunPod) run your container on a GPU and bill per second, from $0.000164/sec on a T4, with scale to zero and roughly one-second container boots. Dedicated endpoints (Together at $5.49 per H100 hour, Baseten at about $6.50) reserve capacity and bill continuously. The rule of thumb: use a token API unless the model is not in the catalog, and use dedicated only when isolation or predictable latency is the requirement rather than cost.
- Three products, three billing units: per token, per second, per GPU hour.
- Per-token APIs have zero idle cost because a warm pool is shared across every customer.
- Serverless GPU is per second and scales to zero. Modal boots a container in about a second.
- Dedicated endpoints bill while idle. That is the point, and the risk.
- A dedicated H100 needs roughly 2.9 billion input tokens a month to beat serverless on cost.
- Cold start is only your problem on the platforms where the model is yours.
The three products
| Per-token model API | Serverless GPU | Dedicated endpoint | |
|---|---|---|---|
| Examples | Fireworks, Together, Baseten Model APIs, DeepInfra, Groq | Modal, RunPod | Together dedicated, Baseten dedicated |
| Billing unit | Per million tokens | Per second of GPU, CPU, memory | Per GPU hour, continuous |
| Representative rate | GLM 5.2 at $1.40 / $4.40 | H100 at $0.001097/sec | H100 at $5.49/hour |
| Idle cost | Zero | Zero | Full price |
| Cold start | Not your problem | Yours to manage | None once warm |
| Model choice | The provider catalog | Anything you can containerize | Anything you can deploy |
| You operate | Nothing | The model server | The model, not the infra |
Cold start economics
Cold start is the cost of not having a warm process when a request arrives. Where that cost lands is the whole difference between the three products.
- Per-token API: there is no cold start you can observe, because thousands of other customers keep the model warm. This is a genuine structural advantage and the reason token APIs win at low and medium volume.
- Serverless GPU: container boot is about one second on Modal, and then your initialization runs. Loading a 40GB model from object storage is where the real seconds go; baking weights into the image cuts that from minutes to seconds, and memory snapshotting cuts it further.
- Dedicated endpoint: no cold start after the first deploy, because the instance never sleeps. You pay for that in idle hours.
Scenario: 500 requests/day, 2 seconds of GPU each = ~17 GPU-minutes/day
Per-token API (GLM 5.2, ~2k in + 500 out per request):
input 500 x 30 days x 2,000 = 30M tokens x $1.40 = $42.00
output 500 x 30 days x 500 = 7.5M tokens x $4.40 = $33.00
-------
$75.00/mo
Serverless GPU (Modal H100, 17 min/day + cold starts):
17 min x 30 days = 510 min = 30,600 s x $0.001097 = $33.57
plus CPU + memory, plus cold-start seconds on sparse traffic
roughly ~$45/mo
Dedicated H100 endpoint (Together, $5.49/hr, 24/7):
730 hours x $5.49 = $4,007/mo
The dedicated line is not a typo. At this volume it is 53x the
token API for the same work.That is the shape of it at low volume. Turn the volume up by 100x and the token API line grows linearly to $7,500 while the dedicated line does not move at all, and the ordering inverts. The crossover for a single dedicated H100 against GLM 5.2 serverless input pricing sits near 2.9 billion input tokens a month, which is roughly 1,100 input tokens per second sustained, every second of every day.
Which vendors sell which
| Vendor | Per-token API | Serverless GPU | Dedicated |
|---|---|---|---|
| Fireworks | Yes, three speed tiers | On-demand GPUs, $7.00/hr H100 | Reserved capacity, negotiated |
| Together | Yes, plus batch at up to 50% off | Clusters at $3.99/hr H100 | Yes, $5.49/hr H100 |
| Baseten | Yes, 13 curated models | No | Yes, per minute, about $6.50/hr H100 |
| DeepInfra | Yes, cheapest rates found | Custom deployments, $2.20/hr H100 | Yes |
| Groq | Yes, two text models, fastest throughput | No | Enterprise only |
| Modal | No | Yes, the reference implementation | Pin warm containers |
| OpenRouter | Yes, aggregated from everyone | No | No |
Two things fall out of that table. Groq is the narrowest catalog in the set (two production text models: gpt-oss-120b at $0.15 / $0.60 with a published 500 tokens per second, and gpt-oss-20b at $0.075 / $0.30 at 1,000 tokens per second) and by far the fastest, which is a coherent strategy rather than an oversight. And Modal is the only vendor here that sells no token API at all, which is why comparing its prices to Fireworks is a category error.
Choosing, in order
Is the model in a provider catalog?
If yes, use a per-token API and stop. Zero ops, zero idle cost, and at typical volumes it is also the cheapest. This resolves the large majority of cases and people talk themselves out of it far too often.
Is it a frontier model?
Then it is single-source. Go to Anthropic or OpenAI, or to an aggregator if you want one key across both. No open-model shop sells Claude Opus 5 or gpt-5.6-sol.
Is the model yours, private, or non-standard?
Serverless GPU. Modal if you write Python and want per-second billing with scale to zero; a GPU cloud if you want long-lived machines you administer.
Do you need isolation, residency, or a hard latency guarantee?
Dedicated endpoint, and budget for it honestly: about $4,000 a month for one H100 running continuously. Buy it for the guarantee, not for the arithmetic.
Can the work wait?
Then none of the above matters as much as moving it to batch, which is 50 percent off at Anthropic, OpenAI, Fireworks, and Together and needs no other change.
Questions people ask
What is serverless inference?
Running model inference without managing servers. In practice the term covers three different products: per-token model APIs where the provider owns the model and you pay for tokens, serverless GPU platforms where you own the model and pay per second of container runtime, and dedicated endpoints where you reserve capacity and pay per GPU hour continuously. They have different prices and different failure modes.
What is the difference between serverless inference and serverless GPU?
Serverless inference usually means a per-token model API: send a prompt, get a completion, pay per million tokens, never think about hardware. Serverless GPU means a platform like Modal gives you a container with a GPU attached, billed per second, and you supply the model weights, the server process, and the scaling logic. The first is minutes of setup; the second is an afternoon at least.
Do serverless inference APIs have cold starts?
Per-token APIs effectively do not, because the provider keeps one warm pool of each model shared across all customers. Serverless GPU platforms do: Modal boots a container in about a second, and everything after that is your model loading, which is where the real time goes. Dedicated endpoints have no cold start after the first deploy, because they never scale down, which is why they cost money while idle.
When is a dedicated endpoint cheaper than serverless?
Rarely on cost alone. A dedicated H100 at Together is $5.49 an hour, about $4,007 a month running continuously, which needs roughly 2.9 billion input tokens a month at GLM 5.2 serverless rates to break even. That is about 1,100 input tokens per second sustained, around the clock. Buy dedicated for isolation, a private model, data residency, or a latency guarantee, not to save money.
Is serverless inference cheaper than running my own GPU?
Below roughly 80 percent GPU utilization, yes, and by a lot. Owning or reserving a GPU means paying for every idle hour plus operating a model server, an autoscaler, and a pager rotation. Modal at an effective $3.95 per H100 hour of actual runtime beats a $3.19 reservation until you are running about 590 hours of a 730-hour month.
Which serverless inference provider should I start with?
If you need a frontier model, the lab directly. If you need a standard open model, any of Fireworks, Together, or Baseten, which quote essentially identical prices, so pick on latency and catalog. If you want the lowest rate, DeepInfra. If you want raw throughput, Groq. If the model is yours, Modal, whose free plan renews $30 of credits every month.
Sources
Every figure above was read from these pages on August 2026. Vendors reprice without notice; if you find a stale number, tell us.
- Modal pricing per-second GPU rates and plan credits
- Modal cold start guide container boot time, snapshots, weight preloading
- Together AI pricing serverless, cluster, and dedicated inference rates
- Baseten pricing Model API rates and per-minute dedicated GPU rates
- Groq models and pricing model list, rates, published tokens per second
- Fireworks pricing on-demand GPU rates and product surfaces