Modal review: serverless GPUs, and when a team needs them

What Modal actually is

The single most useful sentence about Modal: it is not an inference provider in the same sense as Fireworks or Together. Those companies sell you a model. Modal sells you a container with a GPU in it and a very good way to get your code into that container.

The Python-native workflow is the actual product. You write an ordinary function, add a decorator saying what hardware and image it needs, and Modal builds and schedules it. There is no Dockerfile to write for the common cases and no Kubernetes to configure for any case.

import modal

app = modal.App("embeddings")

@app.function(gpu="L4")
def embed(batch: list[str]) -> list[list[float]]:
    from sentence_transformers import SentenceTransformer
    model = SentenceTransformer("BAAI/bge-large-en-v1.5")
    return model.encode(batch).tolist()

# eight B300s for a training job
@app.function(gpu="B300:8")
def train():
    ...

# or express a fallback preference
@app.function(gpu=["H100", "A100-40GB:2"])
def run_on_80gb():
    ...

Available types run T4, L4, A10, L40S, A100 in 40GB and 80GB, RTX PRO 6000, H100, H200, B200, and B300. Multi-GPU is the :n suffix, up to 8 GPUs and 2,304 GB of GPU RAM on newer types; A10 instances cap at 4 GPUs and 96 GB.

Cold starts, honestly

Cold start is the fair criticism of every scale-to-zero platform, and Modal's documented numbers are better than most people assume and worse than the marketing implies.

• Container boot is about one second. That is Modal's own figure and it is the part Modal controls.
• Your initialization is the rest. Global-scope code and modal.enter methods run after boot. Loading a 40GB model from object storage is where the seconds actually go.
• Pre-downloading the weights into the image takes model loading from minutes to seconds for models in the tens of gigabytes, per Modal's own guide.
• Memory snapshotting captures container memory after warm-up and restores it on the next boot. Modal says this substantially reduces cold start; it does not publish a number, so measure it on your workload.

Practical rule: if requests arrive continuously, keep a warm container and cold start is irrelevant. If they arrive a few times an hour, snapshot and pre-bake weights. If they arrive twice a day, cold start is 20 seconds and nobody cares.

What it costs, in shape

Full rates are on the Modal pricing page. The shape matters more than the digits: you pay for GPU seconds, CPU core seconds, and GiB seconds separately, and only while a container is running.

That per-second granularity is the entire economic argument. A job that runs for 90 seconds costs 90 seconds. Compare that to a reserved H100 at $3.19 per hour from Together, which costs $2,329 a month whether or not you send it anything. Modal beats a reserved GPU on cost until you are above roughly 80 percent utilization, at which point the reservation wins.

When a coding team actually needs Modal

Most teams building with coding agents do not need Modal, and it is worth saying so plainly. If your entire AI usage is "call Claude or GPT from an agent," Modal solves a problem you do not have. Here is when it starts earning its place.

Where Modal is the wrong choice

• You want an endpoint today. A token API is a five-minute job. Modal is an afternoon, and a longer one if you have never packaged a model server.
• You do not write Python. Modal is Python-first. You can run any container, but the ergonomics that make Modal worth choosing are Python ergonomics.
• Your load is constant and large. Above roughly 80 percent utilization a reserved GPU is cheaper, and above that a committed contract is cheaper still.
• You need a low, predictable p99 on sporadic traffic. Scale-to-zero and tight tail latency are in tension. You can pin warm containers, but then you are paying for idle and Modal's pricing advantage disappears.

Questions people ask

Keep reading

Continue through Inference providers with Modal pricing: every GPU rate, free credits, and worked costs.