H100 vs B200 Cloud GPUs: Which Rental Actually Saves You Money?

What Each GPU Is Actually For

The H100 (Hopper) and B200 (Blackwell) are both datacenter accelerators, but they target different phases of the ML lifecycle. The H100 is the workhorse of the current generation—it’s what most teams rent for fine-tuning, LoRA, and serving models up to 70B with quantization. It has 80GB of HBM3, which is enough for a 70B model in FP8 or a 13B model with big batch sizes.

The B200 is Blackwell’s flagship, with 192GB of HBM3e and roughly 2.5x the FP8 compute of an H100. That memory capacity is the game-changer. You can fit a 405B model in a single B200 without tensor parallelism across nodes, which kills a huge amount of communication overhead. But you pay for that—both in raw dollars per hour and in the scarcity of supply.

If you’re doing pre-training or massive batch inference, B200 is the obvious choice. If you’re doing typical fine-tuning or serving smaller models, H100 is often more cost-effective.

How Rental Pricing Is Structured Per Hour

Cloud GPU pricing is almost always per GPU-hour, but the fine print matters. H100 instances typically run $2–4 per GPU-hour on demand, while B200 instances can be $6–10 or more. But you’re not just paying for the GPU—you’re paying for the host, RAM, and often a minimum commitment.

Most providers have three pricing models: on-demand (hourly, no commitment), reserved (discounted for 1–3 month commits), and spot (bid-based, can be interrupted). For H100, spot can be 50–70% cheaper, but you need fault-tolerant checkpoints. For B200, spot is rarer because demand is high.

Don’t compare sticker prices. Compare effective cost per hour after you factor in setup time, minimum billing increments (some charge per 15 minutes, others per hour), and whether you pay for idle time during cold starts.

Memory Bandwidth vs Capacity for LLM Work

For LLM inference, memory bandwidth is often the bottleneck, not compute. The H100 has 3.35 TB/s of bandwidth; the B200 has 8 TB/s. That’s a 2.4x improvement, which directly translates to faster token generation for batch workloads. If you’re serving a model that fits in H100 memory, the B200 will generate tokens 2–3x faster—but you’re paying 2–3x more per hour, so the cost per token might be similar.

Memory capacity is a different story. The B200’s 192GB lets you load a 70B model in FP16 without sharding, or a 405B in FP8 on a single GPU. That eliminates inter-node communication, which is often the real latency killer. With H100, you’d need 2–3 GPUs for a 70B in FP16, and the PCIe/NVLink overhead adds 10–20% to inference latency.

My rule of thumb: if your model fits in 80GB, H100 is the better deal. If it needs more than 80GB, B200 is worth considering because you avoid multi-GPU complexity.

When B200 Is Worth the Premium

B200 is worth it when you’re training a model that doesn’t fit in 80GB, or when you’re doing inference on a model that needs >80GB and you want low latency. For example, we fine-tuned a 70B model with a long context (128k tokens) using LoRA on an H100, and it worked—but we had to use gradient checkpointing and offloading, which slowed us down. On a B200, we could fit the full model and activations in memory, cutting training time by 40%.

Another case: serving a 70B model with a large batch size. On H100, you might need 2 GPUs with tensor parallelism, which adds complexity and network latency. On B200, you can do it on one GPU, and the 8 TB/s bandwidth means you can push higher throughput per GPU. If your workload is latency-sensitive, the B200’s single-GPU advantage is huge.

But only if you’re actually using the memory. If you’re running a 7B model, B200 is overkill—you’re paying for memory you won’t touch.

When H100 Is the Smarter Buy

H100 is the smarter buy for most teams, honestly. If you’re doing standard fine-tuning (LoRA, QLoRA) on models up to 70B, or serving models with a batch size that fits in 80GB, the H100 gives you the best cost-per-token. The rental market is mature, so you can find spot instances, reserved pricing, and multiple providers competing.

Also, H100 is more compatible with existing software. Most ML frameworks, inference engines (vLLM, TensorRT-LLM), and Kubernetes setups are battle-tested on Hopper. Blackwell requires newer CUDA versions and sometimes driver updates, which can cause headaches if you’re using custom kernels or older libraries.

For pre-training a small model (under 10B params), H100 is often cheaper per unit of compute when you factor in the ability to scale out across many GPUs. The H100’s NVLink bandwidth (900 GB/s) is sufficient for most data-parallel training, and you can get more H100s for the same budget as fewer B200s.

How to Verify a Provider's Real Effective Rate

Sticker price per hour is meaningless. Here’s what I actually check before renting:

1. Minimum billing increment. Some providers bill per second, others per hour. If you run a 10-minute job, per-hour billing wastes 50 minutes. I calculate effective cost per job, not per hour.

2. Setup and cold-start time. How long from clicking “launch” to getting a shell? If it’s 5 minutes, that’s fine. If it’s 30 minutes, that’s 30 minutes of paid time you’re not using.

3. Idle termination. Do they charge for idle time? Some providers keep billing even if your GPU is idle because of a script hang. I set up auto-termination policies, but I also check if the provider has a grace period.

4. Actual throughput, not theoretical TFLOPS. Run a quick benchmark—like a 1k-token generation with a fixed batch size—and measure tokens/sec. Divide that by the hourly cost to get cost per million tokens. That’s the number that matters.

5. Hidden costs. Egress fees, storage fees, and per-node charges can add 20–30%. Read the pricing page line by line.

I always test with a 1-hour job before committing to a 100-hour reservation. It’s worth the $5–10 to avoid a bad deal.

FAQ

Q: Can I run a 70B model on an H100?
A: Yes, if you use quantization. A 70B model in FP16 needs ~140GB, so you’d need 2 H100s. But with INT8 or FP8 quantization, it fits in 80GB. You’ll lose some accuracy, but for many tasks it’s fine. If you need full precision, go with B200.

Q: Is B200 always faster for inference?
A: Not always. For small models that fit in H100 memory, B200’s higher bandwidth gives you ~2x throughput, but you’re paying 2-3x more per hourcars, so cost per token is often similar. The real win is when you avoid multi-GPU overhead.

Q: How do I estimate cost per million tokens?
A: Run a benchmark with your model and a realistic batch size. Measure tokens per second. Then divide the GPU hourly cost by (tokens/sec * 3600) to get cost per token. Multiply by 1 million. That’s your cost per million tokens—compare that across providers, not the hourly rate.

Q: Should I rent spot instances for training?
A: Only if you have checkpointing that saves every few minutes. Spot H100s are cheap but can be reclaimed with 2-minute warnings. For fine-tuning, it’s risky but doable. For pre-training, avoid spot—you’ll lose too much progress.

Q: What about memory bandwidth vs capacity for long-context LLMs?
A: Long context (128k+ tokens) is memory-hungry. B200’s 192GB lets you keep the full KV cache in memory, avoiding recomputation. H100 can handle it but with smaller batch sizes or offloading, which slows things down. If long context is your workload, B200 is worth the premium.

Q: Are there hidden costs with cloud GPUs?
A: Yes—egress bandwidth, storage, and per-node fees. A provider might advertise $2/hour, but after you add 20GB of model weights and log files, you could pay an extra $0.10/GB for storage. Always check the pricing page for “additional resources.”