H200 vs H100 for LLM inference - where the memory premium actually pays off

The spec difference everyone quotes

On paper, the H200 is an H100 with better memory: the same Hopper compute, the same 700W power envelope, but 141 GB of HBM3e at 4.8 TB/s instead of 80 GB of HBM3 at 3.35 TB/s. That is a 76 percent memory increase and a 43 percent bandwidth increase, which is why the marketing cycle frames the H200 as an easy upgrade. The reality is more conditional, and the condition has a name: your batch size.

When the premium pays off immediately

If you are serving a 70B parameter model with a long context window, the H100 forces you into uncomfortable territory. A 70B model in FP16 needs roughly 140 GB just for weights, which does not fit in 80 GB without quantization, tensor parallelism across two GPUs, or aggressive KV-cache offloading to CPU. The H200 fits those weights comfortably and still leaves room for a meaningful KV cache. For inference-only fleets serving large models, the H200's effective cost per token is often lower than the H100's despite the higher sticker price, because you get more tokens per GPU-hour and fewer communication overheads between sharded GPUs.

The same logic applies to models in the 30B to 70B range with 32K-plus context. Long context inflates the KV cache linearly, and the H200's extra memory absorbs it where an H100 would start spilling. If your workload's arithmetic intensity is dominated by memory traffic rather than FLOPs, you are paying for bandwidth, and the H200 sells bandwidth.

When it does not

For small models and modest context lengths, the H200 premium is hard to justify. A 7B or 13B model fits easily on an H100 with room for large batches, and at that point your bottleneck is compute throughput, not memory capacity. Paying 20 to 40 percent more per hour for memory you will never allocate is simply a worse unit price. The same holds for training-heavy fleets where model parallelism already spreads weights across many GPUs and the per-GPU memory ceiling matters less than interconnect topology.

There is also a supply story worth acknowledging. H200 availability on major cloud providers has historically been tighter than H100, and spot pricing does not always reflect the list premium. If your scheduler can fall back to H100 capacity during peak hours, a blended fleet is often the right answer.

A practical decision rule

Check three numbers before you commit. First, the smallest number of GPUs that can hold your model weights plus your target KV cache. If the H100 can hold your model with the batch size your latency budget requires, stay on H100. Second, your achieved tokens per second per dollar on each GPU, measured, not quoted. Third, your p99 latency under load, because memory headroom translates directly into less queueing. If the H200 lets you serve the same traffic with fewer GPUs at equal latency, the premium is real savings. If it only buys you headroom you do not use, it is a tax.

Bottom line

The H200 is not a generational upgrade, it is a specialized one. It wins decisively for large-model, long-context, inference-first fleets, and it loses politely for small models and compute-bound work. Price the memory, not the logo.