"A100 vs H100: the on-demand vs spot price gap is the real story"

A100 vs H100: the on-demand vs spot price gap is the real story

Every GPU rental page tells you the same headline: "H100 is roughly 3x an A100." True, and useless. The number that actually decides your bill is the spread between on-demand and spot, and the spread between the same chip in SXM vs PCIe form. Both are wider than you think, and both are where a lazy comparison trips you up.

The on-demand prices, pulled off Vast.ai tonight

I grabbed the live index because "list price" and "what the market will actually give you" drift apart fast on these rental marketplaces. As of 18 October 2026, on-demand per GPU-hour:

• A100 SXM — $1.04/hr
• A100 PCIe — $0.27/hr
• H100 SXM — $2.87/hr
• H200 SXM — $6.62/hr

That's the headline gap: H100 SXM at $2.87 is about 2.8x the A100 SXM at $1.04. Nothing you didn't expect.

But here's the bit the spec sheets skip. An A100 PCIe at $0.27/hr is a quarter of the SXM price. Same chip name, same HBM2e memory class, totally different market. If your job is single-GPU inference and you don't need the NVLink mesh of an SXM board, buying the PCIe slot is like finding the same meal for a quarter of the price at a different table.

Spot flips the ranking

This is where my earlier assumptions broke. I used to treat spot as "basically on-demand with a discount you can't rely on." On Vast.ai the gap is not a gentle 10-15% nudge — spot H100s frequently clear 40% under on-demand, and the discount concentrates on the SXM parts nobody wants to commit to long-term.

The honest caveat: spot is spot. A $0.87/hr H100 SXM is real, but it can be reclaimed mid-run. For a batch job you checkpoint anyway, that's a steal. For anything that needs to stay up at 2am Pacific, it's a gamble dressed as a discount. The market prices that risk in, and you should too.

When the cheaper card wins

Run the numbers the way you'd actually use them, not the way a spec table does.

• Fine-tuning a small model overnight: A100 PCIe spot. If it dies at 3am, you resume a checkpoint. $0.27/hr is a rounding error against your engineering time.
• Serving inference with a latency SLA: H100 SXM on-demand. The spot discount is meaningless when a reclaim means a 500ms p99 spike during a demo.
• Batch eval across 10k samples: A100 SXM. You want the NVLink pair for the memory bandwidth, but you don't need Hopper's FP8 for this workload. Paying 2.8x for H100 buys nothing here.

The trap is reaching for H100 because it's "the right GPU" when your actual workload is memory-bound, not FP8-bound. A100 SXM still has 80GB and a wide bus. Hopper's advantage shows up in the training loop, not in a batch that's choking on bandwidth anyway.

What actually moved my budget

The PCIe-vs-SXM spread was the number that changed my decision, not the A100-vs-H100 one. I went in expecting to argue about Hopper vs Ampere, and came out arguing about form factor. If you're pricing a fleet, spend the ten minutes to separate PCIe from SXM in your model — a $0.77/hr gap between two "A100s" is exactly the kind of thing a headline comparison buries.

The market does one more thing that's worth respecting: it prices commitment. Vast.ai's spot market basically says the SXM boards are abundant and renters are skittish. That's information about supply, and it's usually more reliable than any vendor's "availability" claim.