Guides
Cloud GPU Price Comparison: What I Actually Pay
Why Cloud GPU Price Comparison Feels So Messy
I've been renting GPUs for training runs since the A100 days, and I've probably burned through $40k in cloud compute across Lambda, RunPod, Vast, and even a sketchy dedicated box in Finland. If you're trying to decide between H200 and H100 in 2025, the hourly rates alone won't tell you what you need to know. The real cost is hidden in memory bandwidth, spot availability, and how long your job actually runs.
This is a guide to cloud GPU price comparison written by someone who does this weekly. I'll walk through the actual decision process I use, not the marketing numbers. You'll learn how to compare cloud GPU pricing per hour without getting burned by idle time or hidden egress fees.
The Three Numbers That Matter More Than the Sticker Price
When I look at a cloud GPU rental price list, I ignore the headline rate for the first five minutes. Instead, I check three things: memory bandwidth, VRAM capacity, and how the provider handles spot instances. These determine whether you're getting a deal or paying for a paperweight.
Memory Bandwidth: The LLM Inference Bottleneck
For LLM inference, the cheapest cloud GPU rental for LLM inference isn't always the lowest per-hour rate. A H100 with 3.35 TB/s of HBM3 bandwidth will decode tokens noticeably faster than an A100 with 2 TB/s, even if both have 80GB of VRAM. If you're serving a 70B model, that bandwidth difference can mean 30% more requests per second. I'd rather pay $3.50/hour for an H100 than $2.20/hour for an A100 if I'm running production traffic.
VRAM: The Hard Ceiling for Fine-Tuning
For fine-tuning, VRAM is non-negotiable. You can't shard a 7B model into 24GB and expect full fine-tuning to work without gradient checkpointing and offloading. When I look at an affordable cloud GPU for fine-tuning, I check whether the card has 48GB or 80GB. A 48GB L40S might be cheap per hour, but if it forces me to use LoRA instead of full fine-tuning, my iteration time doubles. That's a hidden cost that never shows up in a cloud GPU pricing calculator.
Comparing Cloud GPU Pricing Per Hour: On-Demand vs Spot
The biggest lever in any cloud GPU pricing comparison 2025 is the spot vs on-demand spread. On-demand rates are predictable but often 3-4x higher than spot. Spot instances can be 60-80% cheaper, but they can be yanked with 30 seconds notice. For training jobs with checkpointing, spot is usually worth the risk. For production inference, it's a hard no.
Let's break down what I actually see across providers. I'm not quoting exact prices because they change weekly and vary by region. But the relative gaps hold:
| GPU | On-Demand (relative) | Spot (relative) | Best Use Case |
|---|---|---|---|
| A100 80GB | Baseline | ~50% cheaper | Fine-tuning, moderate inference |
| H100 80GB | ~1.6x A100 | ~40% cheaper | Training, high-throughput inference |
| H200 141GB | ~2x A100 | ~35% cheaper | Large context inference, big batch training |
| L40S 48GB | ~0.7x A100 | ~50% cheaper | Small fine-tunes, dev environments |
Notice how the H100 vs A100 cloud GPU cost per hour gap is not 2x. It's usually closer to 1.5-1.8x. But the H100 finishes training jobs in less wall-clock time because of the extra compute and memory bandwidth. So the total cost per epoch can be lower on H100 even when the hourly rate is higher. I've seen this happen with my own fine-tuning jobs on 7B and 13B models.
How to Build Your Own Cloud GPU Pricing Calculator
Don't rely on the provider's calculator. It doesn't know your workload. Instead, write a simple spreadsheet with four columns: hourly rate, estimated job duration, number of GPUs, and spot preemption probability. Then add a 10% overhead for setup, debugging, and data transfer.
Here's a concrete example. I needed to fine-tune a 13B model on 10M tokens. On A100s, that took about 12 hours on 8 GPUs. On H100s, it took 8 hours on 8 GPUs. If A100 is $2.50/hour and H100 is $4.00/hour, the math is:
- A100: 12 hours × 8 GPUs × $2.50 = $240
- H100: 8 hours × 8 GPUs × $4.00 = $256
Only a 7% difference. But if the H100 spot rate drops to $2.20/hour, the H100 becomes cheaper. That's the kind of trade-off you only see when you do the math yourself. I keep a running Google Sheet with these calculations for every project.
The Best Value Cloud GPU for Training: It Depends on Your Checkpointing
People ask me what the best value cloud GPU for training is, and I always say: the one that finishes your job before the spot instance gets killed. If you're training a small model that fits on a single GPUaine, a 24GB RTX 4090 at $0.50/hour might be the best value. But if you need to scale to 8 GPUs, you're better off with A100s or H100s that have NVLink and better interconnect.
I recently moved a fine-tuning job from 4×A100s to 4×H100s because the spot price on H100 dropped to within 15% of A100. The job finished in 6 hours instead of 9. Even with the higher hourly rate, I saved money because I wasn't paying for idle time while waiting for spot instances to come back after preemption. That's the kind of thing a cloud GPU rental price list won't tell you.
Hidden Costs That Break a Cloud GPU Rental Price List
Every provider has hidden costs. The biggest one is data egress. I once ran a training job on a provider that charged $0.09/GB for egress. The dataset was 2TB, and I had to pull it down twice because I forgot to cache the tokenized version. That added $360 to a job that should have cost $200. Always check egress fees before you commit.
Another hidden cost is idle time. If you're using on-demand GPUs, you're paying for every second the GPU is allocated, even if your code is stuck downloading a model from Hugging Face. I now always pre-download weights and datasets to a persistent volume before starting the job. It's a boring tip, but it's saved me more money than any price comparison.
When to Use a Cloud GPU Pricing Comparison Tool vs Manual Checking
I use a cloud GPU pricing comparison tool for a quick scan, but I never book based on it alone. Those tools aggregate rates from providers, but they don't know about regional availability or spot volatility. I'll check Vast, RunPod, and Lambda directly to see real-time prices for the specific GPU I need. Sometimes a provider that's not on the comparison site has a flash sale on H100s for 30% less.
For a cloud GPU rental price list, I keep a bookmark folder with the direct pricing pages for my top five providers. It takes 10 minutes to scan them all. That's faster than waiting for a comparison site to refresh, and I get the actual rates, not an average.
Conclusion: The Cheapest GPU Is the One That Finishes
Cloud GPU price comparison isn't about finding the lowest number on a pricing page. It's about calculating the total cost of your specific workload, including time, spot risk, and hidden fees. I've seen engineers waste hours trying to save $0.30/hour on a GPU that runs 2x slower. That's not engineering; that's penny-wise and pound-foolish.
My rule of thumb: estimate your job's duration on each GPU, multiply by the hourly rate, add 15% for overhead, and then decide. If you're doing LLM inference, weigh memory bandwidth heavily. If you're training, weigh spot availability and checkpointing. That's how you actually compare cloud GPU pricing per hour without getting fooled by the sticker price.