Bare metal vs cloud GPU: when colocation wins

The default assumption for GPU workloads in 2026 is still "spin it up on AWS." That's the wrong default for at least three common workload shapes. Here's the math, and the cases where bare-metal or colocation crushes hyperscaler rentals.

The hourly rate is a lie

AWS p5.48xlarge (8× H100 SXM5) lists at $98.32/hr on-demand. That's $861K/year for one node. Reserved 1-year is roughly $516K. Reserved 3-year is around $310K.

Our 12-month rental for the same 8× H100 SXM5 HGX node lands around $220K. Colocation of a customer-owned node runs $55K/year in rack + power + cooling + fabric. The capex on the node itself is roughly $280K — break-even vs AWS reserved 1-year happens in month 8.

Three shapes where bare-metal wins

1. Long-running training jobs

If your run takes >2 weeks of continuous GPU time, the hyperscaler hourly is a tax. Bare-metal rental at half the price gives you the same hardware, no preemption risk, and consistent NCCL topology across nodes.

2. Production LLM serving at scale

Serving an LLM 24/7 means your utilization is high by construction. AWS p5 reserved still costs ~$60/hr per node. The same node bare-metal rents for ~$25/hr equivalent. At 100% utilization, that's $300K/year saved per node — and most production fleets are 4–32 nodes.

3. Anything stateful or data-gravity-bound

If your training corpus is 200 TB, you don't want to pay AWS egress every time you move it. Colocating your storage next to your compute, with direct cross-connects to your existing cloud accounts, removes the egress tax entirely.

When cloud GPU still wins

• Bursty workloads. If you need 64 H100s for 48 hours, twice a quarter, hyperscaler on-demand is the right tool — even at $98/hr.
• Geographic latency. Inference fleets that need pops in 12 regions can't replicate hyperscaler footprint with one bare-metal facility.
• Pure experimentation. Spike a node up for 6 hours, kill it, never see it again. Don't sign a contract for that.

The hybrid model most teams land on

Steady-state training and production serving on bare-metal or colo, with hyperscaler GPU as the overflow tier. Direct cross-connects keep the control plane in your existing AWS/GCP/Azure account so IAM, S3, BigQuery, etc. all still work.

This is the model we deploy most often — see our GPU colocation page for the on-ramp options, or bare-metal GPU rentals if you don't want to own the hardware.

The break-even math, simplified

• Your workload runs at >40% sustained GPU utilization → bare-metal wins.
• You need the same hardware for >6 months → colocation wins on TCO.
• You move >50 TB of data per month → egress alone justifies the move.
• You need bursty capacity → stay on hyperscaler for that slice.