What does it really cost to train a 70B parameter model?

"How much does it cost to train a 70B model?" gets a different answer every time someone asks. The headline numbers from papers usually report GPU-hours and leave you to fill in the rest. Here's the rest.

The five cost buckets

• Compute: GPU-hours × hourly rate.
• Power and cooling: rolled into colo/cloud pricing, but worth surfacing.
• Storage and dataset prep: deduplication, tokenization, parallel-filesystem capacity.
• Network fabric: InfiniBand for multi-node — non-trivial line item.
• Engineering: weeks of DevOps + ML eng time. The line everyone forgets.

Assumptions for this estimate

• 70B parameter dense transformer, Llama-3-style architecture.
• 2T tokens of training data (Chinchilla-optimal: ~20× params).
• BF16 + FP8 mixed precision via Transformer Engine.
• 64× H100 SXM5 in an 8-node InfiniBand cluster, 90% MFU.

1. Compute

Total training FLOPs ≈ 6 × N × D where N is parameters and D is tokens. For 70B × 2T tokens that's roughly 8.4 × 10²³ FLOPs.

At 90% MFU on H100 FP8 (≈3,560 TFLOPS sustained per GPU), one GPU does 3.1 × 10²⁰ FLOPs/day. The job needs ~2,710 GPU-days, or about 42 wall-clock days on 64 H100s.

At our 12-month rental list price of ~$3.20/GPU-hour for H100 SXM5 in HGX, that's $208K in compute alone. Month-to-month pricing pushes this to ~$260K. Hyperscaler on-demand H100 is often $4–6/hr, so the same job runs $260K–$390K.

2. Power and cooling

Already included in colo rental, but useful to know: 64× H100 SXM5 + 8 chassis + fabric draws ~80 kW continuous. Over 42 days that's 80,640 kWh. At commercial rates with PUE 1.3, you're looking at ~$13K of electricity that someone (us, or you, or AWS) is paying for.

3. Storage and dataset prep

• Raw + deduped + tokenized 2T-token corpus: ~6 TB tokenized, often 50 TB raw.
• Parallel filesystem (BeeGFS or Weka) tier: ~$1,500/mo for 100 TB usable for the duration of the run = ~$2,100.
• Tokenization compute on CPU: trivial, ~$500.
• Checkpoints (every 1B tokens × ~140GB per): ~280 TB of total checkpoint writes, retained ~30 TB rolling = ~$600.

4. Network fabric

NDR 400G InfiniBand fabric for 64 GPUs (8 rails × 8 nodes): switches, cables, NICs are amortized into the rental. The line item is usually included, but if you're building your own cluster, budget $300–400K in capex for the fabric alone — call it $30K amortized over a 12-month period for this one run.

5. Engineering

The line most cost models miss. A 70B pretraining run typically involves:

• 2 weeks of ML engineer time on data prep and tokenizer choice.
• 1 week of DevOps time on fabric tuning, NCCL config, checkpoint plumbing.
• 4–6 weeks of ML engineer time babysitting the run (loss spikes, restarts, eval).
• 2 weeks of post-training time (instruction tuning, RLHF or DPO, eval suite).

At blended $200K/year fully-loaded for senior ML eng, that's roughly $50K–$70K of human time. The number gets bigger if your team isn't already familiar with FSDP, Megatron, or multi-node debugging.

The total

• Compute (rented): $208K
• Storage + dataset infra: $3K
• Fabric amortization: $30K
• Engineering: $60K
• Total: ~$300K

The "GPU-hours × hourly rate" headline only captures two-thirds of the real cost. The compute number also assumes everything goes well — a botched run from a loss spike at 80% completion adds another $40K easily.

How to spend less

• Don't pretrain from scratch. Fine-tune Llama 3.1 70B or DeepSeek-V3 for <$5K. The base model has already eaten the $300K bill.
• Quantize aggressively. FP8 training, FP4 inference. The Transformer Engine handles the scaling.
• Commit longer for compute. 24-month H100 rentals price 25–30% below month-to-month — meaningful on a $200K compute line.
• Skip the fabric if you can. Plenty of 7B–13B fine-tuning runs fit on a single 8-GPU node. Multi-node is where costs explode.