GPT-2 124M on 10B tokens, 8× H100 (same recipe, rented for 90 min)

lesson 1.9 · Scale

The same GPT-2 run as on the home A5000, on a rented 8× H100 machine: how much faster, and is the result the same?

Run A

done

fineweb_edu_10b · 12L·768d · 124M params · bf16

tokens read

9,999,745,024

step

19,073 / 19,073

train loss: 3.014
best val: 3.055
tokens/s: 3.5M
MFU: 37.7%
peak mem: 21.6 GB
lr: 6.0e-5

0:48:14 · 5.0 kWh · ≈ $42 rented at $52.45/h

Losslower is better · starts near ln(vocab)

The machine, live

8× NVIDIA H100 80GB HBM3 · gcp-h100x8offline · just now

GPU busy

– %

VRAM

– / 637 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

5.03 kWh

≈ $0.75 of electricity

RAM

–

of 1842.6 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

GPU memory (measured)peak per log window · dashed = activations + workspace

Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right

The questions

arithmetic	A 0.72
ice	A 0.19
ocean	A 0.80
orbit	A 0.99
paris	A 0.99
planet	A 0.89
sky	A 0.73
water	A 0.69
whale	A 0.92

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

step 19,073

Run A

Water is the most important part of the water cycle. It provides oxygen to the atmosphere and protects it from the harmful effects of pollution.
Every day, over 1.5 billion people around the world lose part or all of their drinking water. This is a problem that affects all of us, but we all need to be conscious of the water conservation tips.
The first and most obvious tip is to follow the water conservation tips that you can follow. This way, you will be able to save water.
When we drink, the water that we drink is filtered again and the water will be free from contaminants. But what about when we are not drinking enough water?
Some of the things that can go wrong with our drinking water can not only lead to water quality problems but also it’s a potential health hazard.
The first and most obvious thing that can go wrong with our drinking water is the water itself. It is not safe to drink water that we are not properly using. It

What happened

08:38:51 AMAFinished 19073 steps. Best val loss 3.055.
08:38:50 AMACheckpoint at step 19073: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.35s.
08:38:43 AMAStep 19073: val 3.073, train 3.041 (gap +0.032).
08:38:31 AMACheckpoint at step 19000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.25s.
08:38:24 AMAStep 19000: val 3.057, train 3.032 (gap +0.025).
08:37:45 AMAStep 18750: val 3.063, train 3.018 (gap +0.046).
08:37:06 AMAStep 18500: val 3.066, train 3.027 (gap +0.038).
08:36:27 AMAStep 18250: val 3.056, train 3.023 (gap +0.033).
08:35:48 AMACheckpoint at step 18000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.26s.
08:35:41 AMAStep 18000: val 3.058, train 3.042 (gap +0.016).
08:35:02 AMAStep 17750: val 3.086, train 3.041 (gap +0.045).
08:34:23 AMAStep 17500: val 3.059, train 3.027 (gap +0.032).
08:33:43 AMAStep 17250: val 3.084, train 3.034 (gap +0.050).
08:33:04 AMACheckpoint at step 17000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.25s.
08:32:57 AMAStep 17000: val 3.058, train 3.026 (gap +0.032).
08:32:18 AMAStep 16750: val 3.076, train 3.040 (gap +0.037).
08:31:39 AMAStep 16500: val 3.076, train 3.051 (gap +0.025).
08:31:00 AMAStep 16250: val 3.055, train 3.047 (gap +0.009).
08:30:21 AMACheckpoint at step 16000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.25s.
08:30:14 AMAStep 16000: val 3.075, train 3.074 (gap +0.002).
08:29:35 AMAStep 15750: val 3.076, train 3.055 (gap +0.021).
08:28:56 AMAStep 15500: val 3.065, train 3.038 (gap +0.028).
08:28:17 AMAStep 15250: val 3.094, train 3.048 (gap +0.046).
08:27:38 AMACheckpoint at step 15000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.24s.
08:27:31 AMAStep 15000: val 3.088, train 3.069 (gap +0.020).
08:26:52 AMAStep 14750: val 3.093, train 3.070 (gap +0.023).
08:26:13 AMAStep 14500: val 3.090, train 3.050 (gap +0.039).
08:25:34 AMAStep 14250: val 3.083, train 3.063 (gap +0.021).
08:24:55 AMACheckpoint at step 14000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.26s.
08:24:49 AMAStep 14000: val 3.115, train 3.061 (gap +0.054).
08:24:10 AMAStep 13750: val 3.118, train 3.081 (gap +0.037).
08:23:31 AMAStep 13500: val 3.126, train 3.045 (gap +0.080).
08:22:52 AMAStep 13250: val 3.085, train 3.093 (gap -0.008).
08:22:12 AMACheckpoint at step 13000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.25s.
08:22:06 AMAStep 13000: val 3.122, train 3.109 (gap +0.013).
08:21:27 AMAStep 12750: val 3.124, train 3.119 (gap +0.005).
08:20:48 AMAStep 12500: val 3.148, train 3.087 (gap +0.062).
08:20:09 AMAStep 12250: val 3.121, train 3.102 (gap +0.019).
08:19:29 AMACheckpoint at step 12000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.26s.
08:19:22 AMAStep 12000: val 3.143, train 3.101 (gap +0.042).
08:18:43 AMAStep 11750: val 3.148, train 3.139 (gap +0.010).
08:18:04 AMAStep 11500: val 3.150, train 3.142 (gap +0.008).
08:17:25 AMAStep 11250: val 3.148, train 3.119 (gap +0.029).
08:16:46 AMACheckpoint at step 11000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.25s.
08:16:40 AMAStep 11000: val 3.145, train 3.126 (gap +0.019).
08:16:01 AMAStep 10750: val 3.152, train 3.114 (gap +0.038).
08:15:22 AMAStep 10500: val 3.144, train 3.117 (gap +0.027).
08:14:43 AMAStep 10250: val 3.174, train 3.125 (gap +0.049).
08:14:04 AMACheckpoint at step 10000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.25s.
08:13:57 AMAStep 10000: val 3.164, train 3.149 (gap +0.014).
08:13:18 AMAStep 9750: val 3.190, train 3.151 (gap +0.040).
08:12:39 AMAStep 9500: val 3.152, train 3.141 (gap +0.011).
08:12:00 AMAStep 9250: val 3.201, train 3.147 (gap +0.055).
08:11:21 AMACheckpoint at step 9000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.26s.
08:11:14 AMAStep 9000: val 3.190, train 3.173 (gap +0.017).
08:10:35 AMAStep 8750: val 3.189, train 3.171 (gap +0.018).
08:09:55 AMAStep 8500: val 3.195, train 3.180 (gap +0.016).
08:09:16 AMAStep 8250: val 3.198, train 3.182 (gap +0.016).
08:08:37 AMACheckpoint at step 8000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 4.90s.
08:08:31 AMAStep 8000: val 3.204, train 3.170 (gap +0.034).
08:07:52 AMAStep 7750: val 3.221, train 3.191 (gap +0.030).
08:07:13 AMAStep 7500: val 3.231, train 3.199 (gap +0.032).
08:07:00 AMAStep 7410 / 19073 (39%), 3.88B tokens. Train loss 3.201, best val 3.237, fact probes 0.76. 3482229 tok/s; about 00d 00h 25m left.
08:06:34 AMAStep 7250: val 3.241, train 3.213 (gap +0.028).
08:05:55 AMACheckpoint at step 7000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 4.78s.
08:05:49 AMAStep 7000: val 3.237, train 3.239 (gap -0.001).
08:05:10 AMAStep 6750: val 3.242, train 3.224 (gap +0.018).
08:04:31 AMAStep 6500: val 3.256, train 3.218 (gap +0.038).
08:03:52 AMAStep 6250: val 3.273, train 3.209 (gap +0.064).
08:03:13 AMACheckpoint at step 6000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.26s.
08:03:06 AMAStep 6000: val 3.270, train 3.252 (gap +0.018).
08:02:27 AMAStep 5750: val 3.286, train 3.268 (gap +0.019).
08:01:48 AMAStep 5500: val 3.268, train 3.279 (gap -0.011).
08:01:09 AMAStep 5250: val 3.304, train 3.263 (gap +0.040).
08:00:30 AMACheckpoint at step 5000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.25s.
08:00:23 AMAStep 5000: val 3.309, train 3.312 (gap -0.003).
07:59:44 AMAStep 4750: val 3.338, train 3.303 (gap +0.035).
07:59:04 AMAStep 4500: val 3.319, train 3.300 (gap +0.019).
07:58:25 AMAStep 4250: val 3.343, train 3.343 (gap -0.000).
07:57:46 AMACheckpoint at step 4000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.29s.
07:57:39 AMAStep 4000: val 3.371, train 3.368 (gap +0.003).
07:57:00 AMAStep 3750: val 3.384, train 3.380 (gap +0.005).
07:56:21 AMAStep 3500: val 3.415, train 3.395 (gap +0.020).
07:55:42 AMAStep 3250: val 3.454, train 3.392 (gap +0.062).
07:55:02 AMACheckpoint at step 3000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.26s.
07:54:54 AMAStep 3000: val 3.448, train 3.429 (gap +0.018).
07:54:15 AMAStep 2750: val 3.454, train 3.473 (gap -0.019).
07:53:36 AMAStep 2500: val 3.520, train 3.473 (gap +0.047).
07:52:58 AMACheckpoint step 2000 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0002000.pt in 40.8s (in the background; training didn't wait).
07:52:57 AMAStep 2250: val 3.578, train 3.516 (gap +0.062).
07:52:17 AMACheckpoint at step 2000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 5.27s.
07:52:04 AMAStep 2000: val 3.589, train 3.539 (gap +0.049).
07:51:25 AMAStep 1750: val 3.656, train 3.631 (gap +0.025).
07:50:36 AMAResumed at step 1576 from a checkpoint.
07:50:16 AMAResuming run A from its checkpoint at step 1576.
07:35:48 AMACheckpoint step 1576 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0001576.pt in 15.6s (in the background; training didn't wait).
07:35:33 AMAStopped at step 1576; a checkpoint was saved, so it can resume.
07:35:33 AMACheckpoint at step 1576: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 4.70s.
07:34:47 AMAStep 1500: val 3.750, train 3.728 (gap +0.022).
07:33:53 AMAResumed at step 1380 from a checkpoint.
07:33:38 AMAResuming run A from its checkpoint at step 1380.
07:23:19 AMAStopped at step 1380; a checkpoint was saved, so it can resume.
07:23:19 AMACheckpoint at step 1380: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 4.92s.
07:21:35 AMAStep 1250: val 3.896, train 3.839 (gap +0.056).
07:18:25 AMACheckpoint at step 1000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 4.89s.
07:18:12 AMAStep 1000: val 4.060, train 4.021 (gap +0.039).
07:15:11 AMAStep 750: val 4.429, train 4.400 (gap +0.029).
07:12:31 AMAStep 500: val 5.203, train 5.161 (gap +0.043).
07:10:14 AMACheckpoint at step 250: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 4.89s.
07:10:09 AMAStep 250: val 6.082, train 6.032 (gap +0.051).
07:08:39 AMACheckpoint at step 100: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 4.88s.
07:08:14 AMACheckpoint at step 50: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 4.90s.
07:07:48 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0000000.pt in 6.3s (in the background; training didn't wait).
07:07:41 AMACheckpoint at step 0: 498.0 MB (weights, AdamW state, schedule position, RNG states) in 1.75s.
07:07:39 AMAData parallel on 8 GPUs (NVIDIA H100 80GB HBM3): each runs 2 of the 16 micro-batches per step.
07:07:39 AMAExpected loss before training ≈ ln(vocab) = 10.83.
07:07:39 AMATraining 124.48M params on fineweb_edu_10b for 19073 steps (bf16, batch 32×16, ctx 1024).