GPT-2 124M on 10B tokens (Karpathy recipe)

lesson 1.9 · Scale

What does a real GPT-2-sized pretraining run look like, step by step?

Run A

done

fineweb_edu_10b · 12L·768d · 124M params · bf16

tokens read

9,999,745,024

step

19,073 / 19,073

train loss: 3.031
best val: 3.030
tokens/s: 50k
MFU: 38.6%
peak mem: 7.0 GB
lr: 6.0e-5

2d 7h · 12.5 kWh ≈ $1.88 power · ≈ $17 to rent

Losslower is better · starts near ln(vocab)

The machine, live

NVIDIA RTX A5000 · red-gpu-wslupdated just now

GPU busy

39 %

waiting

VRAM

5.2 / 24 GB

room left

Power

68 / 230 W

idle

Temperature

50 °C

cool

SM clock

1020 MHz

steady

Memory bus

10 %

quiet

Fan

30 %

quiet

CPU

0 %

light

PCIe

to GPU 114 MB/sfrom GPU 21 MB/s

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read 0 B/swrite 0 B/s

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in 3.9 KB/sout 3.6 KB/s

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

68 W

≈ $0.01 an hour

last 5 minutes

5.6 Wh

from the card's energy counter

this experiment

12.50 kWh

≈ $1.88 of electricity

RAM

4.3 GB

of 62.7 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

Checkpoint reels

At every checkpoint, the same question to the previous checkpoint and the new one. Same seed, so only the model changed.

step 18,500 → 19,000
step 18,000 → 18,500
step 17,500 → 18,000
step 17,000 → 17,500
step 16,500 → 17,000
step 16,000 → 16,500
step 15,500 → 16,000
step 15,000 → 15,500
step 14,500 → 15,000
step 14,000 → 14,500
step 13,500 → 14,000
step 13,000 → 13,500
step 12,500 → 13,000
step 12,000 → 12,500
step 11,500 → 12,000
step 11,000 → 11,500
step 10,500 → 11,000
step 10,000 → 10,500
step 9,500 → 10,000
step 9,000 → 9,500
step 8,500 → 9,000
step 8,000 → 8,500
step 7,500 → 8,000
step 7,000 → 7,500
step 6,500 → 7,000
step 6,000 → 6,500
step 5,500 → 6,000
step 5,000 → 5,500
step 4,500 → 5,000
step 4,000 → 4,500
step 3,500 → 4,000
step 3,000 → 3,500
step 2,500 → 3,000
step 2,000 → 2,500
step 1,500 → 2,000
step 1,000 → 1,500
step 500 → 1,000
step 0 → 500

What's happening

What to watch · A single ~2.3-day run. Every 500 steps (~262M tokens) there's a checkpoint to try in the Playground. Watch the fact probes: at what point does it learn that water is a liquid?

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

GPU memory (measured)peak per log window · dashed = activations + workspace

Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right

The questions

arithmetic	A 0.69
ice	A 0.17
ocean	A 0.47
orbit	A 0.99
paris	A 1.00
planet	A 0.97
sky	A 0.86
water	A 0.87
whale	A 0.92

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

step 19,073

Run A

Water is the most important part of the water cycle. It provides the water for the ecosystem to grow and function, and the water is then returned to the land and rivers by the streams and rivers that carry the water.
How can we conserve water and use less water?
Use Conservation Measures to conserve water.
- Use less water. Use less water.
- Use less water. Use less water.
- Use less electricity. Use less electricity.
- Reduce the use of water.
- Reduce the use of water.
- Use more water.
- Reduce the amount of water used. Use less water.
For more information on water conservation and what you can do to conserve it, visit the Water Cycle Website.<|endoftext|>A new study published in The Lancet on Dec. 21 found that a new generation of “low calorie” foods may help people in the fight against type 2 diabetes. The researchers found that dieting that involves the consumption of “low-fat

What happened

11:36:14 AMAFinished 19073 steps. Best val loss 3.030.
11:36:14 AMACheckpoint at step 19073: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.77s.
11:36:09 AMAStep 19073: val 3.085, train 3.003 (gap +0.081).
11:24:58 AMACheckpoint reel: step 18,500 vs 19,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0019000.mp4
11:23:18 AMACheckpoint at step 19000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.86s.
11:23:13 AMAStep 19000: val 3.050, train 2.998 (gap +0.052).
10:39:42 AMAStep 18750: val 3.075, train 3.043 (gap +0.032).
10:07:00 AMAStep 18560 / 19073 (97%), 9.73B tokens. Train loss 3.008, best val 3.030, fact probes 0.76. 50207 tok/s; about 00d 01h 29m left.
09:57:07 AMACheckpoint reel: step 18,000 vs 18,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0018500.mp4
09:56:07 AMACheckpoint at step 18500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.76s.
09:56:03 AMAStep 18500: val 3.046, train 3.028 (gap +0.018).
09:12:19 AMAStep 18250: val 3.062, train 3.027 (gap +0.035).
08:30:03 AMACheckpoint reel: step 17,500 vs 18,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0018000.mp4
08:28:33 AMACheckpoint at step 18000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 3.12s.
08:28:27 AMAStep 18000: val 3.044, train 3.068 (gap -0.023).
08:07:00 AMAStep 17870 / 19073 (94%), 9.37B tokens. Train loss 3.036, best val 3.030, fact probes 0.76. 50298 tok/s; about 00d 03h 30m left.
07:44:58 AMAStep 17750: val 3.118, train 3.046 (gap +0.072).
07:03:11 AMACheckpoint reel: step 17,000 vs 17,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0017500.mp4
07:01:25 AMACheckpoint at step 17500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.85s.
07:01:20 AMAStep 17500: val 3.083, train 3.039 (gap +0.043).
06:17:48 AMAStep 17250: val 3.035, train 3.007 (gap +0.029).
06:07:00 AMAStep 17180 / 19073 (90%), 9.01B tokens. Train loss 3.046, best val 3.030, fact probes 0.78. 49986 tok/s; about 00d 05h 31m left.
05:35:13 AMACheckpoint reel: step 16,500 vs 17,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0017000.mp4
05:34:03 AMACheckpoint at step 17000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.82s.
05:33:58 AMAStep 17000: val 3.072, train 3.030 (gap +0.042).
04:50:26 AMAStep 16750: val 3.041, train 2.984 (gap +0.056).
04:08:09 AMACheckpoint reel: step 16,000 vs 16,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0016500.mp4
04:07:00 AMAStep 16500 / 19073 (87%), 8.65B tokens. Train loss 3.007, best val 3.030, fact probes 0.75. 50201 tok/s; about 00d 07h 30m left.
04:06:49 AMACheckpoint at step 16500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.87s.
04:06:44 AMAStep 16500: val 3.059, train 3.085 (gap -0.027).
03:23:08 AMAStep 16250: val 3.044, train 3.064 (gap -0.020).
02:41:20 AMACheckpoint reel: step 15,500 vs 16,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0016000.mp4
02:39:31 AMACheckpoint at step 16000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.76s.
02:39:26 AMAStep 16000: val 3.108, train 3.072 (gap +0.036).
02:07:00 AMAStep 15810 / 19073 (83%), 8.29B tokens. Train loss 3.036, best val 3.030, fact probes 0.75. 50221 tok/s; about 00d 09h 31m left.
01:55:53 AMAStep 15750: val 3.064, train 3.036 (gap +0.028).
01:13:16 AMACheckpoint reel: step 15,000 vs 15,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0015500.mp4
01:12:18 AMACheckpoint at step 15500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.72s.
01:12:13 AMAStep 15500: val 3.077, train 3.024 (gap +0.053).
12:28:39 AMAStep 15250: val 3.030, train 3.061 (gap -0.031).
12:07:00 AMAStep 15120 / 19073 (79%), 7.93B tokens. Train loss 3.060, best val 3.079, fact probes 0.74. 50153 tok/s; about 00d 11h 32m left.
11:46:09 PMACheckpoint reel: step 14,500 vs 15,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0015000.mp4
11:44:59 PMACheckpoint at step 15000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.85s.
11:44:54 PMAStep 15000: val 3.080, train 3.079 (gap +0.001).
11:01:18 PMAStep 14750: val 3.079, train 3.069 (gap +0.010).
10:19:14 PMACheckpoint reel: step 14,000 vs 14,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0014500.mp4
10:17:29 PMACheckpoint at step 14500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.94s.
10:17:24 PMAStep 14500: val 3.113, train 3.034 (gap +0.080).
10:07:00 PMAStep 14440 / 19073 (76%), 7.57B tokens. Train loss 3.053, best val 3.104, fact probes 0.74. 50281 tok/s; about 00d 13h 30m left.
09:33:48 PMAStep 14250: val 3.104, train 3.110 (gap -0.006).
08:51:44 PMACheckpoint reel: step 13,500 vs 14,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0014000.mp4
08:50:00 PMACheckpoint at step 14000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.80s.
08:49:55 PMAStep 14000: val 3.135, train 3.073 (gap +0.062).
08:07:00 PMAStep 13750 / 19073 (72%), 7.21B tokens. Train loss 3.092, best val 3.112, fact probes 0.75. 50108 tok/s; about 00d 15h 31m left.
08:06:09 PMAStep 13750: val 3.121, train 3.072 (gap +0.048).
07:23:38 PMACheckpoint reel: step 13,000 vs 13,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0013500.mp4
07:22:31 PMACheckpoint at step 13500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.70s.
07:22:26 PMAStep 13500: val 3.114, train 2.983 (gap +0.131).
06:38:52 PMAStep 13250: val 3.119, train 3.156 (gap -0.036).
06:07:00 PMAStep 13060 / 19073 (68%), 6.85B tokens. Train loss 3.102, best val 3.112, fact probes 0.76. 50097 tok/s; about 00d 17h 33m left.
05:56:26 PMACheckpoint reel: step 12,500 vs 13,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0013000.mp4
05:55:11 PMACheckpoint at step 13000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.75s.
05:55:06 PMAStep 13000: val 3.143, train 3.151 (gap -0.008).
05:11:31 PMAStep 12750: val 3.134, train 3.125 (gap +0.008).
04:28:32 PMACheckpoint reel: step 12,000 vs 12,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0012500.mp4
04:27:37 PMACheckpoint at step 12500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.76s.
04:27:32 PMAStep 12500: val 3.130, train 3.089 (gap +0.041).
04:07:00 PMAStep 12380 / 19073 (65%), 6.49B tokens. Train loss 3.112, best val 3.112, fact probes 0.74. 49989 tok/s; about 00d 19h 31m left.
03:43:50 PMAStep 12250: val 3.128, train 3.091 (gap +0.037).
03:01:48 PMACheckpoint reel: step 11,500 vs 12,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0012000.mp4
02:59:59 PMACheckpoint at step 12000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.81s.
02:59:54 PMAStep 12000: val 3.154, train 3.103 (gap +0.051).
02:16:12 PMAStep 11750: val 3.128, train 3.105 (gap +0.023).
02:07:00 PMAStep 11690 / 19073 (61%), 6.13B tokens. Train loss 3.116, best val 3.112, fact probes 0.77. 49969 tok/s; about 00d 21h 33m left.
01:34:06 PMACheckpoint reel: step 11,000 vs 11,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0011500.mp4
01:32:20 PMACheckpoint at step 11500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.93s.
01:32:15 PMAStep 11500: val 3.197, train 3.161 (gap +0.036).
12:48:33 PMAStep 11250: val 3.196, train 3.118 (gap +0.078).
12:07:00 PMAStep 11010 / 19073 (58%), 5.77B tokens. Train loss 3.126, best val 3.112, fact probes 0.71. 45801 tok/s; about 00d 23h 31m left.
12:06:31 PMACheckpoint reel: step 10,500 vs 11,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0011000.mp4
12:04:43 PMACheckpoint at step 11000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.84s.
12:04:38 PMAStep 11000: val 3.172, train 3.138 (gap +0.034).
11:20:55 AMAStep 10750: val 3.155, train 3.141 (gap +0.014).
10:38:56 AMACheckpoint reel: step 10,000 vs 10,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0010500.mp4
10:37:00 AMACheckpoint at step 10500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.96s.
10:36:55 AMAStep 10500: val 3.151, train 3.156 (gap -0.004).
10:07:00 AMAStep 10320 / 19073 (54%), 5.41B tokens. Train loss 3.149, best val 3.112, fact probes 0.75. 49990 tok/s; about 00d 25h 32m left.
09:53:16 AMAStep 10250: val 3.130, train 3.106 (gap +0.024).
09:11:14 AMACheckpoint reel: step 9,500 vs 10,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0010000.mp4
09:09:18 AMACheckpoint at step 10000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.77s.
09:09:13 AMAStep 10000: val 3.112, train 3.165 (gap -0.053).
08:25:35 AMAStep 9750: val 3.180, train 3.145 (gap +0.034).
08:07:00 AMAStep 9640 / 19073 (51%), 5.05B tokens. Train loss 3.155, best val 3.151, fact probes 0.74. 49847 tok/s; about 00d 27h 31m left.
07:42:59 AMACheckpoint reel: step 9,000 vs 9,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0009500.mp4
07:41:41 AMACheckpoint at step 9500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.96s.
07:41:36 AMAStep 9500: val 3.200, train 3.172 (gap +0.027).
06:57:59 AMAStep 9250: val 3.222, train 3.106 (gap +0.115).
06:14:57 AMACheckpoint reel: step 8,500 vs 9,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0009000.mp4
06:14:06 AMACheckpoint at step 9000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.86s.
06:14:01 AMAStep 9000: val 3.151, train 3.151 (gap +0.000).
06:07:00 AMAStep 8950 / 19073 (47%), 4.69B tokens. Train loss 3.151, best val 3.162, fact probes 0.73. 50116 tok/s; about 00d 29h 32m left.
05:30:22 AMAStep 8750: val 3.162, train 3.184 (gap -0.023).
04:48:25 AMACheckpoint reel: step 8,000 vs 8,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0008500.mp4
04:46:39 AMACheckpoint at step 8500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.85s.
04:46:34 AMAStep 8500: val 3.172, train 3.197 (gap -0.025).
04:07:00 AMAStep 8270 / 19073 (43%), 4.34B tokens. Train loss 3.183, best val 3.163, fact probes 0.75. 50158 tok/s; about 00d 31h 30m left.
04:02:55 AMAStep 8250: val 3.163, train 3.112 (gap +0.051).
03:20:08 AMACheckpoint reel: step 7,500 vs 8,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0008000.mp4
03:19:14 AMACheckpoint at step 8000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 3.11s.
03:19:08 AMAStep 8000: val 3.246, train 3.152 (gap +0.094).
02:35:36 AMAStep 7750: val 3.215, train 3.205 (gap +0.010).
02:07:00 AMAStep 7580 / 19073 (40%), 3.97B tokens. Train loss 3.204, best val 3.216, fact probes 0.76. 49997 tok/s; about 00d 33h 32m left.
01:53:23 AMACheckpoint reel: step 7,000 vs 7,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0007500.mp4
01:51:49 AMACheckpoint at step 7500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.68s.
01:51:44 AMAStep 7500: val 3.216, train 3.204 (gap +0.012).
01:08:10 AMAStep 7250: val 3.242, train 3.212 (gap +0.030).
12:26:18 AMACheckpoint reel: step 6,500 vs 7,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0007000.mp4
12:24:35 AMACheckpoint at step 7000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.63s.
12:24:30 AMAStep 7000: val 3.239, train 3.221 (gap +0.018).
12:07:00 AMAStep 6890 / 19073 (36%), 3.61B tokens. Train loss 3.215, best val 3.227, fact probes 0.75. 50271 tok/s; about 00d 35h 34m left.
11:40:54 PMAStep 6750: val 3.235, train 3.222 (gap +0.013).
10:58:24 PMACheckpoint reel: step 6,000 vs 6,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0006500.mp4
10:57:02 PMACheckpoint at step 6500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.62s.
10:56:57 PMAStep 6500: val 3.227, train 3.235 (gap -0.008).
10:13:29 PMAStep 6250: val 3.276, train 3.183 (gap +0.093).
10:07:00 PMAStep 6210 / 19073 (33%), 3.26B tokens. Train loss 3.250, best val 3.252, fact probes 0.74. 49947 tok/s; about 00d 37h 32m left.
09:30:41 PMACheckpoint reel: step 5,500 vs 6,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0006000.mp4
09:29:42 PMACheckpoint at step 6000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.62s.
09:29:37 PMAStep 6000: val 3.266, train 3.283 (gap -0.017).
08:46:04 PMAStep 5750: val 3.296, train 3.266 (gap +0.030).
08:07:00 PMAStep 5520 / 19073 (29%), 2.89B tokens. Train loss 3.285, best val 3.252, fact probes 0.68. 50034 tok/s; about 00d 39h 35m left.
08:03:39 PMACheckpoint reel: step 5,000 vs 5,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0005500.mp4
08:02:17 PMACheckpoint at step 5500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.93s.
08:02:12 PMAStep 5500: val 3.252, train 3.287 (gap -0.034).
07:18:42 PMAStep 5250: val 3.269, train 3.250 (gap +0.019).
06:36:40 PMACheckpoint reel: step 4,500 vs 5,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0005000.mp4
06:35:07 PMACheckpoint at step 5000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.88s.
06:35:02 PMAStep 5000: val 3.306, train 3.328 (gap -0.023).
06:07:00 PMAStep 4830 / 19073 (25%), 2.53B tokens. Train loss 3.281, best val 3.325, fact probes 0.65. 50220 tok/s; about 00d 41h 38m left.
05:51:31 PMAStep 4750: val 3.350, train 3.282 (gap +0.068).
05:08:53 PMACheckpoint reel: step 4,000 vs 4,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0004500.mp4
05:07:49 PMACheckpoint at step 4500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.80s.
05:07:44 PMAStep 4500: val 3.325, train 3.299 (gap +0.027).
04:24:11 PMAStep 4250: val 3.346, train 3.317 (gap +0.029).
04:07:00 PMAStep 4150 / 19073 (22%), 2.18B tokens. Train loss 3.335, best val 3.373, fact probes 0.63. 49970 tok/s; about 00d 43h 35m left.
03:41:46 PMACheckpoint reel: step 3,500 vs 4,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0004000.mp4
03:40:24 PMACheckpoint at step 4000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 3.17s.
03:40:19 PMAStep 4000: val 3.403, train 3.319 (gap +0.084).
02:56:36 PMAStep 3750: val 3.372, train 3.381 (gap -0.009).
02:14:02 PMACheckpoint reel: step 3,000 vs 3,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0003500.mp4
02:12:51 PMACheckpoint at step 3500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.69s.
02:12:46 PMAStep 3500: val 3.405, train 3.391 (gap +0.014).
02:07:00 PMAStep 3460 / 19073 (18%), 1.81B tokens. Train loss 3.388, best val 3.467, fact probes 0.61. 50024 tok/s; about 00d 45h 40m left.
01:29:09 PMAStep 3250: val 3.490, train 3.379 (gap +0.110).
12:46:38 PMACheckpoint reel: step 2,500 vs 3,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0003000.mp4
12:45:13 PMACheckpoint at step 3000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.57s.
12:45:08 PMAStep 3000: val 3.469, train 3.404 (gap +0.065).
12:07:00 PMAStep 2780 / 19073 (15%), 1.46B tokens. Train loss 3.470, best val 3.467, fact probes 0.61. 49870 tok/s; about 00d 47h 36m left.
12:01:22 PMAStep 2750: val 3.467, train 3.452 (gap +0.015).
11:18:31 AMACheckpoint reel: step 2,000 vs 2,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0002500.mp4
11:16:43 AMACheckpoint at step 2500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.64s.
11:16:38 AMAStep 2500: val 3.530, train 3.447 (gap +0.083).
10:33:01 AMAStep 2250: val 3.588, train 3.544 (gap +0.044).
10:07:00 AMAStep 2100 / 19073 (11%), 1.10B tokens. Train loss 3.558, best val 3.596, fact probes 0.58. 50140 tok/s; about 00d 49h 29m left.
09:51:15 AMACheckpoint reel: step 1,500 vs 2,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0002000.mp4
09:49:27 AMACheckpoint at step 2000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.94s.
09:49:22 AMAStep 2000: val 3.596, train 3.568 (gap +0.028).
09:05:40 AMAStep 1750: val 3.620, train 3.645 (gap -0.025).
08:23:37 AMACheckpoint reel: step 1,000 vs 1,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0001500.mp4
08:21:59 AMACheckpoint at step 1500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 3.26s.
08:21:53 AMAStep 1500: val 3.743, train 3.739 (gap +0.004).
08:07:00 AMAStep 1410 / 19073 (7%), 739M tokens. Train loss 3.765, best val 3.843, fact probes 0.60. 50296 tok/s; about 00d 51h 38m left.
07:38:20 AMAStep 1250: val 3.843, train 3.814 (gap +0.029).
06:56:13 AMACheckpoint reel: step 500 vs 1,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0001000.mp4
06:54:34 AMACheckpoint at step 1000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.91s.
06:54:29 AMAStep 1000: val 4.060, train 4.004 (gap +0.056).
06:10:57 AMAStep 750: val 4.472, train 4.412 (gap +0.060).
06:07:00 AMAStep 720 / 19073 (4%), 377M tokens. Train loss 4.478, best val 5.240, fact probes 0.47. 50059 tok/s; about 00d 54h 06m left.
05:28:37 AMACheckpoint reel: step 0 vs 500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0000500.mp4
05:27:10 AMACheckpoint at step 500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.54s.
05:27:05 AMAStep 500: val 5.240, train 5.112 (gap +0.128).
04:43:13 AMAStep 250: val 6.106, train 6.038 (gap +0.068).
04:07:00 AMAStep 40 / 19073 (0%), 21M tokens. Train loss 8.921, best val –. 50295 tok/s; about 00d 58h 16m left.
03:59:45 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0000000.pt in 5.0s (in the background; training didn't wait).
03:59:40 AMACheckpoint at step 0: 498.0 MB (weights, AdamW state, schedule position, RNG states) in 0.93s.
03:59:39 AMAExpected loss before training ≈ ln(vocab) = 10.83.
03:59:39 AMATraining 124.48M params on fineweb_edu_10b for 19073 steps (bf16, batch 8×64, ctx 1024).
03:53:36 AMARun A was waiting when the worker restarted; it's back in the queue.