train.galvyn.xyz · lab
read-onlySign in
← lab

GPT-2 124M on 10B tokens (Karpathy recipe)

lesson 1.9 · Scale

What does a real GPT-2-sized pretraining run look like, step by step?

Run A

running

fineweb_edu_10b · 12L·768d · 124M params · bf16

tokens read

7,602,176,000

step

14,500 / 19,073

train loss
3.048
best val
3.104
tokens/s
50k
MFU
38.6%
peak mem
7.0 GB
lr
1.4e-4

42:19:26 · ~13h 20m left · 9.5 kWh ≈ $1.43 power · ≈ $13 to rent

Losslower is better · starts near ln(vocab)
510500010kstepA checkpoint @ 500A checkpoint @ 1000A checkpoint @ 1500A checkpoint @ 2000A checkpoint @ 2500A checkpoint @ 3000A checkpoint @ 3500A checkpoint @ 4000A checkpoint @ 4500A checkpoint @ 5000A checkpoint @ 5500A checkpoint @ 6000A checkpoint @ 6500A checkpoint @ 7000A checkpoint @ 7500A checkpoint @ 8000A checkpoint @ 8500A checkpoint @ 9000A checkpoint @ 9500A checkpoint @ 10000A checkpoint @ 10500A checkpoint @ 11000A checkpoint @ 11500A checkpoint @ 12000A checkpoint @ 12500A checkpoint @ 13000A checkpoint @ 13500A checkpoint @ 14000A checkpoint @ 14500

The machine, live

NVIDIA RTX A5000 · red-gpu-wslupdated just now

GPU busy

99 %

flat out

VRAM

12.6 / 24 GB

room left

Power

223 / 230 W

at the cap

Temperature

79 °C

hot

SM clock

1500 MHz

boosting

Memory bus

46 %

streaming

Fan

54 %

spinning

CPU

7 %

light

PCIe

to GPU 29 MB/sfrom GPU 78 MB/s

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read 72 KB/swrite 0 B/s

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in 5.0 KB/sout 3.4 KB/s

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

223 W

≈ $0.03 an hour

last 5 minutes

18.4 Wh

from the card's energy counter

this experiment

9.50 kWh

≈ $1.43 of electricity

RAM

6.5 GB

of 62.7 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

Checkpoint reels

At every checkpoint, the same question to the previous checkpoint and the new one. Same seed, so only the model changed.

  • step 13,500 → 14,000

  • step 13,000 → 13,500

  • step 12,500 → 13,000

  • step 12,000 → 12,500

  • step 11,500 → 12,000

  • step 11,000 → 11,500

  • step 10,500 → 11,000

  • step 10,000 → 10,500

  • step 9,500 → 10,000

  • step 9,000 → 9,500

  • step 8,500 → 9,000

  • step 8,000 → 8,500

  • step 7,500 → 8,000

  • step 7,000 → 7,500

  • step 6,500 → 7,000

  • step 6,000 → 6,500

  • step 5,500 → 6,000

  • step 5,000 → 5,500

  • step 4,500 → 5,000

  • step 4,000 → 4,500

  • step 3,500 → 4,000

  • step 3,000 → 3,500

  • step 2,500 → 3,000

  • step 2,000 → 2,500

  • step 1,500 → 2,000

  • step 1,000 → 1,500

  • step 500 → 1,000

  • step 0 → 500

What's happening

What to watch · A single ~2.3-day run. Every 500 steps (~262M tokens) there's a checkpoint to try in the Playground. Watch the fact probes: at what point does it learn that water is a liquid?

  1. Queued

    done

  2. Download data

    done

  3. Tokenize

    done

  4. Build and compile

    done

  5. Warm up

    done

  6. Train

    step 14,500 / 19,073

  7. Finished

    next

Now · Train · Forward, backward, AdamW update, again and again. The learning rate follows a cosine down to a tenth of its peak.

  • Evaluateevery 250 steps

    last 14,500 · next 14,750

    loss on held-out text, plus the fact probes (does it know water is a liquid?)

  • Write a sampleevery 500 steps

    last 14,500 · next 15,000

    continue a fixed prompt so you can read it improve

  • Checkpointevery 500 steps

    last 14,500 · next 15,000

    save weights + optimizer + RNG, upload to Cloud Storage; it appears in the Playground

solid = train (smoothed) · dots = validation · dashed = after a crash-test resume
Learning ratewarmup, then cosine
0e+02e-44e-46e-4500010kstep
Gradient normbefore clipping to 1.0 · log scale
110500010kstep
Throughputtokens per second
35k40k45k50k500010kstep
MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)
30.0%35.0%40.0%500010kstep
GPU memory (measured)peak per log window · dashed = activations + workspace
3.9 GB4.9 GB5.9 GB6.8 GB500010kstep
Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right
00.51500010kstep
The questions
arithmeticA 0.62
iceA 0.16
oceanA 0.67
orbitA 0.98
parisA 1.00
planetA 0.98
skyA 0.73
waterA 0.88
whaleA 0.88

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

Run A

Water is the most important part of the water cycle. It provides the water for the ecosystem.
It can also act as a buffer against flooding and erosion.
It is a natural filter that filters out impurities and salt water.
It is a major part of the water supply.
The water that we provide to its users are called the tap water. It is the most important part of the water supply in the UK.
It is a clean, safe, healthy and sustainable water.
It is used to drink, clean the environment and to produce nutrients for plants.
It is used to supply water for domestic, industrial, agricultural and industrial purposes.
It is used to generate heat, light, noise, power, cooling and heat.
It is used to produce hot liquids and for drinking.
It is used to produce cold and hot liquids. It is used to produce hot liquids.
It is used to produce hot liquids and other products.
It is used to generate heat

What happened

  1. ACheckpoint at step 14500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.94s.
  2. AStep 14500: val 3.113, train 3.034 (gap +0.080).
  3. AStep 14440 / 19073 (76%), 7.57B tokens. Train loss 3.053, best val 3.104, fact probes 0.74. 50281 tok/s; about 00d 13h 30m left.
  4. AStep 14250: val 3.104, train 3.110 (gap -0.006).
  5. ACheckpoint reel: step 13,500 vs 14,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0014000.mp4
  6. ACheckpoint at step 14000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.80s.
  7. AStep 14000: val 3.135, train 3.073 (gap +0.062).
  8. AStep 13750 / 19073 (72%), 7.21B tokens. Train loss 3.092, best val 3.112, fact probes 0.75. 50108 tok/s; about 00d 15h 31m left.
  9. AStep 13750: val 3.121, train 3.072 (gap +0.048).
  10. ACheckpoint reel: step 13,000 vs 13,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0013500.mp4
  11. ACheckpoint at step 13500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.70s.
  12. AStep 13500: val 3.114, train 2.983 (gap +0.131).
  13. AStep 13250: val 3.119, train 3.156 (gap -0.036).
  14. AStep 13060 / 19073 (68%), 6.85B tokens. Train loss 3.102, best val 3.112, fact probes 0.76. 50097 tok/s; about 00d 17h 33m left.
  15. ACheckpoint reel: step 12,500 vs 13,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0013000.mp4
  16. ACheckpoint at step 13000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.75s.
  17. AStep 13000: val 3.143, train 3.151 (gap -0.008).
  18. AStep 12750: val 3.134, train 3.125 (gap +0.008).
  19. ACheckpoint reel: step 12,000 vs 12,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0012500.mp4
  20. ACheckpoint at step 12500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.76s.
  21. AStep 12500: val 3.130, train 3.089 (gap +0.041).
  22. AStep 12380 / 19073 (65%), 6.49B tokens. Train loss 3.112, best val 3.112, fact probes 0.74. 49989 tok/s; about 00d 19h 31m left.
  23. AStep 12250: val 3.128, train 3.091 (gap +0.037).
  24. ACheckpoint reel: step 11,500 vs 12,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0012000.mp4
  25. ACheckpoint at step 12000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.81s.
  26. AStep 12000: val 3.154, train 3.103 (gap +0.051).
  27. AStep 11750: val 3.128, train 3.105 (gap +0.023).
  28. AStep 11690 / 19073 (61%), 6.13B tokens. Train loss 3.116, best val 3.112, fact probes 0.77. 49969 tok/s; about 00d 21h 33m left.
  29. ACheckpoint reel: step 11,000 vs 11,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0011500.mp4
  30. ACheckpoint at step 11500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.93s.
  31. AStep 11500: val 3.197, train 3.161 (gap +0.036).
  32. AStep 11250: val 3.196, train 3.118 (gap +0.078).
  33. AStep 11010 / 19073 (58%), 5.77B tokens. Train loss 3.126, best val 3.112, fact probes 0.71. 45801 tok/s; about 00d 23h 31m left.
  34. ACheckpoint reel: step 10,500 vs 11,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0011000.mp4
  35. ACheckpoint at step 11000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.84s.
  36. AStep 11000: val 3.172, train 3.138 (gap +0.034).
  37. AStep 10750: val 3.155, train 3.141 (gap +0.014).
  38. ACheckpoint reel: step 10,000 vs 10,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0010500.mp4
  39. ACheckpoint at step 10500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.96s.
  40. AStep 10500: val 3.151, train 3.156 (gap -0.004).
  41. AStep 10320 / 19073 (54%), 5.41B tokens. Train loss 3.149, best val 3.112, fact probes 0.75. 49990 tok/s; about 00d 25h 32m left.
  42. AStep 10250: val 3.130, train 3.106 (gap +0.024).
  43. ACheckpoint reel: step 9,500 vs 10,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0010000.mp4
  44. ACheckpoint at step 10000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.77s.
  45. AStep 10000: val 3.112, train 3.165 (gap -0.053).
  46. AStep 9750: val 3.180, train 3.145 (gap +0.034).
  47. AStep 9640 / 19073 (51%), 5.05B tokens. Train loss 3.155, best val 3.151, fact probes 0.74. 49847 tok/s; about 00d 27h 31m left.
  48. ACheckpoint reel: step 9,000 vs 9,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0009500.mp4
  49. ACheckpoint at step 9500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.96s.
  50. AStep 9500: val 3.200, train 3.172 (gap +0.027).
  51. AStep 9250: val 3.222, train 3.106 (gap +0.115).
  52. ACheckpoint reel: step 8,500 vs 9,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0009000.mp4
  53. ACheckpoint at step 9000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.86s.
  54. AStep 9000: val 3.151, train 3.151 (gap +0.000).
  55. AStep 8950 / 19073 (47%), 4.69B tokens. Train loss 3.151, best val 3.162, fact probes 0.73. 50116 tok/s; about 00d 29h 32m left.
  56. AStep 8750: val 3.162, train 3.184 (gap -0.023).
  57. ACheckpoint reel: step 8,000 vs 8,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0008500.mp4
  58. ACheckpoint at step 8500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.85s.
  59. AStep 8500: val 3.172, train 3.197 (gap -0.025).
  60. AStep 8270 / 19073 (43%), 4.34B tokens. Train loss 3.183, best val 3.163, fact probes 0.75. 50158 tok/s; about 00d 31h 30m left.
  61. AStep 8250: val 3.163, train 3.112 (gap +0.051).
  62. ACheckpoint reel: step 7,500 vs 8,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0008000.mp4
  63. ACheckpoint at step 8000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 3.11s.
  64. AStep 8000: val 3.246, train 3.152 (gap +0.094).
  65. AStep 7750: val 3.215, train 3.205 (gap +0.010).
  66. AStep 7580 / 19073 (40%), 3.97B tokens. Train loss 3.204, best val 3.216, fact probes 0.76. 49997 tok/s; about 00d 33h 32m left.
  67. ACheckpoint reel: step 7,000 vs 7,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0007500.mp4
  68. ACheckpoint at step 7500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.68s.
  69. AStep 7500: val 3.216, train 3.204 (gap +0.012).
  70. AStep 7250: val 3.242, train 3.212 (gap +0.030).
  71. ACheckpoint reel: step 6,500 vs 7,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0007000.mp4
  72. ACheckpoint at step 7000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.63s.
  73. AStep 7000: val 3.239, train 3.221 (gap +0.018).
  74. AStep 6890 / 19073 (36%), 3.61B tokens. Train loss 3.215, best val 3.227, fact probes 0.75. 50271 tok/s; about 00d 35h 34m left.
  75. AStep 6750: val 3.235, train 3.222 (gap +0.013).
  76. ACheckpoint reel: step 6,000 vs 6,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0006500.mp4
  77. ACheckpoint at step 6500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.62s.
  78. AStep 6500: val 3.227, train 3.235 (gap -0.008).
  79. AStep 6250: val 3.276, train 3.183 (gap +0.093).
  80. AStep 6210 / 19073 (33%), 3.26B tokens. Train loss 3.250, best val 3.252, fact probes 0.74. 49947 tok/s; about 00d 37h 32m left.
  81. ACheckpoint reel: step 5,500 vs 6,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0006000.mp4
  82. ACheckpoint at step 6000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.62s.
  83. AStep 6000: val 3.266, train 3.283 (gap -0.017).
  84. AStep 5750: val 3.296, train 3.266 (gap +0.030).
  85. AStep 5520 / 19073 (29%), 2.89B tokens. Train loss 3.285, best val 3.252, fact probes 0.68. 50034 tok/s; about 00d 39h 35m left.
  86. ACheckpoint reel: step 5,000 vs 5,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0005500.mp4
  87. ACheckpoint at step 5500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.93s.
  88. AStep 5500: val 3.252, train 3.287 (gap -0.034).
  89. AStep 5250: val 3.269, train 3.250 (gap +0.019).
  90. ACheckpoint reel: step 4,500 vs 5,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0005000.mp4
  91. ACheckpoint at step 5000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.88s.
  92. AStep 5000: val 3.306, train 3.328 (gap -0.023).
  93. AStep 4830 / 19073 (25%), 2.53B tokens. Train loss 3.281, best val 3.325, fact probes 0.65. 50220 tok/s; about 00d 41h 38m left.
  94. AStep 4750: val 3.350, train 3.282 (gap +0.068).
  95. ACheckpoint reel: step 4,000 vs 4,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0004500.mp4
  96. ACheckpoint at step 4500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.80s.
  97. AStep 4500: val 3.325, train 3.299 (gap +0.027).
  98. AStep 4250: val 3.346, train 3.317 (gap +0.029).
  99. AStep 4150 / 19073 (22%), 2.18B tokens. Train loss 3.335, best val 3.373, fact probes 0.63. 49970 tok/s; about 00d 43h 35m left.
  100. ACheckpoint reel: step 3,500 vs 4,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0004000.mp4
  101. ACheckpoint at step 4000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 3.17s.
  102. AStep 4000: val 3.403, train 3.319 (gap +0.084).
  103. AStep 3750: val 3.372, train 3.381 (gap -0.009).
  104. ACheckpoint reel: step 3,000 vs 3,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0003500.mp4
  105. ACheckpoint at step 3500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.69s.
  106. AStep 3500: val 3.405, train 3.391 (gap +0.014).
  107. AStep 3460 / 19073 (18%), 1.81B tokens. Train loss 3.388, best val 3.467, fact probes 0.61. 50024 tok/s; about 00d 45h 40m left.
  108. AStep 3250: val 3.490, train 3.379 (gap +0.110).
  109. ACheckpoint reel: step 2,500 vs 3,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0003000.mp4
  110. ACheckpoint at step 3000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.57s.
  111. AStep 3000: val 3.469, train 3.404 (gap +0.065).
  112. AStep 2780 / 19073 (15%), 1.46B tokens. Train loss 3.470, best val 3.467, fact probes 0.61. 49870 tok/s; about 00d 47h 36m left.
  113. AStep 2750: val 3.467, train 3.452 (gap +0.015).
  114. ACheckpoint reel: step 2,000 vs 2,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0002500.mp4
  115. ACheckpoint at step 2500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.64s.
  116. AStep 2500: val 3.530, train 3.447 (gap +0.083).
  117. AStep 2250: val 3.588, train 3.544 (gap +0.044).
  118. AStep 2100 / 19073 (11%), 1.10B tokens. Train loss 3.558, best val 3.596, fact probes 0.58. 50140 tok/s; about 00d 49h 29m left.
  119. ACheckpoint reel: step 1,500 vs 2,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0002000.mp4
  120. ACheckpoint at step 2000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.94s.
  121. AStep 2000: val 3.596, train 3.568 (gap +0.028).
  122. AStep 1750: val 3.620, train 3.645 (gap -0.025).
  123. ACheckpoint reel: step 1,000 vs 1,500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0001500.mp4
  124. ACheckpoint at step 1500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 3.26s.
  125. AStep 1500: val 3.743, train 3.739 (gap +0.004).
  126. AStep 1410 / 19073 (7%), 739M tokens. Train loss 3.765, best val 3.843, fact probes 0.60. 50296 tok/s; about 00d 51h 38m left.
  127. AStep 1250: val 3.843, train 3.814 (gap +0.029).
  128. ACheckpoint reel: step 500 vs 1,000, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0001000.mp4
  129. ACheckpoint at step 1000: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.91s.
  130. AStep 1000: val 4.060, train 4.004 (gap +0.056).
  131. AStep 750: val 4.472, train 4.412 (gap +0.060).
  132. AStep 720 / 19073 (4%), 377M tokens. Train loss 4.478, best val 5.240, fact probes 0.47. 50059 tok/s; about 00d 54h 06m left.
  133. ACheckpoint reel: step 0 vs 500, same question. https://ezoxjqzcxrewbalztttl.supabase.co/storage/v1/object/public/train-galvyn-xyz-clips/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0000500.mp4
  134. ACheckpoint at step 500: 1493.9 MB (weights, AdamW state, schedule position, RNG states) in 2.54s.
  135. AStep 500: val 5.240, train 5.112 (gap +0.128).
  136. AStep 250: val 6.106, train 6.038 (gap +0.068).
  137. AStep 40 / 19073 (0%), 21M tokens. Train loss 8.921, best val –. 50295 tok/s; about 00d 58h 16m left.
  138. ACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/db9c28c3-9a80-4a58-b4de-5ba6f4b5277e/step-0000000.pt in 5.0s (in the background; training didn't wait).
  139. ACheckpoint at step 0: 498.0 MB (weights, AdamW state, schedule position, RNG states) in 0.93s.
  140. AExpected loss before training ≈ ln(vocab) = 10.83.
  141. ATraining 124.48M params on fineweb_edu_10b for 19073 steps (bf16, batch 8×64, ctx 1024).
  142. ARun A was waiting when the worker restarted; it's back in the queue.