train.galvyn.xyz · lab
read-onlySign in
← lab

Break it: targets equal inputs

lesson 1.8 · Break it

What does a classic data bug look like from the outside?

data_bug: A none vs B no_shift

Run A

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

16,384,000

step

2,000 / 2,000

train loss
5.325
best val
5.300
tokens/s
22k
MFU
3.7%
peak mem
6.4 GB
lr
1.0e-4

0:14:01 · 0.016 kWh ≈ <$0.01 power · ≈ $0.07 to rent

Run B

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

16,384,000

step

2,000 / 2,000

train loss
0.020
best val
0.027
tokens/s
22k
MFU
3.7%
peak mem
6.4 GB
lr
1.0e-4

0:13:02 · 0.015 kWh ≈ <$0.01 power · ≈ $0.07 to rent

Losslower is better · starts near ln(vocab)
0510500100015002000stepA checkpoint @ 50B checkpoint @ 50A checkpoint @ 100B checkpoint @ 100A checkpoint @ 250B checkpoint @ 250B checkpoint @ 500A checkpoint @ 500A checkpoint @ 750B checkpoint @ 750A checkpoint @ 1000B checkpoint @ 1000B checkpoint @ 1250A checkpoint @ 1250B checkpoint @ 1500A checkpoint @ 1500A checkpoint @ 1750B checkpoint @ 1750B checkpoint @ 2000A checkpoint @ 2000

The machine, live

NVIDIA L4 · gcp-l4-lab2offline · just now

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

0.032 kWh

≈ <$0.01 of electricity

RAM

–

of 31.3 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

What to watch · B's loss should collapse towards zero almost at once: too good to be true, because it only learns to copy its input. Its samples never improve. A loss far below what's possible is a bug, not a breakthrough.

Run A

  1. Queued

    done

  2. Download data

    done

  3. Tokenize

    done

  4. Build and compile

    done

  5. Warm up

    done

  6. Train

    done

  7. Finished

    done

Run B

  1. Queued

    done

  2. Download data

    done

  3. Tokenize

    done

  4. Build and compile

    done

  5. Warm up

    done

  6. Train

    done

  7. Finished

    done

solid = train (smoothed) · dots = validation · dashed = after a crash-test resume
Learning ratewarmup, then cosine
0e+05e-41e-3500100015002000step
Gradient normbefore clipping to 1.0 · log scale
0.11500100015002000step
Throughputtokens per second
20k40k60k500100015002000step
MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)
5.0%10.0%500100015002000step
GPU memory (measured)peak per log window · dashed = activations + workspace
5.9 GB6.3 GB500100015002000step
Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right
00.20.4500100015002000step
The questions
arithmeticA 0.49B 0.15
iceA 0.64B 0.00
oceanA 0.54B 0.00
orbitA 0.71B 0.00
parisA 0.56B 0.00
planetA 0.27B 0.00
skyA 0.68B 0.01
waterA 0.42B 0.01
whaleA 0.03B 0.00

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

Run A

Water is the most important part of the future. This can also be the most important part of the water, so the energy is necessary for our body, and it’s important to help you will be to keep your body off. You’re not only interested in making sure to find some of these reasons to help your body and the best needed to get rid of your body. You can’t see to get you on the energy, but also that’s not yet important. This is when you do not know with it.
The new way to make your body to take your first step. You’re not trying to do with your body, and you will try it to put on to the heart and heart of your body up in a side of the body. It will always be sure to do with your body and make you ready for your body.
It’s a good way to do not know that you need to do it. A more severe

Run B

Water is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is

What happened

  1. BFinished 2000 steps. Best val loss 0.027.
  2. BCheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
  3. BStep 2000: val 0.027, train 0.023 (gap +0.003).
  4. BStep 1800: val 0.031, train 0.025 (gap +0.006).
  5. BCheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
  6. BStep 1600: val 0.031, train 0.029 (gap +0.002).
  7. BCheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.63s.
  8. BStep 1400: val 0.031, train 0.029 (gap +0.002).
  9. BCheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
  10. BStep 1200: val 0.036, train 0.031 (gap +0.004).
  11. BCheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.57s.
  12. BStep 1000: val 0.033, train 0.030 (gap +0.003).
  13. BStep 800: val 0.030, train 0.032 (gap -0.002).
  14. BCheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.61s.
  15. BStep 600: val 0.034, train 0.038 (gap -0.004).
  16. BCheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.56s.
  17. BStep 400: val 0.061, train 0.048 (gap +0.013).
  18. BCheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.95s.
  19. BStep 200: val 0.144, train 0.139 (gap +0.005).
  20. BCheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
  21. BCheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.56s.
  22. BCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/e36d077d-45a2-47ba-bc66-3975c55f009f/step-0000000.pt in 1.7s (in the background; training didn't wait).
  23. BCheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.52s.
  24. BData bug on purpose: targets equal inputs, so the model only learns to copy. Loss collapses; samples won't improve.
  25. BExpected loss before training ≈ ln(vocab) = 10.83.
  26. BTraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).
  27. AFinished 2000 steps. Best val loss 5.300.
  28. ACheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.17s.
  29. AStep 2000: val 5.300, train 5.318 (gap -0.018).
  30. AStep 1800: val 5.362, train 5.304 (gap +0.057).
  31. ACheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.58s.
  32. AStep 1600: val 5.435, train 5.367 (gap +0.068).
  33. ACheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.37s.
  34. AStep 1400: val 5.462, train 5.411 (gap +0.050).
  35. ACheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.58s.
  36. AStep 1200: val 5.596, train 5.556 (gap +0.040).
  37. ACheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.57s.
  38. AStep 1000: val 5.673, train 5.627 (gap +0.046).
  39. AStep 800: val 5.782, train 5.786 (gap -0.004).
  40. ACheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.46s.
  41. AStep 600: val 5.975, train 5.990 (gap -0.015).
  42. ACheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.64s.
  43. AStep 400: val 6.265, train 6.193 (gap +0.072).
  44. ACheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.64s.
  45. AStep 200: val 6.571, train 6.549 (gap +0.022).
  46. ACheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.52s.
  47. ACheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.55s.
  48. ACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/ca7a87ed-cead-450a-af4a-bd1e9941008f/step-0000000.pt in 1.9s (in the background; training didn't wait).
  49. ACheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 1.16s.
  50. AExpected loss before training ≈ ln(vocab) = 10.83.
  51. ATraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).