Break it: bad initialisation

lesson 1.8 · Break it

Why initialise weights with std 0.02 instead of 1?

init_std: A 0.02 vs B 1

Run A

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

16,384,000

step

2,000 / 2,000

train loss: 5.325
best val: 5.300
tokens/s: 22k
MFU: 3.7%
peak mem: 6.4 GB
lr: 1.0e-4

0:14:01 · 0.016 kWh ≈ <$0.01 power · ≈ $0.07 to rent

Run B

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

16,384,000

step

2,000 / 2,000

train loss: 7.552
best val: 7.625
tokens/s: 22k
MFU: 3.7%
peak mem: 6.4 GB
lr: 1.0e-4

0:14:01 · 0.016 kWh ≈ <$0.01 power · ≈ $0.07 to rent

Losslower is better · starts near ln(vocab)

The machine, live

NVIDIA L4 · gcp-l4-lab2offline · just now

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

0.033 kWh

≈ <$0.01 of electricity

RAM

–

of 31.3 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

What to watch · B should start far above ln(vocab) (confidently wrong predictions) and spend its first hundreds of steps undoing the initialisation. Compare the curves at step 100.

Run A

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

Run B

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

GPU memory (measured)peak per log window · dashed = activations + workspace

Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right

The questions

arithmetic	A 0.49	B 0.43
ice	A 0.64	B 0.88
ocean	A 0.54	B 0.71
orbit	A 0.71	B 0.69
paris	A 0.56	B 0.41
planet	A 0.27	B 0.22
sky	A 0.68	B 0.31
water	A 0.42	B 0.30
whale	A 0.03	B 0.03

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

step 2,000

Run A

Water is the most important part of the future. This can also be the most important part of the water, so the energy is necessary for our body, and it’s important to help you will be to keep your body off. You’re not only interested in making sure to find some of these reasons to help your body and the best needed to get rid of your body. You can’t see to get you on the energy, but also that’s not yet important. This is when you do not know with it.
The new way to make your body to take your first step. You’re not trying to do with your body, and you will try it to put on to the heart and heart of your body up in a side of the body. It will always be sure to do with your body and make you ready for your body.
It’s a good way to do not know that you need to do it. A more severe

Run B

Water is.
 to
 the� and. of the. ( the of the. The not to of, and the in, is of the, and and a by of of to
 be the the to
 of the can on. The.
 or the the is the., and. The with. to the: are and the be, as and of the.
,.
. the in. to, you, that. This the to that of the.
.
. This on to in a the the with as). The of� the the,.
 at�
, the,, as. to.
- the and.
 and, the,
, and, to the the not, and of the In in a and and’, a the of the is the. The.

,.
The and in the a. the from the a.
 and, have the that, the
� the The that a the

What happened

09:45:25 AMAFinished 2000 steps. Best val loss 5.300.
09:45:25 AMBFinished 2000 steps. Best val loss 7.625.
09:45:25 AMACheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.16s.
09:45:25 AMBCheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.47s.
09:45:21 AMAStep 2000: val 5.300, train 5.318 (gap -0.018).
09:45:20 AMBStep 2000: val 7.625, train 7.657 (gap -0.032).
09:44:01 AMAStep 1800: val 5.362, train 5.304 (gap +0.057).
09:44:01 AMBStep 1800: val 7.653, train 7.654 (gap -0.002).
09:43:36 AMBCheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.38s.
09:43:36 AMACheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.45s.
09:42:37 AMBStep 1600: val 7.687, train 7.641 (gap +0.047).
09:42:37 AMAStep 1600: val 5.435, train 5.367 (gap +0.068).
09:41:54 AMBCheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.45s.
09:41:54 AMACheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.51s.
09:41:13 AMBStep 1400: val 7.650, train 7.628 (gap +0.023).
09:41:12 AMAStep 1400: val 5.462, train 5.411 (gap +0.050).
09:40:11 AMBCheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.48s.
09:40:11 AMACheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.45s.
09:39:49 AMAStep 1200: val 5.596, train 5.556 (gap +0.040).
09:39:49 AMBStep 1200: val 7.726, train 7.717 (gap +0.009).
09:38:29 AMBCheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.38s.
09:38:29 AMACheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.40s.
09:38:25 AMBStep 1000: val 7.733, train 7.721 (gap +0.012).
09:38:25 AMAStep 1000: val 5.673, train 5.627 (gap +0.046).
09:37:05 AMBStep 800: val 7.787, train 7.770 (gap +0.018).
09:37:05 AMAStep 800: val 5.782, train 5.786 (gap -0.004).
09:36:40 AMACheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.58s.
09:36:40 AMBCheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.44s.
09:35:41 AMBStep 600: val 8.040, train 8.062 (gap -0.022).
09:35:41 AMAStep 600: val 5.975, train 5.990 (gap -0.015).
09:34:58 AMBCheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.46s.
09:34:58 AMACheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.40s.
09:34:16 AMAStep 400: val 6.265, train 6.193 (gap +0.072).
09:34:16 AMBStep 400: val 9.683, train 9.650 (gap +0.033).
09:33:15 AMACheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.50s.
09:33:15 AMBCheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.52s.
09:32:52 AMAStep 200: val 6.571, train 6.549 (gap +0.022).
09:32:52 AMBStep 200: val 19.548, train 19.539 (gap +0.009).
09:32:10 AMACheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.48s.
09:32:10 AMBCheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.59s.
09:31:48 AMACheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.53s.
09:31:48 AMBCheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.65s.
09:31:27 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/6514a70a-839b-40c9-b98e-211fd5326f5c/step-0000000.pt in 2.2s (in the background; training didn't wait).
09:31:27 AMBCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/e8e73525-f95d-455d-8231-a54a5c755f9b/step-0000000.pt in 1.7s (in the background; training didn't wait).
09:31:25 AMBCheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.99s.
09:31:25 AMACheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 1.05s.
09:31:24 AMBInit std 1.0 instead of 0.02: the first loss may not be ln(vocab).
09:31:24 AMAExpected loss before training ≈ ln(vocab) = 10.83.
09:31:24 AMBExpected loss before training ≈ ln(vocab) = 10.83.
09:31:24 AMATraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).
09:31:23 AMBTraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).