train.galvyn.xyz · lab
read-onlySign in
← lab

Break it: remove LayerNorm

lesson 1.8 · Break it

What is LayerNorm doing for training stability?

layernorm: A true vs B false

Run A

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

16,384,000

step

2,000 / 2,000

train loss
5.287
best val
5.256
tokens/s
74k
MFU
12.5%
peak mem
6.4 GB
lr
3.0e-4

0:04:43 · 0.005 kWh ≈ <$0.01 power · ≈ $0.02 to rent

Run B

failed

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

1,433,600

step

175 / 2,000

train loss
7215248.000
best val
–
tokens/s
34k
MFU
5.7%
peak mem
6.3 GB
lr
3.0e-3

0:00:44 · 0.001 kWh ≈ <$0.01 power · ≈ <$0.01 to rent

Diverged: loss became NaN at step 175 (gradient norm reached 15,057 at step 160). Without LayerNorm at lr 3e-3, this is the expected result of the lesson.

Losslower is better · starts near ln(vocab)
01.0M2.0M3.0M500100015002000stepB checkpoint @ 50A checkpoint @ 50B checkpoint @ 100A checkpoint @ 100A checkpoint @ 250A checkpoint @ 500A checkpoint @ 750A checkpoint @ 1000A checkpoint @ 1250A checkpoint @ 1500A checkpoint @ 1750A checkpoint @ 2000

The machine, live

NVIDIA L4 · gcp-l4-lab2offline · just now

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

0.006 kWh

≈ <$0.01 of electricity

RAM

–

of 31.3 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

What to watch · Without LayerNorm nothing holds the residual stream at a stable scale: watch B's gradient norm and loss for spikes, and a worse final loss.

Run A

  1. Queued

    done

  2. Download data

    done

  3. Tokenize

    done

  4. Build and compile

    done

  5. Warm up

    done

  6. Train

    done

  7. Finished

    done

Run B

  1. Queued

    done

  2. Download data

    done

  3. Tokenize

    done

  4. Build and compile

    done

  5. Warm up

    done

  6. Train

    stopped here

  7. Finished

    next

Stopped at · Train · Forward, backward, AdamW update, again and again. The learning rate follows a cosine down to a tenth of its peak.

  • Evaluateevery 200 steps

    next 200

    loss on held-out text, plus the fact probes (does it know water is a liquid?)

  • Write a sampleevery 500 steps

    last 0 · next 500

    continue a fixed prompt so you can read it improve

  • Checkpointevery 250 steps

    last 100 · next 250

    save weights + optimizer + RNG, upload to Cloud Storage; it appears in the Playground

solid = train (smoothed) · dots = validation · dashed = after a crash-test resume
Learning ratewarmup, then cosine
0e+01e-32e-33e-3500100015002000step
Gradient normbefore clipping to 1.0 · log scale
110100100010k500100015002000step
Throughputtokens per second
20k40k60k80k500100015002000step
MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)
5.0%10.0%500100015002000step
GPU memory (measured)peak per log window · dashed = activations + workspace
5.9 GB6.3 GB500100015002000step
Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right
0.350.40.45500100015002000step
The questions
arithmeticA 0.56B –
iceA 0.54B –
oceanA 0.78B –
orbitA 0.72B –
parisA 0.24B –
planetA 0.27B –
skyA 0.75B –
waterA 0.36B –
whaleA 0.13B –

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

Run A

Water is the most harmful causes of the soil. This can also be the most important factor in preventing water, preventing the health of the plant.
Sens and other health products like oil and other pesticides
Fry-friendly forms on the plant.
Sustriculture is also a source of water.
Osteopable water is a vital source of water and water in your body. A process in the body, you can also lead to the best care of the plants.
Punging is a free waste chain of the plant. It is a great way to prevent water from your home. The plant needs to be able to make sure you should make a better protection from your plant.
Wossing is a good advantage of your water quality in a healthy and healthy life. It is a great part of the system. It consists of the best way to prevent the bacteria.
Vene seeds are a good idea to check the water quality.
Treat your plant can

Run B

–

What happened

  1. AFinished 2000 steps. Best val loss 5.256.
  2. ACheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.58s.
  3. AStep 2000: val 5.256, train 5.271 (gap -0.015).
  4. AStep 1800: val 5.325, train 5.259 (gap +0.066).
  5. ACheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.52s.
  6. AStep 1600: val 5.413, train 5.342 (gap +0.071).
  7. ACheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.57s.
  8. AStep 1400: val 5.452, train 5.401 (gap +0.051).
  9. ACheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.52s.
  10. AStep 1200: val 5.607, train 5.563 (gap +0.044).
  11. ACheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
  12. AStep 1000: val 5.686, train 5.638 (gap +0.047).
  13. AStep 800: val 5.810, train 5.808 (gap +0.001).
  14. ACheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
  15. AStep 600: val 6.010, train 6.024 (gap -0.014).
  16. ACheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
  17. AStep 400: val 6.286, train 6.215 (gap +0.071).
  18. ACheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.52s.
  19. AStep 200: val 6.619, train 6.598 (gap +0.021).
  20. BLoss became nan at step 175. Stopping: a too-high learning rate is the usual cause.
  21. BCheckpoint at step 100: 360.7 MB (weights, AdamW state, schedule position, RNG states) in 1.64s.
  22. ACheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
  23. BCheckpoint at step 50: 360.7 MB (weights, AdamW state, schedule position, RNG states) in 1.64s.
  24. ACheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
  25. BCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/f23769e1-26f7-44f3-8e06-379c8ecaa5c3/step-0000000.pt in 1.5s (in the background; training didn't wait).
  26. ACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/2ec8eb1c-2aec-4cb7-b75c-cf6381e48924/step-0000000.pt in 1.7s (in the background; training didn't wait).
  27. BCheckpoint at step 0: 120.2 MB (weights, AdamW state, schedule position, RNG states) in 0.52s.
  28. BLayerNorm removed: nothing keeps the residual stream at a stable scale, so watch the gradient norm.
  29. BExpected loss before training ≈ ln(vocab) = 10.83.
  30. BTraining 30.05M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).
  31. ACheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.51s.
  32. AExpected loss before training ≈ ln(vocab) = 10.83.
  33. ATraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).
  34. BResuming run B from its checkpoint at step 0.
  35. AResuming run A from its checkpoint at step 0.
  36. BCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/f23769e1-26f7-44f3-8e06-379c8ecaa5c3/step-0000000.pt in 1.7s (in the background; training didn't wait).
  37. ACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/2ec8eb1c-2aec-4cb7-b75c-cf6381e48924/step-0000000.pt in 1.7s (in the background; training didn't wait).
  38. BRun crashed: OutOfMemoryError: CUDA out of memory. Tried to allocate 1.54 GiB. GPU 0 has a total capacity of 22.03 GiB of which 1.10 GiB is free. Process 8745 has 12.74 GiB memory in use. Process 8971 has 3.38 GiB memory in use. Including non-PyTorch memory, this process has 4.79 GiB memory in use. Of the allocated memory 4.51 GiB is allocated by PyTorch, and 50.88 MiB is reserved by PyTorch but unallocated.
  39. ARun crashed: OutOfMemoryError: CUDA out of memory. Tried to allocate 1.54 GiB. GPU 0 has a total capacity of 22.03 GiB of which 1.10 GiB is free. Process 8745 has 12.74 GiB memory in use. Including non-PyTorch memory, this process has 3.38 GiB memory in use. Process 8972 has 4.79 GiB memory in use. Of the allocated memory 3.13 GiB is allocated by PyTorch, and 27.99 MiB is reserved by PyTorch but unallocated.
  40. ACheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.82s.
  41. BCheckpoint at step 0: 120.2 MB (weights, AdamW state, schedule position, RNG states) in 0.70s.
  42. BLayerNorm removed: nothing keeps the residual stream at a stable scale, so watch the gradient norm.
  43. AExpected loss before training ≈ ln(vocab) = 10.83.
  44. BExpected loss before training ≈ ln(vocab) = 10.83.
  45. BTraining 30.05M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).
  46. ATraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).