Warmup vs no warmup

lesson 1.4 · Warmup

Why ramp the learning rate up instead of starting at full speed?

warmup: A 200 vs B 0

Run A

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

16,384,000

step

2,000 / 2,000

train loss: 5.232
best val: 5.206
tokens/s: 22k
MFU: 3.7%
peak mem: 6.4 GB
lr: 3.0e-4

0:13:35 · 0.016 kWh ≈ <$0.01 power · ≈ $0.07 to rent

Run B

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

16,384,000

step

2,000 / 2,000

train loss: 6.342
best val: 6.389
tokens/s: 22k
MFU: 3.7%
peak mem: 6.4 GB
lr: 3.0e-4

0:12:26 · 0.015 kWh ≈ <$0.01 power · ≈ $0.06 to rent

Losslower is better · starts near ln(vocab)

The machine, live

NVIDIA L4 · gcp-l4-laboffline · just now

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

0.031 kWh

≈ <$0.01 of electricity

RAM

–

of 31.3 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

What to watch · Without warmup the first steps hit freshly initialised weights at full learning rate: look for an early spike in B's loss and gradient norm.

Run A

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

Run B

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
skipped
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

GPU memory (measured)peak per log window · dashed = activations + workspace

Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right

The questions

arithmetic	A 0.55	B 0.52
ice	A 0.31	B 0.63
ocean	A 0.54	B 0.62
orbit	A 0.73	B 0.54
paris	A 0.29	B 0.15
planet	A 0.44	B 0.43
sky	A 0.70	B 0.42
water	A 0.60	B 0.22
whale	A 0.07	B 0.16

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

step 2,000

Run A

Water is the most important part of the soil. This can also be the most important factor in the soil when it is the soil is in which the pH and is a pH of the soil.
This is the most important part of the soil, which is the plant is made by the root and the alkal. This can be the soil in the soil as it is the roots of your soil. A pH in the soil, the soil, is an mineral nutrient that uses the soil.
This is a good place in a soil, with the leaves, especially if these plant roots will be collected at the leaves, the moisture, the leaves are also found in different ways. This is the place in the soil and the soil is the roots. It is also used in soil and it’s a unique tree that is the area of the soil and soil. It also can also help to reduce the soil and soil. It is also not required that water is done.
In general, the

Run B

Water is. The year is a few. We are the child is the child for the more important to the time is the very important is a good of any and a few-to-year years. If the
In the most important of a day is that is the same type of the most. The time were to its children are only the most of a small data can be used on the more.
There are to get you find that is an important to go on the right.
-level in fact can learn a lot of the other.
The new and the same will be to take the first can be the body.
You can need to be done what are all the time, it is a child to do the same point and a high quality of a result of the world, a lot of these kinds of the first way to prevent a few.
The next change, it will be the time.
For example have a lot of the next.
In this is the

What happened

09:39:08 AMBFinished 2000 steps. Best val loss 6.389.
09:39:07 AMBCheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:39:05 AMBStep 2000: val 6.389, train 6.419 (gap -0.030).
09:37:46 AMBStep 1800: val 6.425, train 6.402 (gap +0.023).
09:37:23 AMBCheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:36:26 AMBStep 1600: val 6.506, train 6.430 (gap +0.076).
09:35:44 AMBCheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.55s.
09:35:05 AMBStep 1400: val 6.484, train 6.444 (gap +0.039).
09:34:05 AMBCheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.55s.
09:33:46 AMBStep 1200: val 6.579, train 6.563 (gap +0.016).
09:32:33 AMBCheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
09:32:30 AMBStep 1000: val 6.605, train 6.562 (gap +0.043).
09:31:55 AMAFinished 2000 steps. Best val loss 5.206.
09:31:54 AMACheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.59s.
09:31:51 AMBStep 800: val 6.635, train 6.616 (gap +0.019).
09:31:51 AMAStep 2000: val 5.206, train 5.221 (gap -0.015).
09:31:28 AMBCheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.56s.
09:30:31 AMAStep 1800: val 5.277, train 5.209 (gap +0.068).
09:30:30 AMBStep 600: val 6.693, train 6.713 (gap -0.020).
09:30:07 AMACheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.65s.
09:29:54 AMBCheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.56s.
09:29:13 AMBStep 400: val 6.840, train 6.770 (gap +0.070).
09:29:09 AMAStep 1600: val 5.370, train 5.299 (gap +0.071).
09:28:26 AMACheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.58s.
09:28:19 AMBCheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.57s.
09:27:58 AMBStep 200: val 7.026, train 6.999 (gap +0.027).
09:27:46 AMAStep 1400: val 5.414, train 5.362 (gap +0.052).
09:27:16 AMBCheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.56s.
09:26:55 AMBCheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.63s.
09:26:46 AMACheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.96s.
09:26:42 AMBCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/ada51cf3-d298-42f1-ac0e-f271e31e08e1/step-0000000.pt in 0.7s (in the background; training didn't wait).
09:26:41 AMBCheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.48s.
09:26:41 AMBExpected loss before training ≈ ln(vocab) = 10.83.
09:26:41 AMBTraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).
09:26:30 AMAStep 1200: val 5.571, train 5.525 (gap +0.046).
09:25:10 AMACheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.13s.
09:25:06 AMAStep 1000: val 5.658, train 5.612 (gap +0.046).
09:23:47 AMAStep 800: val 5.783, train 5.787 (gap -0.004).
09:23:22 AMACheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.43s.
09:22:25 AMAStep 600: val 6.003, train 6.015 (gap -0.012).
09:21:41 AMACheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.29s.
09:21:02 AMAStep 400: val 6.295, train 6.223 (gap +0.072).
09:20:00 AMACheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.98s.
09:19:43 AMAStep 200: val 6.635, train 6.612 (gap +0.023).
09:19:00 AMACheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:18:40 AMACheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.61s.
09:18:21 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/d1c5b52f-464f-47aa-b8fa-1431f3970020/step-0000000.pt in 0.7s (in the background; training didn't wait).
09:18:20 AMACheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.48s.
09:18:19 AMAExpected loss before training ≈ ln(vocab) = 10.83.
09:18:19 AMATraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).