Overfit one batch

lesson 1.1 · Overfit one batch

Can the model memorise a single batch? (If it can't, the training loop is broken.)

overfit_batch: A true vs B false

Run A

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

4,096,000

step

500 / 500

train loss: 0.000
best val: 8.841
tokens/s: 22k
MFU: 3.7%
peak mem: 6.4 GB
lr: 3.0e-4

0:05:09 · 0.006 kWh ≈ <$0.01 power · ≈ $0.03 to rent

Run B

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

4,096,000

step

500 / 500

train loss: 6.057
best val: 6.081
tokens/s: 22k
MFU: 3.6%
peak mem: 6.4 GB
lr: 3.0e-4

0:05:09 · 0.006 kWh ≈ <$0.01 power · ≈ $0.03 to rent

Losslower is better · starts near ln(vocab)

The machine, live

NVIDIA L4 · gcp-l4-laboffline · just now

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

0.012 kWh

≈ <$0.01 of electricity

RAM

–

of 31.3 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

What to watch · A's train loss should head to near zero: the first sanity check of any training loop. Its val loss should rise at the same time, because memorising one batch teaches nothing general. B trains normally for comparison.

Run A

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

Run B

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

GPU memory (measured)peak per log window · dashed = activations + workspace

Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right

The questions

arithmetic	A 0.27	B 0.51
ice	A 0.14	B 0.92
ocean	A 0.71	B 0.68
orbit	A 1.00	B 0.78
paris	A 0.03	B 0.35
planet	A 0.35	B 0.25
sky	A 0.27	B 0.54
water	A 0.87	B 0.09
whale	A 0.35	B 0.00

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

step 500

Run A

Water is with it� air all away for our environment.
People can make it to within his finger seven times before the most important political management of the postane to eat.
In sediment.
The study also found that state, and entrails; and they threw cedar-wood, and hyssop, Research, Table Mountain
In March 2015 the Peninsula burnt. The biggest veld fire since 2000 raged across Table Mountain National Park. People lost homes and businesses. Bees Marais, one of the country’s top helicopter rescue pilots, tragically lost his life in the line of duty while fighting fire at Cape Point. The blaze and the acrid smoke cloyed the air all over Cape Town, turning the sky scarlet and orange as the sun went down each day.
by Zoë Poulsen - Cape Town Botanist | Mar 3, 2015 | Cape Floristic Region, Cape Peninsula, Conservation, Ecology, Fire, Fynbos, Geophytes

Run B

Water is the other factors called a few people. This can need to the body for the main system that can develop the health system.
The case of any kind of the results of the first place of the number of the body can result of a particular way or another. For example, the most important to be the condition that the risk of the development of the body of the problem of your baby's.
There are to get in the patient is an individual is that of the condition.
This is in which can be a lot of the risk of symptoms of a simple. If you can be aware of your symptoms that your body.
You can need to be done.
Here has a child has been a child to do the same time? You may be in a result in your child with a lot of these symptoms.
What is the most of the brain is to treat the children.
This is a good way to have been able to be the children.
In this is the

What happened

09:04:35 AMBFinished 500 steps. Best val loss 6.081.
09:04:35 AMAFinished 500 steps. Best val loss 8.841.
09:04:35 AMACheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.49s.
09:04:35 AMBCheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.58s.
09:04:31 AMAStep 500: val 13.352, train 0.000 (gap +13.351).
09:04:31 AMBStep 500: val 6.081, train 6.036 (gap +0.045).
09:04:07 AMAStep 450: val 13.096, train 0.000 (gap +13.096).
09:04:06 AMBStep 450: val 6.116, train 6.082 (gap +0.034).
09:03:43 AMAStep 400: val 12.855, train 0.001 (gap +12.854).
09:03:43 AMBStep 400: val 6.211, train 6.140 (gap +0.071).
09:03:13 AMAStep 350: val 12.269, train 0.001 (gap +12.267).
09:03:13 AMBStep 350: val 6.277, train 6.257 (gap +0.020).
09:02:41 AMAStep 300: val 14.136, train 0.000 (gap +14.135).
09:02:41 AMBStep 300: val 6.372, train 6.316 (gap +0.056).
09:02:10 AMACheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.55s.
09:02:10 AMBCheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.62s.
09:02:07 AMAStep 250: val 13.636, train 0.000 (gap +13.636).
09:02:07 AMBStep 250: val 6.512, train 6.408 (gap +0.103).
09:01:36 AMAStep 200: val 12.982, train 0.000 (gap +12.982).
09:01:36 AMBStep 200: val 6.599, train 6.578 (gap +0.021).
09:01:05 AMAStep 150: val 12.603, train 0.001 (gap +12.602).
09:01:05 AMBStep 150: val 6.728, train 6.700 (gap +0.028).
09:00:34 AMACheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.41s.
09:00:34 AMBCheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.59s.
09:00:31 AMAStep 100: val 11.832, train 0.001 (gap +11.831).
09:00:31 AMBStep 100: val 6.950, train 6.936 (gap +0.014).
09:00:01 AMBCheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.61s.
09:00:01 AMACheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.58s.
08:59:58 AMAStep 50: val 8.841, train 2.478 (gap +6.363).
08:59:58 AMBStep 50: val 7.304, train 7.275 (gap +0.029).
08:59:29 AMBCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/d5a01448-3e4a-4462-8100-3bfb2cd4a236/step-0000000.pt in 1.1s (in the background; training didn't wait).
08:59:29 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/d15c2887-11b4-48c9-8407-ee65790c0fbd/step-0000000.pt in 1.1s (in the background; training didn't wait).
08:59:28 AMACheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 1.46s.
08:59:28 AMBCheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
08:59:26 AMAOverfit-one-batch is on: every step sees the same batch. Train loss should go to ~0 while val gets worse.
08:59:26 AMBExpected loss before training ≈ ln(vocab) = 10.83.
08:59:26 AMAExpected loss before training ≈ ln(vocab) = 10.83.
08:59:26 AMBTraining 30.06M params on fineweb_edu for 500 steps (bf16, batch 32×1, ctx 256).
08:59:26 AMATraining 30.06M params on fineweb_edu for 500 steps (bf16, batch 32×1, ctx 256).