Overfitting: 1 vs 5 epochs

lesson 2.5 · Overfitting

Train on a small dataset once or five times: when does more training start to hurt?

epochs: A 1 vs B 5steps: A 150 vs B 750

Run A

done

null · ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt · 124M params ·

tokens read

120,400

step

150 / 0

train loss: 2.570
best val: 2.742
tokens/s: 5837
MFU: –
peak mem: 2.4 GB
lr: 3.0e-6

0:01:34 · ≈ <$0.01 to rent

Run B

done

null · ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt · 124M params ·

tokens read

601,856

step

750 / 0

train loss: 0.519
best val: 2.759
tokens/s: 6078
MFU: –
peak mem: 2.4 GB
lr: 3.0e-6

0:04:03 · ≈ $0.02 to rent

Losslower is better · loss on answer tokens only · starts from the pretrained model

The machine, live

NVIDIA L4 · gcp-l4-ft1offline · never

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

–

starts with training

RAM

–

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

Run A

Queued
done
Download data
already on disk
Tokenize
already on disk
Build and compile
done
Warm up
done
Train
done
Finished
done

Run B

Queued
done
Download data
already on disk
Tokenize
already on disk
Build and compile
done
Warm up
done
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

Waiting for data…

GPU memory (measured)peak per log window · dashed = activations + workspace

Samplessame prompt and random draw every time, so only the model changes

step 750

Run A (step 150)

 Paris.

Run B

 Paris.

What happened

11:10:56 AMBFine-tuning finished: 3.3% → 6.0% of held-out questions. Best val loss on answers 2.759.
11:10:54 AMBStep 750: 6.0% of 300 held-out questions, 7/20 hand-written; val loss on answers 3.203.
11:10:16 AMBStep 600: 8.3% of 300 held-out questions, 7/20 hand-written; val loss on answers 3.097.
11:09:38 AMBStep 450: 7.7% of 300 held-out questions, 9/20 hand-written; val loss on answers 2.951.
11:09:00 AMBStep 300: 6.0% of 300 held-out questions, 9/20 hand-written; val loss on answers 2.822.
11:08:22 AMBStep 150: 6.0% of 300 held-out questions, 11/20 hand-written; val loss on answers 2.759.
11:07:47 AMBCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/39fbba64-7fdd-467c-8af8-c0909a27fa6e/step-0000000.pt in 5.0s (in the background; training didn't wait).
11:07:41 AMBStep 0: 3.3% of 300 held-out questions, 8/20 hand-written; val loss on answers 4.700.
11:06:52 AMBFine-tuning ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt on 'nq5k' (4,800 examples, 5 epoch(s), 750 steps). Trainable parameters: 124,475,904 of 124,475,904 (100.00%).
11:06:39 AMAFine-tuning finished: 3.3% → 5.3% of held-out questions. Best val loss on answers 2.742.
11:06:38 AMAStep 150: 5.3% of 300 held-out questions, 10/20 hand-written; val loss on answers 2.742.
11:06:02 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/bac7a04d-bb37-4f2c-8622-684564a205e1/step-0000000.pt in 4.6s (in the background; training didn't wait).
11:05:56 AMAStep 0: 3.3% of 300 held-out questions, 8/20 hand-written; val loss on answers 4.700.
11:05:04 AMAFine-tuning ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt on 'nq5k' (4,800 examples, 1 epoch(s), 150 steps). Trainable parameters: 124,475,904 of 124,475,904 (100.00%).