Checkpoint often vs rarely

lesson 1.7 · Checkpointing

How much work does a crash cost, and does a resume really continue where it left off?

ckpt_every: A 100 vs B 1000

Run A

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

24,576,000

step

3,000 / 3,000

train loss: 4.869
best val: 5.094
tokens/s: 33k
MFU: 5.5%
peak mem: 6.4 GB
lr: 1.0e-4

0:08:07 · 0.020 kWh ≈ <$0.01 power · ≈ $0.04 to rent

Run B

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

24,576,000

step

3,000 / 3,000

train loss: 4.869
best val: 5.094
tokens/s: 74k
MFU: 12.4%
peak mem: 6.4 GB
lr: 1.0e-4

0:08:17 · 0.018 kWh ≈ <$0.01 power · ≈ $0.04 to rent

Losslower is better · starts near ln(vocab)

The machine, live

NVIDIA L4 · gcp-l4-laboffline · just now

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

0.037 kWh

≈ <$0.01 of electricity

RAM

–

of 31.3 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

What to watch · Press Crash test on both runs after step 1000. A loses at most 100 steps, B up to 1000. The dashed curve after each resume should sit on the original; the Resume check measures how closely.

Run A

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

Run B

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

GPU memory (measured)peak per log window · dashed = activations + workspace

Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right

The questions

arithmetic	A 0.39	B 0.39
ice	A 0.57	B 0.57
ocean	A 0.66	B 0.66
orbit	A 0.59	B 0.59
paris	A 0.38	B 0.38
planet	A 0.33	B 0.33
sky	A 0.72	B 0.72
water	A 0.54	B 0.54
whale	A 0.16	B 0.16

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

step 3,000

Run A

Water is the most important part of the following:
The area of the body is the main source of water, the body is the main form of water and is a few of the body. The body has the largest side of the body. The body is the first part of the body, and the body is the most frequent form of water. The body is the result of the body, the body is the main function of the body.
The first phase is called the heart.
The body uses the liver to develop a body of the muscles. It is called a muscles that are very soft and is the blood. The body becomes an abnormal body, with the blood. This cell has a blood vessels, body, and the heart muscles.
There is a three types of blood, usually called a blood vessels, where the body is more intense and less slowly.
The body is a hormone called the stomach and is the primary form of blood vessels, the body’s body. It

Run B

Water is the most important part of the following:
The area of the body is the main source of water, the body is the main form of water and is a few of the body. The body has the largest side of the body. The body is the first part of the body, and the body is the most frequent form of water. The body is the result of the body, the body is the main function of the body.
The first phase is called the heart.
The body uses the liver to develop a body of the muscles. It is called a muscles that are very soft and is the blood. The body becomes an abnormal body, with the blood. This cell has a blood vessels, body, and the heart muscles.
There is a three types of blood, usually called a blood vessels, where the body is more intense and less slowly.
The body is a hormone called the stomach and is the primary form of blood vessels, the body’s body. It

Resume check

After a crash, the steps since the last checkpoint are lost and recomputed. If the checkpoint restored everything (weights, AdamW state, schedule, the RNG that picks batches), the recomputed losses match the originals.

A · resume 1 steps 1310–1380 (8 points): max |Δloss| 0.0e+0, mean 0.0e+0Bit-identical: the resume reproduced the lost steps exactly.
B · resume 1 steps 1010–1490 (49 points): max |Δloss| 0.0e+0, mean 0.0e+0Bit-identical: the resume reproduced the lost steps exactly.

What happened

09:49:18 AMBFinished 3000 steps. Best val loss 5.094.
09:49:18 AMBCheckpoint at step 3000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:49:16 AMBStep 3000: val 5.094, train 5.086 (gap +0.008).
09:49:08 AMAFinished 3000 steps. Best val loss 5.094.
09:49:08 AMACheckpoint at step 3000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
09:49:05 AMAStep 3000: val 5.094, train 5.086 (gap +0.008).
09:48:36 AMACheckpoint at step 2900: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
09:48:35 AMBStep 2800: val 5.094, train 5.082 (gap +0.012).
09:48:10 AMACheckpoint at step 2800: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:48:08 AMAStep 2800: val 5.094, train 5.082 (gap +0.012).
09:47:44 AMBStep 2600: val 5.114, train 5.116 (gap -0.002).
09:47:39 AMACheckpoint at step 2700: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
09:47:13 AMACheckpoint at step 2600: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:47:12 AMAStep 2600: val 5.114, train 5.116 (gap -0.002).
09:46:54 AMBStep 2400: val 5.163, train 5.143 (gap +0.020).
09:46:43 AMACheckpoint at step 2500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
09:46:15 AMACheckpoint at step 2400: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
09:46:13 AMAStep 2400: val 5.163, train 5.143 (gap +0.020).
09:46:05 AMBStep 2200: val 5.192, train 5.167 (gap +0.025).
09:45:45 AMACheckpoint at step 2300: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
09:45:18 AMACheckpoint at step 2200: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.71s.
09:45:17 AMBCheckpoint step 2000 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/daba75b6-dca7-488f-b099-f04cf024b841/step-0002000.pt in 1.3s (in the background; training didn't wait).
09:45:16 AMAStep 2200: val 5.192, train 5.167 (gap +0.025).
09:45:16 AMBCheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:45:13 AMBStep 2000: val 5.234, train 5.252 (gap -0.018).
09:44:50 AMACheckpoint at step 2100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:44:23 AMACheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:44:22 AMBStep 1800: val 5.328, train 5.269 (gap +0.059).
09:44:20 AMAStep 2000: val 5.234, train 5.252 (gap -0.018).
09:43:51 AMACheckpoint at step 1900: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:43:32 AMBStep 1600: val 5.429, train 5.361 (gap +0.068).
09:43:25 AMACheckpoint at step 1800: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
09:43:23 AMAStep 1800: val 5.328, train 5.269 (gap +0.059).
09:42:55 AMACheckpoint at step 1700: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
09:42:41 AMBStep 1400: val 5.475, train 5.424 (gap +0.051).
09:42:29 AMACheckpoint at step 1600: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
09:42:27 AMAStep 1600: val 5.429, train 5.361 (gap +0.068).
09:41:58 AMACheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.52s.
09:41:53 AMBStep 1200: val 5.612, train 5.570 (gap +0.042).
09:41:32 AMACheckpoint step 1400 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/61898328-5722-4fc9-8849-750c0da90d85/step-0001400.pt in 1.4s (in the background; training didn't wait).
09:41:30 AMACheckpoint at step 1400: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
09:41:29 AMAStep 1400: val 5.475, train 5.424 (gap +0.051).
09:41:01 AMBResumed at step 1000 from a checkpoint.
09:41:00 AMAResumed at step 1300 from a checkpoint.
09:40:56 AMBCrash test: killed the process with SIGKILL (no chance to save). Resuming from the checkpoint at step 1000.
09:40:55 AMACrash test: killed the process with SIGKILL (no chance to save). Resuming from the checkpoint at step 1300.
09:40:33 AMACheckpoint at step 1300: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.60s.
09:40:32 AMBStep 1400: val 5.475, train 5.424 (gap +0.051).
09:40:06 AMACheckpoint at step 1200: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
09:40:04 AMAStep 1200: val 5.612, train 5.570 (gap +0.042).
09:39:42 AMBStep 1200: val 5.612, train 5.570 (gap +0.042).
09:39:36 AMACheckpoint at step 1100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.57s.
09:39:09 AMACheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.94s.
09:39:06 AMAStep 1000: val 5.685, train 5.638 (gap +0.047).
09:38:49 AMBCheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.61s.
09:38:46 AMBStep 1000: val 5.685, train 5.638 (gap +0.047).
09:38:24 AMACheckpoint at step 900: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.57s.
09:37:45 AMACheckpoint at step 800: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:37:43 AMAStep 800: val 5.786, train 5.790 (gap -0.004).
09:37:27 AMBStep 800: val 5.786, train 5.790 (gap -0.004).
09:37:01 AMACheckpoint at step 700: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.62s.
09:36:22 AMACheckpoint at step 600: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:36:20 AMAStep 600: val 5.981, train 5.996 (gap -0.015).
09:36:09 AMBStep 600: val 5.981, train 5.996 (gap -0.015).
09:35:38 AMACheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:34:58 AMACheckpoint at step 400: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:34:56 AMAStep 400: val 6.270, train 6.198 (gap +0.072).
09:34:51 AMBStep 400: val 6.270, train 6.198 (gap +0.072).
09:34:13 AMACheckpoint at step 300: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.57s.
09:33:53 AMACheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.76s.
09:33:51 AMBCheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.56s.
09:33:34 AMACheckpoint at step 200: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.59s.
09:33:32 AMBStep 200: val 6.571, train 6.549 (gap +0.022).
09:33:32 AMAStep 200: val 6.571, train 6.549 (gap +0.022).
09:32:49 AMBCheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.43s.
09:32:49 AMACheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.57s.
09:32:28 AMACheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.58s.
09:32:28 AMBCheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.47s.
09:32:07 AMBCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/daba75b6-dca7-488f-b099-f04cf024b841/step-0000000.pt in 0.8s (in the background; training didn't wait).
09:32:07 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/61898328-5722-4fc9-8849-750c0da90d85/step-0000000.pt in 0.8s (in the background; training didn't wait).
09:32:06 AMBCheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.83s.
09:32:06 AMACheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.83s.
09:32:05 AMBExpected loss before training ≈ ln(vocab) = 10.83.
09:32:05 AMAExpected loss before training ≈ ln(vocab) = 10.83.
09:32:05 AMBTraining 30.06M params on fineweb_edu for 3000 steps (bf16, batch 32×1, ctx 256).
09:32:05 AMATraining 30.06M params on fineweb_edu for 3000 steps (bf16, batch 32×1, ctx 256).