Our GPT-2 vs an open model (SmolLM2-135M)

lesson 2.2 · Pretrained vs scratch

Same size, same fine-tuning: what do 2 trillion tokens of pretraining buy over our 10 billion?

base: A ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt vs B hf:HuggingFaceTB/SmolLM2-135Mbase_step: A 19073 vs B undefinedbase_tokens: A 9999745024 vs B undefinedhf_model: A undefined vs B HuggingFaceTB/SmolLM2-135Msteps: A 4251 vs B 4254

Run A

done

null · ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt · 124M params ·

tokens read

35,055,328

step

4,251 / 0

train loss: 2.055
best val: 1.948
tokens/s: 29k
MFU: –
peak mem: 6.1 GB
lr: 3.0e-6

0:23:11 · ≈ $0.12 to rent

Run B

done

null · hf:HuggingFaceTB/SmolLM2-135M · 135M params ·

tokens read

34,963,008

step

4,254 / 0

train loss: 1.730
best val: 1.525
tokens/s: 15k
MFU: –
peak mem: 8.4 GB
lr: 3.0e-6

0:53:12 · ≈ $0.27 to rent

Losslower is better · loss on answer tokens only · starts from the pretrained model

The machine, live

NVIDIA L4 · gcp-l4-ft4offline · never

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

–

starts with training

RAM

–

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

Run A

Queued
done
Download data
already on disk
Tokenize
already on disk
Build and compile
done
Warm up
done
Train
done
Finished
done

Run B

Queued
done
Download data
already on disk
Tokenize
already on disk
Build and compile
done
Warm up
done
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

Waiting for data…

GPU memory (measured)peak per log window · dashed = activations + workspace

Samplessame prompt and random draw every time, so only the model changes

step 4,254

Run A (step 4251)

 Paris.

Run B

 The capital of France is Paris.

What happened

12:21:02 PMBFine-tuning finished: 6.7% → 9.0% of held-out questions. Best val loss on answers 1.525.
12:21:00 PMBStep 4254: 9.0% of 300 held-out questions, 17/20 hand-written; val loss on answers 1.526.
12:16:52 PMBStep 4000: 8.7% of 300 held-out questions, 16/20 hand-written; val loss on answers 1.526.
12:07:00 PMBStep 3160 / (316000%), 26M tokens. Train loss 1.459, best val 1.530. 14622 tok/s; about left.
12:05:20 PMBStep 3000: 8.7% of 300 held-out questions, 16/20 hand-written; val loss on answers 1.530.
11:53:48 AMBStep 2000: 8.0% of 300 held-out questions, 15/20 hand-written; val loss on answers 1.538.
11:42:25 AMBStep 1000: 8.3% of 300 held-out questions, 15/20 hand-written; val loss on answers 1.546.
11:30:58 AMBCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/b69741a6-9169-42d9-bfdd-0d4df1e0354d/step-0000000.pt in 1.3s (in the background; training didn't wait).
11:30:56 AMBStep 0: 6.7% of 300 held-out questions, 17/20 hand-written; val loss on answers 1.854.
11:27:49 AMBFine-tuning hf:HuggingFaceTB/SmolLM2-135M on 'qa' (136,156 examples, 1 epoch(s), 4,254 steps). Trainable parameters: 134,515,008 of 134,515,008 (100.00%).
11:26:53 AMAFine-tuning finished: 3.3% → 7.0% of held-out questions. Best val loss on answers 1.948.
11:26:52 AMAStep 4251: 7.0% of 300 held-out questions, 9/20 hand-written; val loss on answers 1.948.
11:25:18 AMAStep 4000: 7.0% of 300 held-out questions, 9/20 hand-written; val loss on answers 1.948.
11:20:07 AMAStep 3000: 5.7% of 300 held-out questions, 11/20 hand-written; val loss on answers 1.963.
11:14:53 AMAStep 2000: 6.0% of 300 held-out questions, 11/20 hand-written; val loss on answers 1.984.
11:09:41 AMAStep 1000: 5.0% of 300 held-out questions, 11/20 hand-written; val loss on answers 2.014.
11:04:36 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/2e88aa76-43a2-444c-9ffc-6d2e0c4b7cbc/step-0000000.pt in 1.8s (in the background; training didn't wait).
11:04:33 AMAStep 0: 3.3% of 300 held-out questions, 8/20 hand-written; val loss on answers 2.453.
11:03:42 AMAFine-tuning ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt on 'qa' (136,053 examples, 1 epoch(s), 4,251 steps). Trainable parameters: 124,475,904 of 124,475,904 (100.00%).