Pretrained vs scratch

lesson 2.2 · Pretrained vs scratch

How much of answering questions comes from pretraining? Same fine-tuning on our pretrained GPT-2 vs the same model from random weights.

base: A ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt vs B scratchbase_step: A 19073 vs B undefinedbase_tokens: A 9999745024 vs B undefined

Run A

done

null · ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt · 124M params ·

tokens read

35,055,328

step

4,251 / 0

train loss: 2.054
best val: 1.948
tokens/s: 29k
MFU: –
peak mem: 6.1 GB
lr: 3.0e-6

0:23:32 · ≈ $0.12 to rent

Run B

done

null · scratch · 124M params ·

tokens read

35,055,328

step

4,251 / 0

train loss: 5.206
best val: 5.188
tokens/s: 29k
MFU: –
peak mem: 6.1 GB
lr: 3.0e-6

0:24:38 · ≈ $0.12 to rent

Losslower is better · loss on answer tokens only · starts from the pretrained model

The machine, live

NVIDIA L4 · gcp-l4-ft3offline · never

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

–

starts with training

RAM

–

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

Run A

Queued
done
Download data
already on disk
Tokenize
already on disk
Build and compile
done
Warm up
done
Train
done
Finished
done

Run B

Queued
done
Download data
already on disk
Tokenize
already on disk
Build and compile
done
Warm up
done
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

Waiting for data…

GPU memory (measured)peak per log window · dashed = activations + workspace

Samplessame prompt and random draw every time, so only the model changes

step 4,251

Run A

 Paris.

Run B

 The first step in the United States is the United States is the United States of the United States. The United States is the United States of the United States,

What happened

11:53:47 AMBFine-tuning finished: 0.0% → 0.7% of held-out questions. Best val loss on answers 5.188.
11:53:46 AMBStep 4251: 0.7% of 300 held-out questions, 0/20 hand-written; val loss on answers 5.188.
11:52:07 AMBStep 4000: 0.3% of 300 held-out questions, 0/20 hand-written; val loss on answers 5.201.
11:46:44 AMBStep 3000: 0.7% of 300 held-out questions, 0/20 hand-written; val loss on answers 5.287.
11:41:25 AMBStep 2000: 0.3% of 300 held-out questions, 0/20 hand-written; val loss on answers 5.464.
11:36:07 AMBStep 1000: 0.7% of 300 held-out questions, 0/20 hand-written; val loss on answers 5.827.
11:30:46 AMBCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/08c844ae-62df-4a8c-ba1b-86eca59693d5/step-0000000.pt in 4.4s (in the background; training didn't wait).
11:30:40 AMBStep 0: 0.0% of 300 held-out questions, 0/20 hand-written; val loss on answers 11.003.
11:29:09 AMBFine-tuning scratch on 'qa' (136,053 examples, 1 epoch(s), 4,251 steps). Trainable parameters: 124,475,904 of 124,475,904 (100.00%).
11:28:57 AMAFine-tuning finished: 3.3% → 6.7% of held-out questions. Best val loss on answers 1.948.
11:28:55 AMAStep 4251: 6.7% of 300 held-out questions, 11/20 hand-written; val loss on answers 1.948.
11:27:20 AMAStep 4000: 7.0% of 300 held-out questions, 10/20 hand-written; val loss on answers 1.948.
11:22:05 AMAStep 3000: 6.0% of 300 held-out questions, 10/20 hand-written; val loss on answers 1.964.
11:16:46 AMAStep 2000: 5.3% of 300 held-out questions, 11/20 hand-written; val loss on answers 1.983.
11:11:28 AMAStep 1000: 5.0% of 300 held-out questions, 12/20 hand-written; val loss on answers 2.013.
11:06:21 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/52ccf94e-dadb-4ed4-b9e7-f1d26658dfdd/step-0000000.pt in 4.6s (in the background; training didn't wait).
11:06:15 AMAStep 0: 3.3% of 300 held-out questions, 8/20 hand-written; val loss on answers 2.453.
11:05:24 AMAFine-tuning ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt on 'qa' (136,053 examples, 1 epoch(s), 4,251 steps). Trainable parameters: 124,475,904 of 124,475,904 (100.00%).