Phase 2: teach GPT-2 124M to answer questions (supervised fine-tuning)

lesson 2.1 · Teach it to answer

Pretraining taught it facts but not to answer and stop. What changes after ~140k question → answer examples?

Run A

done

fineweb_edu_10b · 12L·768d · 124M params ·

tokens read

37,678,816

step

4,320 / 4,300

train loss: 1.920
best val: 1.975
tokens/s: 30k
MFU: –
peak mem: –
lr: 3.0e-6

0:22:35 · ≈ $0.11 to rent

Losslower is better · loss on answer tokens only · starts from the pretrained model

The machine, live

NVIDIA L4 · gcp-l4-sftoffline · just now

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

–

starts with training

RAM

–

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

Queued
done
Download data
already on disk
Tokenize
already on disk
Build and compile
done
Warm up
skipped
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

Waiting for data…

GPU memory (measured)peak per log window · dashed = activations + workspace

Waiting for data…

Samplessame prompt and random draw every time, so only the model changes

step 4,320

Run A

 The capital of France is Paris.

What happened

09:17:27 AMARe-graded after the run: 11/20, not 12/20. "Boiling point of water in Celsius" was answered "approximately 32.8°C (100.6°F)" and the grader accepted it because the text contains "100". The grader now matches whole words and numbers.
09:15:32 AMASFT finished: 7/20 → 12/20 questions correct. Best val loss on answers 1.975.
09:15:29 AMAStep 4320: 12/20 correct; val loss on answers 1.975.
09:13:47 AMAStep 4000: 13/20 correct; val loss on answers 1.977.
09:11:10 AMAStep 3500: 10/20 correct; val loss on answers 1.981.
09:08:33 AMAStep 3000: 14/20 correct; val loss on answers 1.987.
09:05:58 AMAStep 2500: 14/20 correct; val loss on answers 1.998.
09:03:21 AMAStep 2000: 12/20 correct; val loss on answers 2.007.
09:00:42 AMAStep 1500: 14/20 correct; val loss on answers 2.017.
08:58:06 AMAStep 1000: 11/20 correct; val loss on answers 2.038.
08:55:30 AMAStep 500: 10/20 correct; val loss on answers 2.066.
08:53:01 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/18b4f4d5-949c-4699-9246-8667a73dd218/step-0000000.pt in 1.7s (in the background; training didn't wait).
08:52:57 AMABefore fine-tuning: 7/20 questions answered correctly (greedy). Val loss on answers 2.453.
08:52:57 AMASFT of the base model (step 19,073, 10.0B tokens) on 138,268 Q&A examples for 4,320 steps.
08:51:55 AMASFT crashed: OutOfMemoryError: CUDA out of memory. Tried to allocate 2.96 GiB. GPU 0 has a total capacity of 22.03 GiB of which 1.46 GiB is free. Including non-PyTorch memory, this process has 20.56 GiB memory in use. Of the allocated memory 14.47 GiB is allocated by PyTorch, and 5.85 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management
08:50:27 AMASFT crashed: OutOfMemoryError: CUDA out of memory. Tried to allocate 608.00 MiB. GPU 0 has a total capacity of 22.03 GiB of which 52.00 MiB is free. Process 1805 has 20.61 GiB memory in use. Including non-PyTorch memory, this process has 1.36 GiB memory in use. Of the allocated memory 1.01 GiB is allocated by PyTorch, and 124.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentati
08:49:39 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/18b4f4d5-949c-4699-9246-8667a73dd218/step-0000000.pt in 1.7s (in the background; training didn't wait).
08:49:36 AMABefore fine-tuning: 7/20 questions answered correctly (greedy). Val loss on answers 2.453.
08:49:36 AMASFT of the base model (step 19,073, 10.0B tokens) on 138,268 Q&A examples for 4,320 steps.