General vs narrow fine-tuning data

lesson 2.4 · Data matters

Fine-tune on general instructions or on one narrow task: how does each do on both kinds of question?

mix: A alpaca vs B nqsteps: A 1558 vs B 2692

Run A

done

null · ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt · 124M params ·

tokens read

18,136,392

step

1,558 / 0

train loss: 1.827
best val: 1.885
tokens/s: 31k
MFU: –
peak mem: 6.1 GB
lr: 3.0e-6

0:15:48 · ≈ $0.08 to rent

Run B

done

null · ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt · 124M params ·

tokens read

2,154,640

step

2,692 / 0

train loss: 2.342
best val: 2.464
tokens/s: 5979
MFU: –
peak mem: 2.5 GB
lr: 3.0e-6

0:08:42 · ≈ $0.04 to rent

Losslower is better · loss on answer tokens only · starts from the pretrained model

The machine, live

NVIDIA L4 · gcp-l4-ft2offline · never

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

–

starts with training

RAM

–

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

Run A

Queued
done
Download data
already on disk
Tokenize
already on disk
Build and compile
done
Warm up
done
Train
done
Finished
done

Run B

Queued
done
Download data
already on disk
Tokenize
already on disk
Build and compile
done
Warm up
done
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

Waiting for data…

GPU memory (measured)peak per log window · dashed = activations + workspace

Samplessame prompt and random draw every time, so only the model changes

step 2,692

Run A (step 1558)

 The capital of France is Paris.

Run B

 Paris.

What happened

11:28:25 AMBFine-tuning finished: 3.3% → 8.0% of held-out questions. Best val loss on answers 2.464.
11:28:23 AMBStep 2692: 8.0% of 300 held-out questions, 8/20 hand-written; val loss on answers 2.464.
11:27:42 AMBStep 2500: 8.7% of 300 held-out questions, 8/20 hand-written; val loss on answers 2.468.
11:26:16 AMBStep 2000: 8.7% of 300 held-out questions, 7/20 hand-written; val loss on answers 2.506.
11:24:51 AMBStep 1500: 7.3% of 300 held-out questions, 7/20 hand-written; val loss on answers 2.561.
11:23:26 AMBStep 1000: 5.7% of 300 held-out questions, 7/20 hand-written; val loss on answers 2.632.
11:22:02 AMBStep 500: 6.3% of 300 held-out questions, 6/20 hand-written; val loss on answers 2.735.
11:20:36 AMBCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/5197a448-829f-44d3-86c0-a0bb57dd84b9/step-0000000.pt in 1.7s (in the background; training didn't wait).
11:20:34 AMBStep 0: 3.3% of 300 held-out questions, 8/20 hand-written; val loss on answers 4.644.
11:19:42 AMBFine-tuning ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt on 'nq' (86,167 examples, 1 epoch(s), 2,692 steps). Trainable parameters: 124,475,904 of 124,475,904 (100.00%).
11:19:29 AMAFine-tuning finished: 3.3% → 3.3% of held-out questions. Best val loss on answers 1.885.
11:19:28 AMAStep 1558: 3.3% of 300 held-out questions, 12/20 hand-written; val loss on answers 1.885.
11:17:55 AMAStep 1500: 3.7% of 300 held-out questions, 11/20 hand-written; val loss on answers 1.885.
11:13:32 AMAStep 1000: 4.0% of 300 held-out questions, 13/20 hand-written; val loss on answers 1.904.
11:09:04 AMAStep 500: 3.3% of 300 held-out questions, 13/20 hand-written; val loss on answers 1.944.
11:04:36 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/9cba1796-db07-42d1-bbdf-92d0754c55a7/step-0000000.pt in 1.8s (in the background; training didn't wait).
11:04:33 AMAStep 0: 3.3% of 300 held-out questions, 8/20 hand-written; val loss on answers 2.320.
11:03:41 AMAFine-tuning ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt on 'alpaca' (49,886 examples, 1 epoch(s), 1,558 steps). Trainable parameters: 124,475,904 of 124,475,904 (100.00%).