LoRA vs full fine-tuning (our GPT-2 124M)

lesson 2.3 · LoRA vs full

Train 2% of the weights or all of them: what changes in quality, memory and speed?

lora: A false vs B truelora_config: A undefined vs B [object Object]lr: A 0.00003 vs B 0.0002

Run A

failed

null · ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt · 124M params ·

tokens read

31,036,784

step

3,760 / 0

train loss: 2.072
best val: 1.952
tokens/s: 16k
MFU: –
peak mem: 6.1 GB
lr: 3.9e-6

2d 21h · ≈ $21 to rent

Spot VM gcp-l4-ft7 was preempted by Google at 12:31 UTC (step 3760 of 4251; last checkpoint step 3000).

Run B

done

null · ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt · 127M params ·

tokens read

35,055,328

step

4,251 / 0

train loss: 2.055
best val: 1.966
tokens/s: 32k
MFU: –
peak mem: 4.7 GB
lr: 2.0e-5

0:23:49 · ≈ $0.12 to rent

Losslower is better · loss on answer tokens only · starts from the pretrained model

The machine, live

GPU · gcp-l4-ft7offline · just now

GPU busy

– %

VRAM

– / 24 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

–

starts with training

RAM

–

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

Run A

Queued
done
Download data
already on disk
Tokenize
already on disk
Build and compile
done
Warm up
done
Train
stopped here
Finished
next

Stopped at · Train · Forward, backward, AdamW update, again and again. The learning rate follows a cosine down to a tenth of its peak.

Evaluateevery 250 steps
last 3,750 · next 4,000
loss on held-out text, plus the fact probes (does it know water is a liquid?)
Checkpointevery 1,000 steps
last 3,000 · next 4,000
save weights + optimizer + RNG, upload to Cloud Storage; it appears in the Playground

Run B

Queued
done
Download data
already on disk
Tokenize
already on disk
Build and compile
done
Warm up
done
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

Waiting for data…

GPU memory (measured)peak per log window · dashed = activations + workspace

Samplessame prompt and random draw every time, so only the model changes

step 4,251

Run A (step 3000)

 Paris is the capital of France.

Run B

 Paris.

What happened

06:10:00 PMANo progress for 339 minutes: the last metric was at 12:30 UTC. Check the worker and the GPU.
06:07:00 PMAStep 3760 / (376000%), 31M tokens. Train loss 2.072, best val 1.952. 16163 tok/s; about left.
05:10:00 PMANo progress for 279 minutes: the last metric was at 12:30 UTC. Check the worker and the GPU.
04:10:00 PMANo progress for 219 minutes: the last metric was at 12:30 UTC. Check the worker and the GPU.
04:07:00 PMAStep 3760 / (376000%), 31M tokens. Train loss 2.072, best val 1.952. 16163 tok/s; about left.
03:00:00 PMANo progress for 149 minutes: the last metric was at 12:30 UTC. Check the worker and the GPU.
02:07:00 PMAStep 3760 / (376000%), 31M tokens. Train loss 2.072, best val 1.952. 16163 tok/s; about left.
02:00:00 PMANo progress for 89 minutes: the last metric was at 12:30 UTC. Check the worker and the GPU.
12:50:00 PMANo progress for 19 minutes: the last metric was at 12:30 UTC. Check the worker and the GPU.
12:35:26 PMBFine-tuning finished: 3.3% → 5.3% of held-out questions. Best val loss on answers 1.966.
12:35:24 PMBStep 4251: 5.3% of 300 held-out questions, 10/20 hand-written; val loss on answers 1.966.
12:33:39 PMBStep 4000: 5.3% of 300 held-out questions, 11/20 hand-written; val loss on answers 1.967.
12:28:32 PMBStep 3000: 5.3% of 300 held-out questions, 11/20 hand-written; val loss on answers 1.977.
12:27:07 PMAStep 3000: 6.7% of 300 held-out questions, 10/20 hand-written; val loss on answers 1.964.
12:23:20 PMBStep 2000: 5.3% of 300 held-out questions, 11/20 hand-written; val loss on answers 1.993.
12:21:49 PMAStep 2000: 5.3% of 300 held-out questions, 11/20 hand-written; val loss on answers 1.983.
12:18:12 PMBStep 1000: 4.7% of 300 held-out questions, 11/20 hand-written; val loss on answers 2.023.
12:16:31 PMAStep 1000: 5.3% of 300 held-out questions, 11/20 hand-written; val loss on answers 2.013.
12:13:15 PMBCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/65f45660-5eb8-4bf9-829b-7a039f36a563/step-0000000.pt in 4.8s (in the background; training didn't wait).
12:13:08 PMBStep 0: 3.3% of 300 held-out questions, 8/20 hand-written; val loss on answers 2.453.
12:11:37 PMBFine-tuning ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt with LoRA on 'qa' (136,053 examples, 1 epoch(s), 4,251 steps). Trainable parameters: 2,359,296 of 126,835,200 (1.86%).
12:11:22 PMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/279de5a6-5c3e-4bf7-ae21-539137a3c7b0/step-0000000.pt in 1.7s (in the background; training didn't wait).
12:11:19 PMAStep 0: 3.3% of 300 held-out questions, 8/20 hand-written; val loss on answers 2.453.
12:10:27 PMAFine-tuning ours:gs://train-galvyn-xyz-checkpoints/runs/1a0cd0c6-df9f-471e-b9fa-096a419cccea/step-0019073.pt on 'qa' (136,053 examples, 1 epoch(s), 4,251 steps). Trainable parameters: 124,475,904 of 124,475,904 (100.00%).