Small vs GPT-2-sized model

lesson 1.9 · Scale

What does 30× the transformer buy on the same web text?

compile: A false vs B trued: A 256 vs B 768heads: A 4 vs B 12layers: A 4 vs B 12

Run A

done

fineweb_edu · 4L·256d · 16M params · bf16

tokens read

24,576,000

step

3,000 / 3,000

train loss: 5.082
best val: 5.282
tokens/s: 39k
MFU: 3.5%
peak mem: 5.9 GB
lr: 1.0e-4

0:14:52 · 0.018 kWh ≈ <$0.01 power · ≈ $0.07 to rent

Run B

done

fineweb_edu · 12L·768d · 124M params · bf16

tokens read

24,576,000

step

3,000 / 3,000

train loss: 4.828
best val: 5.055
tokens/s: 19k
MFU: 13.0%
peak mem: 6.5 GB
lr: 1.0e-4

0:28:23 · 0.034 kWh ≈ <$0.01 power · ≈ $0.14 to rent

Losslower is better · starts near ln(vocab)

The machine, live

NVIDIA L4 · gcp-l4-lab2offline · just now

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

0.051 kWh

≈ <$0.01 of electricity

RAM

–

of 31.3 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

What to watch · Same data and steps; B is GPT-2 small's shape. Expect B's loss to stay below A's at every step, and B to use the GPU far better (MFU) even though each of its steps costs more.

Run A

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

Run B

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

GPU memory (measured)peak per log window · dashed = activations + workspace

Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right

The questions

arithmetic	A 0.53	B 0.42
ice	A 0.57	B 0.50
ocean	A 0.52	B 0.69
orbit	A 0.67	B 0.59
paris	A 0.26	B 0.32
planet	A 0.17	B 0.19
sky	A 0.54	B 0.64
water	A 0.35	B 0.44
whale	A 0.07	B 0.15

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

step 3,000

Run A

Water is the most important part of the water. This can be used in the water. The water can also develop the water in the plant which will affect the soil. This can vary under temperature and will impact the plants. It can result in a particular area or location. For example, the water can be used to be performed by the water in order to help with the soil.
Once the water is the water is to get the water that is done, the water will increase your water.
How does the water?
To the soil it take more nutrients to reduce the nitrogen flow. It can be recommended if the moisture pressure becomes better to make sure water is done. This is important if water is good enough, it is necessary to give a chance of dry in a container and not getting a good water. If the water is ready to provide water, it can also be required for a water water peroxide (see below below). In the other hand, it will be very great if the

Run B

Water is the most harmful to the future.
The importance of reducing the global carbon emissions in China is the first step of the COVID-19 pandemic is a country-long new country in the state.
In the next hand, a global global warming program is an increasing source of 2.5 million people to produce 1.4 million people. This is an important problem in the global economy.
There are also other things that are needed to make this problem.
In the past, there are some other types of things that can be done with people who would like to start with electricity. The first thing that the United States would be more likely is to be more serious than the world, it is quite a simple question.
We have to know how much to make it more energy, we will not have to make it a difference in what energy is.
But in the first time we will have a lot of energy but have a lot of ideas about it.
What is the

What happened

10:14:04 AMBFinished 3000 steps. Best val loss 5.055.
10:14:04 AMBCheckpoint at step 3000: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 6.20s.
10:13:56 AMBStep 3000: val 5.055, train 5.047 (gap +0.009).
10:12:26 AMBStep 2800: val 5.056, train 5.049 (gap +0.007).
10:11:59 AMBCheckpoint at step 2750: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 6.63s.
10:10:48 AMBStep 2600: val 5.083, train 5.088 (gap -0.005).
10:09:59 AMBCheckpoint at step 2500: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 6.12s.
10:09:10 AMBStep 2400: val 5.142, train 5.124 (gap +0.018).
10:07:59 AMBCheckpoint at step 2250: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 6.07s.
10:07:31 AMBStep 2200: val 5.173, train 5.153 (gap +0.019).
10:07:00 AMBStep 2130 / 3000 (71%), 17M tokens. Train loss 5.381, best val 5.236, fact probes 0.46. 18783 tok/s; about 00d 00h 08m left.
10:06:00 AMBCheckpoint at step 2000: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 6.34s.
10:05:52 AMBStep 2000: val 5.236, train 5.252 (gap -0.016).
10:04:22 AMBStep 1800: val 5.338, train 5.276 (gap +0.061).
10:03:55 AMBCheckpoint at step 1750: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 5.83s.
10:02:45 AMBStep 1600: val 5.456, train 5.383 (gap +0.072).
10:01:56 AMBCheckpoint at step 1500: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 6.57s.
10:01:06 AMBStep 1400: val 5.497, train 5.445 (gap +0.053).
10:00:26 AMAFinished 3000 steps. Best val loss 5.282.
10:00:26 AMACheckpoint at step 3000: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.82s.
10:00:24 AMAStep 3000: val 5.282, train 5.275 (gap +0.007).
10:00:09 AMBCheckpoint at step 1250: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 5.88s.
09:59:47 AMAStep 2800: val 5.284, train 5.270 (gap +0.014).
09:59:42 AMBStep 1200: val 5.652, train 5.608 (gap +0.045).
09:59:34 AMACheckpoint at step 2750: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.83s.
09:59:01 AMAStep 2600: val 5.299, train 5.301 (gap -0.003).
09:58:37 AMACheckpoint at step 2500: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.86s.
09:58:08 AMBCheckpoint at step 1000: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 6.30s.
09:58:06 AMAStep 2400: val 5.339, train 5.320 (gap +0.020).
09:57:59 AMBStep 1000: val 5.723, train 5.679 (gap +0.044).
09:57:18 AMACheckpoint at step 2250: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.83s.
09:57:01 AMAStep 2200: val 5.358, train 5.336 (gap +0.022).
09:55:54 AMACheckpoint at step 2000: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.82s.
09:55:52 AMAStep 2000: val 5.397, train 5.415 (gap -0.019).
09:55:49 AMBStep 800: val 5.817, train 5.818 (gap -0.001).
09:55:11 AMBCheckpoint at step 750: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 6.17s.
09:54:52 AMAStep 1800: val 5.480, train 5.421 (gap +0.059).
09:54:32 AMACheckpoint at step 1750: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.82s.
09:53:44 AMAStep 1600: val 5.575, train 5.508 (gap +0.067).
09:53:31 AMBStep 600: val 5.991, train 5.998 (gap -0.008).
09:53:10 AMACheckpoint at step 1500: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.84s.
09:52:36 AMAStep 1400: val 5.604, train 5.554 (gap +0.051).
09:52:22 AMBCheckpoint at step 500: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 6.22s.
09:51:50 AMACheckpoint at step 1250: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.82s.
09:51:33 AMAStep 1200: val 5.735, train 5.697 (gap +0.038).
09:51:13 AMBStep 400: val 6.309, train 6.240 (gap +0.068).
09:50:27 AMACheckpoint at step 1000: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.82s.
09:50:24 AMAStep 1000: val 5.790, train 5.744 (gap +0.045).
09:49:33 AMBCheckpoint at step 250: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 6.35s.
09:49:22 AMAStep 800: val 5.895, train 5.895 (gap +0.000).
09:49:02 AMACheckpoint at step 750: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.84s.
09:48:56 AMBStep 200: val 6.607, train 6.583 (gap +0.024).
09:48:20 AMAStep 600: val 6.063, train 6.075 (gap -0.013).
09:47:45 AMACheckpoint at step 500: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.85s.
09:47:36 AMBCheckpoint at step 100: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 6.15s.
09:47:14 AMAStep 400: val 6.339, train 6.267 (gap +0.072).
09:46:58 AMBCheckpoint at step 50: 1486.8 MB (weights, AdamW state, schedule position, RNG states) in 7.34s.
09:46:33 AMACheckpoint at step 250: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.82s.
09:46:17 AMAStep 200: val 6.634, train 6.612 (gap +0.023).
09:45:54 AMACheckpoint at step 100: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.82s.
09:45:48 AMBCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/71272462-4e44-457f-8dfa-535f781d411d/step-0000000.pt in 4.8s (in the background; training didn't wait).
09:45:45 AMACheckpoint at step 50: 193.3 MB (weights, AdamW state, schedule position, RNG states) in 0.86s.
09:45:44 AMBCheckpoint at step 0: 495.6 MB (weights, AdamW state, schedule position, RNG states) in 2.14s.
09:45:41 AMBExpected loss before training ≈ ln(vocab) = 10.83.
09:45:41 AMBTraining 123.89M params on fineweb_edu for 3000 steps (bf16, batch 32×1, ctx 256).
09:45:36 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/35cd9bae-ad26-4927-88a6-5fa1484eaa88/step-0000000.pt in 1.3s (in the background; training didn't wait).
09:45:34 AMACheckpoint at step 0: 64.4 MB (weights, AdamW state, schedule position, RNG states) in 0.24s.
09:45:34 AMAExpected loss before training ≈ ln(vocab) = 10.83.
09:45:34 AMATraining 16.10M params on fineweb_edu for 3000 steps (bf16, batch 32×1, ctx 256).