GPT-2 tokens vs raw bytes

lesson 1.5 · Tokens vs bytes

Same web text, same model: what changes when it reads bytes instead of GPT-2 tokens?

dataset: A fineweb_edu vs B fineweb_edu_bytes

Run A

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

16,384,000

step

2,000 / 2,000

train loss: 5.325
best val: 5.300
tokens/s: 23k
MFU: 3.9%
peak mem: 6.4 GB
lr: 1.0e-4

0:13:53 · 0.016 kWh ≈ <$0.01 power · ≈ $0.07 to rent

Run B

done

fineweb_edu_bytes · 6L·384d · 11M params · bf16

tokens read

16,384,000

step

2,000 / 2,000

train loss: 1.460
best val: 1.400
tokens/s: 66k
MFU: 4.3%
peak mem: 0.9 GB
lr: 1.0e-4

0:04:18 · 0.005 kWh ≈ <$0.01 power · ≈ $0.02 to rent

Losslower is better · starts near ln(vocab)

The machine, live

NVIDIA L4 · gcp-l4-laboffline · just now

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

0.021 kWh

≈ <$0.01 of electricity

RAM

–

of 31.3 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

What to watch · Losses aren't comparable as-is: one GPT-2 token covers several bytes of English text, so divide A's loss by its bytes per token to get loss per byte. With the same step budget A also reads several times more text. Compare samples: B has to learn spelling before it can learn words.

Run A

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

Run B

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

GPU memory (measured)peak per log window · dashed = activations + workspace

Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right

The questions

arithmetic	A 0.49	B –
ice	A 0.64	B –
ocean	A 0.54	B –
orbit	A 0.71	B –
paris	A 0.56	B –
planet	A 0.27	B –
sky	A 0.68	B –
water	A 0.42	B –
whale	A 0.03	B –

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

step 2,000

Run A

Water is the most important part of the future. This can also be the most important part of the water, so the energy is necessary for our body, and it’s important to help you will be to keep your body off. You’re not only interested in making sure to find some of these reasons to help your body and the best needed to get rid of your body. You can’t see to get you on the energy, but also that’s not yet important. This is when you do not know with it.
The new way to make your body to take your first step. You’re not trying to do with your body, and you will try it to put on to the heart and heart of your body up in a side of the body. It will always be sure to do with your body and make you ready for your body.
It’s a good way to do not know that you need to do it. A more severe

Run B

Water is an extinguished to make and the primarily boost. Sometimes drink a protection not the forevers have more alternative in the energy and numerous conductions to be able to be to sure than the capacity

What happened

09:13:19 AMAFinished 2000 steps. Best val loss 5.300.
09:13:19 AMACheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.61s.
09:13:16 AMAStep 2000: val 5.300, train 5.318 (gap -0.018).
09:11:57 AMAStep 1800: val 5.362, train 5.304 (gap +0.057).
09:11:33 AMACheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.69s.
09:10:35 AMAStep 1600: val 5.435, train 5.367 (gap +0.068).
09:09:52 AMACheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:09:14 AMAStep 1400: val 5.462, train 5.411 (gap +0.050).
09:08:12 AMACheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.62s.
09:07:56 AMAStep 1200: val 5.596, train 5.556 (gap +0.040).
09:06:35 AMACheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.55s.
09:06:32 AMAStep 1000: val 5.673, train 5.627 (gap +0.046).
09:05:17 AMAStep 800: val 5.782, train 5.786 (gap -0.004).
09:04:55 AMACheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
09:04:27 AMAStep 600: val 5.975, train 5.990 (gap -0.015).
09:03:45 AMACheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.56s.
09:03:42 AMBFinished 2000 steps. Best val loss 1.400.
09:03:42 AMBCheckpoint at step 2000: 130.2 MB (weights, AdamW state, schedule position, RNG states) in 0.49s.
09:03:40 AMBStep 2000: val 1.400, train 1.386 (gap +0.014).
09:03:14 AMBStep 1800: val 1.411, train 1.385 (gap +0.026).
09:03:06 AMBCheckpoint at step 1750: 130.2 MB (weights, AdamW state, schedule position, RNG states) in 0.48s.
09:02:53 AMAStep 400: val 6.265, train 6.193 (gap +0.072).
09:02:48 AMBStep 1600: val 1.421, train 1.408 (gap +0.013).
09:02:34 AMBCheckpoint at step 1500: 130.2 MB (weights, AdamW state, schedule position, RNG states) in 0.50s.
09:02:19 AMBStep 1400: val 1.465, train 1.448 (gap +0.017).
09:02:02 AMBCheckpoint at step 1250: 130.2 MB (weights, AdamW state, schedule position, RNG states) in 0.56s.
09:01:56 AMBStep 1200: val 1.487, train 1.488 (gap -0.001).
09:01:36 AMACheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.55s.
09:01:31 AMBCheckpoint at step 1000: 130.2 MB (weights, AdamW state, schedule position, RNG states) in 0.55s.
09:01:28 AMBStep 1000: val 1.563, train 1.551 (gap +0.012).
09:01:10 AMAStep 200: val 6.571, train 6.549 (gap +0.022).
09:01:03 AMBStep 800: val 1.637, train 1.631 (gap +0.005).
09:00:55 AMBCheckpoint at step 750: 130.2 MB (weights, AdamW state, schedule position, RNG states) in 0.49s.
09:00:36 AMBStep 600: val 1.746, train 1.739 (gap +0.007).
09:00:25 AMBCheckpoint at step 500: 130.2 MB (weights, AdamW state, schedule position, RNG states) in 0.56s.
09:00:17 AMACheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.57s.
09:00:11 AMBStep 400: val 1.945, train 1.947 (gap -0.001).
08:59:54 AMBCheckpoint at step 250: 130.2 MB (weights, AdamW state, schedule position, RNG states) in 1.57s.
08:59:54 AMACheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.85s.
08:59:47 AMBStep 200: val 2.466, train 2.462 (gap +0.004).
08:59:33 AMBCheckpoint at step 100: 130.2 MB (weights, AdamW state, schedule position, RNG states) in 0.49s.
08:59:29 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/b652b55a-b875-4458-ada8-87780502967b/step-0000000.pt in 1.1s (in the background; training didn't wait).
08:59:28 AMACheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 1.51s.
08:59:28 AMBCheckpoint at step 50: 130.2 MB (weights, AdamW state, schedule position, RNG states) in 1.16s.
08:59:26 AMAExpected loss before training ≈ ln(vocab) = 10.83.
08:59:26 AMATraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).
08:59:25 AMBCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/98d90ac3-ec3c-416b-bd6c-8ae0a2a6e9cc/step-0000000.pt in 0.5s (in the background; training didn't wait).
08:59:24 AMBCheckpoint at step 0: 43.4 MB (weights, AdamW state, schedule position, RNG states) in 0.15s.
08:59:24 AMBExpected loss before training ≈ ln(vocab) = 5.55.
08:59:24 AMBTraining 10.84M params on fineweb_edu_bytes for 2000 steps (bf16, batch 32×1, ctx 256).