bf16 vs fp32

lesson 1.6 · Precision

What does mixed precision buy? (This re-run is on an NVIDIA L4.)

precision: A bf16 vs B fp32

Run A

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

32,768,000

step

2,000 / 2,000

train loss: 4.860
best val: 4.897
tokens/s: 73k
MFU: 12.2%
peak mem: 12.5 GB
lr: 1.0e-4

0:15:23 · 0.018 kWh ≈ <$0.01 power · ≈ $0.08 to rent

Run B

done

fineweb_edu · 6L·384d · 30M params · fp32

tokens read

32,768,000

step

2,000 / 2,000

train loss: 4.858
best val: 4.898
tokens/s: 34k
MFU: 23.0%
peak mem: 14.9 GB
lr: 1.0e-4

0:17:27 · 0.021 kWh ≈ <$0.01 power · ≈ $0.09 to rent

Losslower is better · starts near ln(vocab)

The machine, live

NVIDIA L4 · gcp-l4-lab2offline · just now

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

0.039 kWh

≈ <$0.01 of electricity

RAM

–

of 31.3 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

What to watch · Nearly the same curve per step (bf16 rounding makes them drift a little), but B should be slower in tokens/s and hold more activation memory; weights, gradients and AdamW cost the same in both.

Run A

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

Run B

Queued
done
Download data
done
Tokenize
done
Build and compile
done
Warm up
done
Train
done
Finished
done

log losssmoothing0.60solid = train (smoothed) · dots = validation · dashed = after a crash-test resume

Learning ratewarmup, then cosine

Gradient normbefore clipping to 1.0 · log scale

Throughputtokens per second

MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)

GPU memory (measured)peak per log window · dashed = activations + workspace

Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right

The questions

arithmetic	A 0.55	B 0.53
ice	A 0.48	B 0.56
ocean	A 0.55	B 0.58
orbit	A 0.63	B 0.63
paris	A 0.33	B 0.19
planet	A 0.26	B 0.39
sky	A 0.56	B 0.60
water	A 0.60	B 0.52
whale	A 0.12	B 0.14

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

step 2,000

Run A

Water is the most harmful to the body. It is the most effective water supply for the water supply. And while the water is not completely broken and then there is a few of the important nutrients in the water supply.
How can we clean a water supply or plant waste use?
Citing a water waste is to help your plants and the nutrients needed to save the plants.
How can water supply the water supply to get you clean?
- Your soil that uses water from water.
- If water can be a clean, the water can be reduced to a water tank.
- When water is already water, the water will be more likely to happen. If the water has a water quality, it should not be able to be able to heat up in a container and not too much water.
- How can you save water?
- How can you clean up the water supply?
- How can you clean water have a water?
- How can you clean water if

Run B

Water is the most important part of the water. This can also be the most important part of the water water.
The water is the water and water. It takes the water of the water in the water.
In the place of the water and water, the water is the source of water. The water is the water is the water the water.
The water can be used in the water. It doesn’t help you save water, but this will help you make water clean water. This could reduce water, and it can help you get rid of wastewater.
This is one of the first nutrients that you’re going!
- Some water are very cool, so it is good to keep our water away from the water.
- For a few years, you’ll eat a lot of water.
- You’ll also add water to your water.
- You’ve just have a lot of air.
- The water is the

What happened

10:40:28 AMBFinished 2000 steps. Best val loss 4.898.
10:40:27 AMBCheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.52s.
10:40:25 AMBStep 2000: val 4.898, train 4.887 (gap +0.011).
10:38:42 AMBStep 1800: val 4.937, train 4.879 (gap +0.058).
10:38:11 AMBCheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.58s.
10:36:58 AMBStep 1600: val 5.031, train 4.950 (gap +0.080).
10:36:03 AMBCheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.55s.
10:35:12 AMBStep 1400: val 5.070, train 5.029 (gap +0.041).
10:33:54 AMBCheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.61s.
10:33:28 AMBStep 1200: val 5.178, train 5.144 (gap +0.034).
10:31:45 AMBCheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.60s.
10:31:43 AMBStep 1000: val 5.282, train 5.244 (gap +0.038).
10:30:00 AMBStep 800: val 5.414, train 5.403 (gap +0.011).
10:29:29 AMBCheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.56s.
10:28:16 AMBStep 600: val 5.621, train 5.587 (gap +0.034).
10:27:21 AMBCheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.63s.
10:26:30 AMBStep 400: val 5.941, train 5.868 (gap +0.073).
10:25:12 AMBCheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.41s.
10:24:46 AMBStep 200: val 6.364, train 6.347 (gap +0.017).
10:23:52 AMBCheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
10:23:26 AMBCheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.52s.
10:23:03 AMBCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/52eb5996-1115-4705-b0e2-6b5cefbdfa90/step-0000000.pt in 1.7s (in the background; training didn't wait).
10:23:01 AMBCheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.51s.
10:23:00 AMBExpected loss before training ≈ ln(vocab) = 10.83.
10:23:00 AMBTraining 30.06M params on fineweb_edu for 2000 steps (fp32, batch 64×1, ctx 256).
10:22:55 AMBResuming run B from its checkpoint at step 0.
10:15:57 AMAFinished 2000 steps. Best val loss 4.897.
10:15:57 AMACheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
10:15:55 AMAStep 2000: val 4.897, train 4.885 (gap +0.012).
10:15:06 AMAStep 1800: val 4.936, train 4.879 (gap +0.057).
10:14:51 AMACheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
10:14:26 AMBCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/52eb5996-1115-4705-b0e2-6b5cefbdfa90/step-0000000.pt in 1.8s (in the background; training didn't wait).
10:14:25 AMBRun crashed: OutOfMemoryError: CUDA out of memory. Tried to allocate 3.07 GiB. GPU 0 has a total capacity of 22.03 GiB of which 510.00 MiB is free. Process 8745 has 12.74 GiB memory in use. Including non-PyTorch memory, this process has 8.79 GiB memory in use. Of the allocated memory 8.56 GiB is allocated by PyTorch, and 3.06 MiB is reserved by PyTorch but unallocated.
10:14:24 AMBCheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.48s.
10:14:23 AMBExpected loss before training ≈ ln(vocab) = 10.83.
10:14:23 AMBTraining 30.06M params on fineweb_edu for 2000 steps (fp32, batch 64×1, ctx 256).
10:14:15 AMAStep 1600: val 5.028, train 4.950 (gap +0.078).
10:13:40 AMACheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.60s.
10:12:46 AMAStep 1400: val 5.071, train 5.030 (gap +0.041).
10:11:31 AMACheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.61s.
10:11:04 AMAStep 1200: val 5.176, train 5.141 (gap +0.035).
10:09:23 AMACheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
10:09:20 AMAStep 1000: val 5.284, train 5.247 (gap +0.037).
10:07:38 AMAStep 800: val 5.411, train 5.399 (gap +0.012).
10:07:05 AMACheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
10:07:00 AMAStep 730 / 2000 (37%), 12M tokens. Train loss 5.384, best val 5.626, fact probes 0.44. 32054 tok/s; about 00d 00h 11m left.
10:05:54 AMAStep 600: val 5.626, train 5.591 (gap +0.035).
10:04:57 AMACheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.59s.
10:04:03 AMAStep 400: val 5.942, train 5.868 (gap +0.074).
10:02:46 AMACheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.59s.
10:02:19 AMAStep 200: val 6.365, train 6.349 (gap +0.015).
10:01:30 AMACheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
10:01:03 AMACheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
10:00:37 AMACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/5f211d89-80f3-4248-80ed-00476c4bbbe9/step-0000000.pt in 1.6s (in the background; training didn't wait).
10:00:35 AMACheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.48s.
10:00:34 AMAExpected loss before training ≈ ln(vocab) = 10.83.
10:00:34 AMATraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 64×1, ctx 256).