train.galvyn.xyz · lab
read-onlySign in
← lab

Baseline, twice

lesson 1.2 · Baseline, twice

How much do two runs differ when only the random seed changes?

seed: A 1 vs B 2

Run A

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

16,384,000

step

2,000 / 2,000

train loss
5.250
best val
5.330
tokens/s
22k
MFU
3.7%
peak mem
6.4 GB
lr
1.0e-4

0:13:24 · 0.016 kWh ≈ <$0.01 power · ≈ $0.07 to rent

Run B

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

16,384,000

step

2,000 / 2,000

train loss
5.423
best val
5.328
tokens/s
22k
MFU
3.8%
peak mem
6.4 GB
lr
1.0e-4

0:13:27 · 0.016 kWh ≈ <$0.01 power · ≈ $0.07 to rent

Losslower is better · starts near ln(vocab)
6810500100015002000stepB checkpoint @ 50A checkpoint @ 50A checkpoint @ 100B checkpoint @ 100A checkpoint @ 250B checkpoint @ 250A checkpoint @ 500B checkpoint @ 500B checkpoint @ 750A checkpoint @ 750A checkpoint @ 1000B checkpoint @ 1000A checkpoint @ 1250B checkpoint @ 1250A checkpoint @ 1500B checkpoint @ 1500A checkpoint @ 1750B checkpoint @ 1750A checkpoint @ 2000B checkpoint @ 2000

The machine, live

NVIDIA L4 · gcp-l4-laboffline · just now

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

0.032 kWh

≈ <$0.01 of electricity

RAM

–

of 31.3 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

What to watch · The gap between these curves is the noise floor: any A/B difference smaller than this means nothing.

Run A

  1. Queued

    done

  2. Download data

    done

  3. Tokenize

    done

  4. Build and compile

    done

  5. Warm up

    done

  6. Train

    done

  7. Finished

    done

Run B

  1. Queued

    done

  2. Download data

    done

  3. Tokenize

    done

  4. Build and compile

    done

  5. Warm up

    done

  6. Train

    done

  7. Finished

    done

solid = train (smoothed) · dots = validation · dashed = after a crash-test resume
Learning ratewarmup, then cosine
0e+05e-41e-3500100015002000step
Gradient normbefore clipping to 1.0 · log scale
0.5125500100015002000step
Throughputtokens per second
10k20k30k500100015002000step
MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)
2.0%4.0%6.0%500100015002000step
GPU memory (measured)peak per log window · dashed = activations + workspace
5.9 GB6.3 GB500100015002000step
Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right
0.40.450.5500100015002000step
The questions
arithmeticA 0.47B 0.56
iceA 0.29B 0.34
oceanA 0.42B 0.57
orbitA 0.81B 0.69
parisA 0.44B 0.50
planetA 0.40B 0.07
skyA 0.59B 0.59
waterA 0.61B 0.59
whaleA 0.01B 0.02

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

Run A

Water is very important.
The United States, and a country.
“There is a lot of how to be put to the water they have to come along for the land. It is not just a small amount of time to be removed to the water.
As a country, the country is a growing source of food that has already gone down. They will never keep it right on the water. It will have been a good idea to buy for food.
If you have an oil-rich, it can get this good option of being used and keep the water that is a way to increase your money.
If you need to clean your products, try a healthy diet that is a good, and a high risk can be used for your energy. And it will be a very important tool in the summer.
And now, this is not a growing food that will be stored in your summer. This makes the best ways to start your home.
When you want to spend the

Run B

Water is a key role of the world’s growing. But it will be difficult to build their own energy to achieve the need to increase the risk of a good and more sustainable than other human life.
We should not believe that food will help us develop and develop sustainable health. While the way to develop food quality, it will be better able to maintain our lives. And the water is not the need to become as a result of good health.
We are able to provide the opportunity for our children and our children with our children, their children, and their kids to continue their daily life before their lives. The next step is to make their children who are doing.
When we think about how to eat and eat for your children, the way to protect the children around them!
There are two ways that we need to grow in the home, so we might have to look a lot of food and eat to eat. And they are, not our children will also have fun and not

What happened

  1. BFinished 2000 steps. Best val loss 5.328.
  2. BCheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.55s.
  3. BStep 2000: val 5.328, train 5.281 (gap +0.046).
  4. AFinished 2000 steps. Best val loss 5.330.
  5. ACheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
  6. AStep 2000: val 5.330, train 5.262 (gap +0.068).
  7. BStep 1800: val 5.347, train 5.292 (gap +0.055).
  8. AStep 1800: val 5.360, train 5.312 (gap +0.048).
  9. BCheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
  10. ACheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
  11. BStep 1600: val 5.381, train 5.372 (gap +0.009).
  12. AStep 1600: val 5.403, train 5.381 (gap +0.023).
  13. BCheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
  14. ACheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
  15. BStep 1400: val 5.472, train 5.416 (gap +0.056).
  16. AStep 1400: val 5.474, train 5.440 (gap +0.034).
  17. BCheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
  18. ACheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.61s.
  19. BStep 1200: val 5.530, train 5.482 (gap +0.048).
  20. AStep 1200: val 5.573, train 5.520 (gap +0.053).
  21. BCheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.07s.
  22. ACheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.97s.
  23. BStep 1000: val 5.654, train 5.625 (gap +0.030).
  24. AStep 1000: val 5.672, train 5.627 (gap +0.045).
  25. BStep 800: val 5.866, train 5.739 (gap +0.127).
  26. AStep 800: val 5.837, train 5.783 (gap +0.054).
  27. BCheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.65s.
  28. ACheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.55s.
  29. BStep 600: val 6.031, train 5.947 (gap +0.085).
  30. AStep 600: val 6.008, train 5.992 (gap +0.017).
  31. BCheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.69s.
  32. ACheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.56s.
  33. BStep 400: val 6.227, train 6.210 (gap +0.017).
  34. AStep 400: val 6.269, train 6.227 (gap +0.042).
  35. BCheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.67s.
  36. ACheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.56s.
  37. BStep 200: val 6.561, train 6.514 (gap +0.047).
  38. AStep 200: val 6.600, train 6.561 (gap +0.038).
  39. BCheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.55s.
  40. ACheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
  41. BCheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.64s.
  42. ACheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.62s.
  43. BCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/e9ae5251-c0be-4ecc-ae68-77fbe062307b/step-0000000.pt in 0.7s (in the background; training didn't wait).
  44. BCheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.50s.
  45. ACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/828273b3-29b0-421c-baeb-91e765236cf3/step-0000000.pt in 0.7s (in the background; training didn't wait).
  46. BExpected loss before training ≈ ln(vocab) = 10.83.
  47. BTraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).
  48. ACheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.50s.
  49. AExpected loss before training ≈ ln(vocab) = 10.83.
  50. ATraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).