train.galvyn.xyz · lab
read-onlySign in
← lab

Learning rate: 1e-3 vs 1e-2

lesson 1.3 · Learning rate

Where does the learning rate become too high for this model?

lr: A 0.001 vs B 0.01

Run A

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

16,384,000

step

2,000 / 2,000

train loss
5.325
best val
5.300
tokens/s
22k
MFU
3.7%
peak mem
6.4 GB
lr
1.0e-4

0:13:05 · 0.015 kWh ≈ <$0.01 power · ≈ $0.07 to rent

Run B

done

fineweb_edu · 6L·384d · 30M params · bf16

tokens read

16,384,000

step

2,000 / 2,000

train loss
6.665
best val
6.705
tokens/s
22k
MFU
3.7%
peak mem
6.4 GB
lr
1.0e-3

0:13:39 · 0.016 kWh ≈ <$0.01 power · ≈ $0.07 to rent

Losslower is better · starts near ln(vocab)
6810500100015002000stepA checkpoint @ 50B checkpoint @ 50A checkpoint @ 100B checkpoint @ 100B checkpoint @ 250A checkpoint @ 250A checkpoint @ 500B checkpoint @ 500B checkpoint @ 750A checkpoint @ 750A checkpoint @ 1000B checkpoint @ 1000A checkpoint @ 1250B checkpoint @ 1250A checkpoint @ 1500B checkpoint @ 1500B checkpoint @ 1750A checkpoint @ 1750A checkpoint @ 2000B checkpoint @ 2000

The machine, live

NVIDIA L4 · gcp-l4-laboffline · just now

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

0.032 kWh

≈ <$0.01 of electricity

RAM

–

of 31.3 GB in WSL

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

What to watch · Too high usually shows up as a plateau or early spikes while the rate is at its peak, easing as cosine decay brings it down. AdamW and clipping make outright explosions rare. If B wins instead, try 1e-1 next: the right rate depends on the model.

Run A

  1. Queued

    done

  2. Download data

    done

  3. Tokenize

    done

  4. Build and compile

    done

  5. Warm up

    done

  6. Train

    done

  7. Finished

    done

Run B

  1. Queued

    done

  2. Download data

    done

  3. Tokenize

    done

  4. Build and compile

    done

  5. Warm up

    done

  6. Train

    done

  7. Finished

    done

solid = train (smoothed) · dots = validation · dashed = after a crash-test resume
Learning ratewarmup, then cosine
0e+05e-31e-2500100015002000step
Gradient normbefore clipping to 1.0 · log scale
110100500100015002000step
Throughputtokens per second
20k40k60k500100015002000step
MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)
5.0%10.0%500100015002000step
GPU memory (measured)peak per log window · dashed = activations + workspace
5.9 GB6.3 GB500100015002000step
Does it know? Fact probesP(right word) / P(right + wrong): 0.5 = no idea, 1 = sure and right
0.350.40.45500100015002000step
The questions
arithmeticA 0.49B 0.57
iceA 0.64B 0.84
oceanA 0.54B 0.50
orbitA 0.71B 0.67
parisA 0.56B 0.48
planetA 0.27B 0.45
skyA 0.68B 0.33
waterA 0.42B 0.13
whaleA 0.03B 0.07

Prompts include "Q: Is water a liquid at room temperature? A:" (Yes vs No), "The capital of France is" (Paris vs London) and "The largest planet in our solar system is" (Jupiter vs Mars). Full list: trainer/src/galtrain/train.py.

Samplessame prompt and random draw every time, so only the model changes

Run A

Water is the most important part of the future. This can also be the most important part of the water, so the energy is necessary for our body, and it’s important to help you will be to keep your body off. You’re not only interested in making sure to find some of these reasons to help your body and the best needed to get rid of your body. You can’t see to get you on the energy, but also that’s not yet important. This is when you do not know with it.
The new way to make your body to take your first step. You’re not trying to do with your body, and you will try it to put on to the heart and heart of your body up in a side of the body. It will always be sure to do with your body and make you ready for your body.
It’s a good way to do not know that you need to do it. A more severe

Run B

Water is.
The
-based and is a well. (The body for the main
It can develop the time in the same body and same own people a few- or to help in the risk to see your body can be. You.
Here and the use can also do to. The same way to your good example and the most longer as this of the problem of your baby or well and in the first important you can be a question. We will be the entire types.
-sized, we learn a lot of the United of more than a simple. If you think to take the first, the same time.
You is more-year. If they are the world, it is a child to do to the same children of the time in a result and the world, a time to make that we do only to provide a variety. It are to give the world.
This will be more you can be not be a child are to get you are that is the

What happened

  1. BFinished 2000 steps. Best val loss 6.705.
  2. BCheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.42s.
  3. BStep 2000: val 6.705, train 6.733 (gap -0.028).
  4. BStep 1800: val 6.753, train 6.735 (gap +0.018).
  5. BCheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.50s.
  6. BStep 1600: val 6.831, train 6.768 (gap +0.063).
  7. BCheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.45s.
  8. BStep 1400: val 6.805, train 6.770 (gap +0.035).
  9. BCheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 3.10s.
  10. AFinished 2000 steps. Best val loss 5.300.
  11. ACheckpoint at step 2000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.57s.
  12. BStep 1200: val 6.938, train 6.928 (gap +0.010).
  13. AStep 2000: val 5.300, train 5.318 (gap -0.018).
  14. BCheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.57s.
  15. AStep 1800: val 5.362, train 5.304 (gap +0.057).
  16. BStep 1000: val 6.960, train 6.922 (gap +0.039).
  17. ACheckpoint at step 1750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.56s.
  18. AStep 1600: val 5.435, train 5.367 (gap +0.068).
  19. BStep 800: val 6.940, train 6.905 (gap +0.035).
  20. BCheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.58s.
  21. ACheckpoint at step 1500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.55s.
  22. AStep 1400: val 5.462, train 5.411 (gap +0.050).
  23. BStep 600: val 6.924, train 6.937 (gap -0.013).
  24. BCheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.62s.
  25. ACheckpoint at step 1250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.57s.
  26. AStep 1200: val 5.596, train 5.556 (gap +0.040).
  27. BStep 400: val 6.956, train 6.887 (gap +0.069).
  28. BCheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.45s.
  29. ACheckpoint at step 1000: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 2.57s.
  30. AStep 1000: val 5.673, train 5.627 (gap +0.046).
  31. BStep 200: val 7.026, train 7.006 (gap +0.020).
  32. BCheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.53s.
  33. AStep 800: val 5.782, train 5.786 (gap -0.004).
  34. BCheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.60s.
  35. BCheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/ff97b66f-44aa-47c3-a31d-fc09838a95de/step-0000000.pt in 0.9s (in the background; training didn't wait).
  36. ACheckpoint at step 750: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.92s.
  37. BCheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 1.39s.
  38. BExpected loss before training ≈ ln(vocab) = 10.83.
  39. BTraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).
  40. AStep 600: val 5.975, train 5.990 (gap -0.015).
  41. ACheckpoint at step 500: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.56s.
  42. AStep 400: val 6.265, train 6.193 (gap +0.072).
  43. ACheckpoint at step 250: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.54s.
  44. AStep 200: val 6.571, train 6.549 (gap +0.022).
  45. ACheckpoint at step 100: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.57s.
  46. ACheckpoint at step 50: 360.9 MB (weights, AdamW state, schedule position, RNG states) in 1.55s.
  47. ACheckpoint step 0 copied off the machine to gs://train-galvyn-xyz-checkpoints/runs/8218b3e5-5e83-48b2-86f7-12b821bbe473/step-0000000.pt in 0.7s (in the background; training didn't wait).
  48. ACheckpoint at step 0: 120.3 MB (weights, AdamW state, schedule position, RNG states) in 0.50s.
  49. AExpected loss before training ≈ ln(vocab) = 10.83.
  50. ATraining 30.06M params on fineweb_edu for 2000 steps (bf16, batch 32×1, ctx 256).