train.galvyn.xyz · lab
read-onlySign in
← lab

Eval: loss vs accuracy vs answers

lesson 2.6 · Eval

Do held-out loss, question accuracy and the answers themselves tell the same story?

Run A

done

null · eval · ? params ·

tokens read

0

step

4,320 / 0

train loss
–
best val
1.808
tokens/s
–
MFU
–
peak mem
–
lr
–

0:03:18 · ≈ $0.02 to rent

Losslower is better · loss on answer tokens only · starts from the pretrained model
1.822.22.401000200030004000step

The machine, live

NVIDIA L4 · gcp-l4-ft1offline · never

GPU busy

– %

VRAM

– / 22 GB

Power

– / 230 W

Temperature

– °C

SM clock

– MHz

Memory bus

– %

Fan

– %

CPU

– %

PCIe

to GPU –from GPU –

Batches of token ids cross the PCIe bus into the GPU each step; weights never leave it.

Disk

read –write –

Reads: token windows from the tokenized dataset (mostly from page cache). Writes: checkpoints, about 1.5 GB each for GPT-2 124M.

Network

in –out –

In: dataset downloads from Hugging Face. Out: metrics to Supabase and checkpoint uploads to Cloud Storage.

drawing now

–

last 5 minutes

–

from the card's energy counter

this experiment

–

starts with training

RAM

–

Sampled every 2 s on the box (NVML + psutil) and pushed here over Supabase Realtime. GPU energy is the whole card. Electricity at an assumed $0.15/kWh.

What's happening

  1. Queued

    done

  2. Download data

    already on disk

  3. Tokenize

    already on disk

  4. Build and compile

    done

  5. Warm up

    done

  6. Train

    done

  7. Finished

    done

solid = train (smoothed) · dots = validation · dashed = after a crash-test resume
Learning ratewarmup, then cosine
Waiting for data…
Gradient normbefore clipping to 1.0 · log scale
Waiting for data…
Throughputtokens per second
Waiting for data…
MFUshare of this precision's peak (bf16 111, fp32 27.8 TFLOPS)
Waiting for data…
GPU memory (measured)peak per log window · dashed = activations + workspace
Waiting for data…
Samplessame prompt and random draw every time, so only the model changes

Run A

 A spider has a total of 6 legs.

What happened

  1. AEvaluated 6 checkpoints.
  2. ACheckpoint 4320: 6.0% of 300 held-out questions, 12/20 hand-written; val loss 1.808.
  3. ACheckpoint 4000: 7.0% of 300 held-out questions, 13/20 hand-written; val loss 1.810.
  4. ACheckpoint 3000: 5.7% of 300 held-out questions, 12/20 hand-written; val loss 1.837.
  5. ACheckpoint 2000: 5.3% of 300 held-out questions, 12/20 hand-written; val loss 1.877.
  6. ACheckpoint 1000: 4.0% of 300 held-out questions, 11/20 hand-written; val loss 1.952.
  7. ACheckpoint 0: 3.3% of 300 held-out questions, 8/20 hand-written; val loss 2.453.
  8. AEvaluating 6 checkpoints of run 18b4f4d5 on held-out answers (2776 examples) and 300 + 20 questions.