Deeper

Lab · c24.s02.L1

Tokeniser

The token is the unit of prediction: a tokeniser fitted to text decides where the sentence is cut, and each token becomes one training example.

Question: What does a model learn from, and what objective does it use?

TokeniserA sentence appears as a row of chips, one per token. Chips before the gold-ringed target are solid navy; chips after it are dashed; a short underline marks a leading space. A slider sets how many letter merges the tokeniser has applied: a sentence becomes twenty-six single characters at zero merges or six whole words at twenty-four, so its tasks fall from twenty-five to five. The earlier chips spell the context and the next token sits in a box.TokeniserOne sentence, cut into tokensbeforeafterto predicta short underline in a chip marks a leading spacethemodellearnsthenexttoken6 tokens · vocabulary 50 · 5.17 characters per tokenthe context: tokens before the target"the model learns the"the token to predictnextprediction tasks: 5one per token145"the model learns the" → "next".Task 4 of 5: the ring marks the token the model must produce.24 merges (last: n + ext· → next·) · vocabulary 50 · 5.17 characters per token
Synthetic Synthetic: a nine-line corpus written for this lab with invented frequencies, and a 24-merge table fitted to it by data/scripts/c24.s02.L1.py. The frames cut four sentences at 0 to 24 merges: the first sentence is 26 single characters at 0 merges and 6 tokens at 24, so its 25 prediction tasks become 5. The splits are exact and repeatable; this is not the vocabulary of any production tokeniser.
Lab controls
24 merges

Left/Down and Right/Up move by 3; Page Down and Page Up move by 12; Home sets 0; End sets 24

the model learns the next token

Up and Down move one sentence; Home selects the first; End selects the fourth

700 ms

Left/Down and Right/Up move by 100 ms; Home sets 400 ms; End sets 1200 ms

Frame 1 of 6: Task 4 of 5Task 4 of 5. The 4 solid chips before the ring spell the context, the ringed chip is the token to produce ("next"), and the dashed chips after it lie outside this task. The lab opens here: 6 tokens give 5 prediction tasks, and each task asks for one whole token worth 5.17 characters.

Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.

Text description of this lab

A sentence appears as a row of chips, one per token. Chips before the gold-ringed target are solid navy; chips after it are dashed; a short underline marks a leading space. A slider sets how many letter merges the tokeniser has applied: a sentence becomes twenty-six single characters at zero merges or six whole words at twenty-four, so its tasks fall from twenty-five to five. The earlier chips spell the context and the next token sits in a box.

Return to the section that hosts this lab

Caption status: synthetic. Every lab declares one of published, re-simulated, schematic, synthetic or interpolated.