Byte Pair Encoding: Learning Reusable Text Pieces
How can repeated neighbouring symbols become a reusable vocabulary for new text?
How this lesson unfolds.
Begin with How can a tokenizer build useful pieces instead of memorising every word? The sequence then connects Tokens do not have a fixed character length and builds toward What the model finally receives.
How can a tokenizer build useful pieces instead of memorising every word?
BPE repeatedly combines frequent neighbouring units to build a reusable subword vocabulary.
- Helpful foundationsCharacters, words, and tokenization
- Next connectionTokens do not have a fixed character length
More context
A merge machine combines two neighbouring blocks into a reusable piece, illustrating how BPE grows a vocabulary from repeated local patterns.
How many characters must a token contain?
No fixed number: learned frequency patterns determine whether a token is one character or several.
- Helpful foundationsWhy learn reusable pieces?
- Next connectionThe compression idea behind BPE
More context
A table compares possible learned pieces of one to five characters, showing that BPE tokens do not have a fixed character length.
What happens when one adjacent pair appears again and again?
Replacing a repeated adjacent pair with one new symbol shortens the representation while preserving a reversible rule.
- Helpful foundationsTokens do not have a fixed character length
- Next connectionWhy the replacement remains reversible
More context
The repeated sequence a b a b a b becomes X X X after saving the reversible rule X expands to a plus b.
If two symbols become one, how can the original text return?
The merge table stores how every created symbol expands back into earlier symbols.
- Helpful foundationsThe compression idea behind BPE
- Next connectionFrom compression to token learning
More context
A merge tree expands the token the into th plus e, and then expands th into t plus h, demonstrating reversibility.
How does the compression loop become a tokenizer-training loop?
Count adjacent pairs, select a frequent pair, merge it everywhere, then recount the changed sequence.
- Helpful foundationsWhy the replacement remains reversible
- Next connectionMark every word ending
More context
A four-part loop shows the BPE training cycle: count adjacent pairs, select a frequent pair, merge every occurrence, then recount.
Why add a word-end symbol before counting pairs?
The symbol </w> distinguishes a piece inside a longer word from the same letters at a word boundary.
- Helpful foundationsFrom compression to token learning
- Next connectionBuild the starting inventory
More context
The words in and inside are written as character sequences ending in the single boundary symbol less-than slash w greater-than.
What exists before the first merge?
Start with individual characters plus one complete word-end symbol, then count each symbol in the training sentence.
- Helpful foundationsMark every word ending
- Next connectionCount adjacent pairs
More context
The training sentence, its seven normalised words, and selected initial character counts establish the state before any merge.
Which neighbours repeat most before the first merge?
Four pairs tie at frequency two; a fixed scan order selects t+h as the first maximum.
- Helpful foundationsBuild the starting inventory
- Next connectionMerge the first winner
More context
A frequency table shows four tied pairs at count two and explains the deterministic left-to-right scan that selects t plus h.
What changes when t+h becomes th?
Every selected t+h occurrence is replaced by th; the active symbols and all adjacent-pair counts must then be rebuilt.
- Helpful foundationsCount adjacent pairs
- Next connectionRecount after the sequence changes
More context
The words there and the are shown before and after every t plus h occurrence is replaced by the token th.
Why can the next winner be different?
The new token th creates new adjacencies; tied maxima are resolved again using the same scan rule.
- Helpful foundationsMerge the first winner
- Next connectionA learned piece can grow again
More context
After the first merge, a new pair table shows th plus e, e plus word-end, and n plus word-end tied at frequency two.
Can a token created earlier take part in a later merge?
Yes. The created token th merges with e to form the larger reusable token the.
- Helpful foundationsRecount after the sequence changes
- Next connectionFind the final repeated pair
More context
A vertical construction diagram shows t plus h becoming th, then th plus e becoming the.
Which adjacency still repeats after the second merge?
Only n+</w> has frequency two, marking an n specifically at the end of a word.
- Helpful foundationsA learned piece can grow again
- Next connectionMerge the boundary-aware pair
More context
A pair table after the second merge highlights n plus word-end as the only pair with frequency two.
What does n</w> mean after the merge?
The token n</w> means n at a word ending; it does not replace every letter n.
- Helpful foundationsFind the final repeated pair
- Next connectionWhy this small demonstration stops
More context
The word in changes from i, n, word-end to i, n-word-end, while an n inside a word remains unchanged.
Why stop after only three merges?
In this tiny sentence every remaining adjacent pair now occurs once, so frequency supplies no repeated winner.
- Helpful foundationsMerge the boundary-aware pair
- Next connectionRead the learned representation
More context
A large one shows that every remaining pair occurs once; a note contrasts this tiny demonstration with a production vocabulary budget.
What pieces exist after the three merges?
The learned pieces reflect repeated patterns in this corpus; different training text would produce a different vocabulary.
- Helpful foundationsWhy this small demonstration stops
- Next connectionActive inventory versus learned history
More context
A table shows the exact final segmentation of all seven training words after the three learned merges.
If an earlier token disappears from the sentence, is its rule lost?
Active counts show what exists now; merge history permanently records how larger pieces were built.
- Helpful foundationsRead the learned representation
- Next connectionTraining and encoding are different
More context
Two columns distinguish pieces active in the current segmentation from the permanent ordered merge history.
Does each new prompt train another tokenizer?
Training learns and saves an ordered merge list; encoding applies those saved rules to new text.
- Helpful foundationsActive inventory versus learned history
- Next connectionReuse learned pieces on an unseen word
More context
Two separate flows contrast tokenizer training, which saves rules, with encoding, which applies those saved rules to new text.
How can BPE represent a word absent from the training sentence?
Encoding replays learned merges and returns the best available combination of known pieces; the unseen word need not become one token.
- Helpful foundationsTraining and encoding are different
- Next connectionModern BPE can begin from bytes
More context
The unseen word theater starts as characters and word-end, then saved rules combine t plus h and th plus e into the.
How can one tokenizer represent arbitrary languages and emoji?
Byte-level starting units provide a reversible route from UTF-8 text to ranked merges and token pieces.
- Helpful foundationsReuse learned pieces on an unseen word
- Next connectionWhat the model finally receives
More context
A UTF-8 table gives byte values for ASCII, an accented character, and an emoji code point before a text-to-byte-to-token pipeline.
What leaves the tokenizer and enters the model?
Encoding returns token pieces, maps them to stable numerical IDs, and passes those IDs to the embedding lookup.
- Helpful foundationsModern BPE can begin from bytes
- Next connectionEmbedding lookup and model input
More context
A pipeline maps the unseen word theater to token pieces, illustrative stable IDs, and then embedding vectors for the model.