CH 07 · Glossary: Foundational Modelling · 07/28
Read as one pageTokenization
Tokenization is the step that chops raw text into tokens before the model sees it. A model cannot read letters directly; it needs a fixed vocabulary of tokens, each mapped to a number. The tokenizer is the tool that performs this split consistently. It is a small but important detail, because how text is split affects how efficiently a model can process different languages and symbols.
Finished this idea?
Your learning progress stays in this browser.