NLP Fundamentals
Tokenization
The process of breaking text into the discrete units (tokens) a language model actually processes.
Definition
Tokenization is the step that converts human-readable text into the numerical units a model can work with. Different models use different schemes — byte-pair encoding, WordPiece, SentencePiece — but the principle is the same: text is split into sub-word fragments, each mapped to an integer. A single English word may be one token, or several; punctuation, spaces, and emoji also become tokens. Every model has a maximum number of tokens it can consider at once, which sets the practical limit on prompt length and response length.
Why it matters
Tokenization affects three things writers care about. First, cost: most APIs price per token, so longer prompts and longer outputs cost more. Second, context window: a model that 'sees' 128,000 tokens can work with a long document; one limited to 8,000 cannot. Third, behaviour: tokenization quirks can surprise — for example, models sometimes mishandle numbers because each digit may be a separate token. For practical writing work, treat 1,000 tokens as roughly 750 English words. When a tool truncates output or 'forgets' earlier instructions, the cause is usually a context-window limit measured in tokens, not in words or characters.
Examples
Tokenized text
The sentence 'Tokenization is interesting.' might split into ['Token', 'ization', ' is', ' interesting', '.'] — five tokens for four words.
Cost implication
A 3,000-word document with a 500-word response is roughly 4,700 tokens — multiplied by per-token API pricing on both input and output sides.
Frequently asked
Do all models tokenize text the same way?
No. Each model family ships its own tokenizer. The same paragraph may consume more tokens in one model than another.
Does tokenization affect output quality?
Indirectly. It affects how much context the model can hold, and how it represents rare or numeric content — both of which can change the answer.
How can I estimate token count?
Most providers publish a tokenizer tool. As a rough rule, divide character count by four for English text.
Related terms
Large Language Model (LLM)
A neural network trained on very large text corpora to predict the next token in a sequence, capable of producing fluent natural-language output.
Context Window
The maximum amount of text — measured in tokens — that a language model can consider at once when generating its response, including both the prompt and the output.
Embeddings
Numerical vector representations of text whose distances capture semantic similarity between pieces of content.
Put the concept to work
Open the rewrite engine and apply this principle to a draft of your own.
Try a rewrite