Tokenization

Simple Definition

Tokenization is how AI models read text. Before processing anything, a model breaks your input into chunks called tokens. A token is not exactly a word. It’s more like a word fragment. For most English text, one token is about 3–4 characters or roughly 0.75 of a word.

So “Hello, how are you?” might become 6 tokens: Hello, ,, how, are, you, ?

Why Models Use Tokens

AI language models don’t process letters or full words directly. They work with tokens from a fixed vocabulary learned during training. This makes processing more efficient and allows models to handle words they’ve never seen before by breaking them into known pieces.

Unusual or technical words often take more tokens. The word “tokenization” might use 3–4 tokens, while common words like “the” or “is” use just one.

Why Tokenization Matters to You

Understanding tokens matters for practical reasons:

• Cost: most AI APIs charge per token (input + output). More tokens = higher cost.
• Context limits: every model has a maximum context window measured in tokens. If your conversation or document exceeds this limit, the model can’t process all of it at once.
• Speed: more tokens take longer to process and generate.

Rough Token Estimates

Different Tokenizers

Different models use different tokenization systems (called tokenizers). OpenAI models use a tokenizer called tiktoken. Meta’s Llama and Google’s Gemma use different ones. This means the same text may cost different numbers of tokens depending on which model you use.

Related Terms

• Token, the individual unit produced by tokenization
• Context Window, the token limit a model can handle at once
• LLM, the models that use tokenization to process text
• Inference, the process where tokenized input is run through a model