Imagine you ask ChatGPT a simple question:
“Explain Redis in simple words.”
To us, this is just a normal English sentence.
But an LLM doesn’t process that sentence as complete words. Before the model can understand anything, the text is first broken into smaller units called tokens.
Why?
Why can’t an LLM simply process the words we type? Or, if words are difficult to handle, why not process every character individually?
The answer reveals one of the most important engineering decisions behind modern Large Language Models.
Understanding tokens also helps explain:
- Why LLM APIs charge per token
- Why context windows are measured in tokens
- How models deal with words they have never seen before
- Why tokenization happens before the Transformer processes your prompt
Let’s understand it from an engineering perspective.
Why LLMs Use Tokens Instead of Words