Imagine you ask ChatGPT a simple question:

“Explain Redis in simple words.”

To us, this is just a normal English sentence.

But an LLM doesn’t process that sentence as complete words. Before the model can understand anything, the text is first broken into smaller units called tokens.

Why?

Why can’t an LLM simply process the words we type? Or, if words are difficult to handle, why not process every character individually?

The answer reveals one of the most important engineering decisions behind modern Large Language Models.

Understanding tokens also helps explain:

• Why LLM APIs charge per token
• Why context windows are measured in tokens
• How models deal with words they have never seen before
• Why tokenization happens before the Transformer processes your prompt

Let’s understand it from an engineering perspective.

Why LLMs Use Tokens Instead of Words

---

The First Problem: LLMs Don’t Read Text

Suppose you send this prompt:

You can read this sentence instantly.

An LLM can’t.

At its core, a neural network performs mathematical operations on numbers. It cannot directly perform those calculations on the string "Redis".

So before the model can process our prompt, we need to solve a fundamental problem:

How do we convert human language into numerical representations that a model can process?

One seemingly obvious solution is to assign a number to every word.

---

Approach 1: Give Every Word a Number

Imagine creating a dictionary like this:

Now our sentence could be represented as something like:

This looks simple.

But there is a major scalability problem.

Language never stops growing

New words appear constantly.

Think about terms such as:

Some of these didn’t exist—or weren’t widely used—a few years ago.

Tomorrow, new frameworks, companies, products, libraries, and technologies will appear.

If every possible word required its own entry, the vocabulary would become extremely large.

And we aren’t dealing with only English.

A useful model may need to process:

• English
• Hindi
• Japanese
• Arabic
• French
• Java
• Python
• SQL
• JSON
• URLs
• Emojis
• Mathematical notation

There is another important problem: unknown words.

Imagine someone creates a new database tomorrow called:

If the model’s vocabulary only contained complete words and SuperCacheX wasn’t present, how would the model represent it?

A pure word-level vocabulary struggles with words it has never seen before.

So representing every possible word individually isn’t a practical solution.

---

Approach 2: Why Not Process Characters?

We could go to the opposite extreme.

Instead of storing every possible word, we could process individual characters.

For example:

becomes:

Now the vocabulary can be dramatically smaller.

We only need representations for the characters the tokenizer supports.

Problem solved?

Not quite.

Consider the word:

At the character level, the model receives something closer to:

One meaningful unit has now turned into many separate input units.

Imagine doing this for thousands of words.

The sequence becomes much longer.

Longer sequences generally mean more processing and more memory usage for the model.

So we now have two extremes:

Approach	Advantage	Problem
Complete words	Shorter sequences	Very large vocabulary and unknown-word problems
Characters	Small vocabulary	Much longer sequences

We need something between the two.

That is where tokens come in.

---

The Engineering Solution: Tokens

Modern LLM tokenizers generally break text into reusable pieces rather than requiring every possible complete word to exist in the vocabulary.

A token might represent:

• A complete word
• Part of a word
• Punctuation
• Whitespace combined with text
• Other frequently occurring text patterns

For example, a tokenizer might represent a familiar word as one token:

While another word might be split into multiple pieces:

The exact split depends on the tokenizer used by the model.

This becomes particularly useful when the model encounters unfamiliar text.

Imagine a new database called:

The exact string might not exist as a single token.

But the tokenizer may be able to represent it using smaller known pieces, conceptually like:

The important idea is not that these are the exact tokens a particular tokenizer will produce.

The important idea is:

A tokenizer can construct unfamiliar text from smaller pieces already present in its vocabulary.

This gives us a practical compromise.

The vocabulary doesn’t need to contain every possible word, while the model also doesn’t need to process everything character by character.

---

Tokens Are a Trade-Off

This is the key mental model.

Tokenization is essentially an engineering trade-off between:

Vocabulary size

and

Sequence length

If we use complete words:

If we use individual characters:

Tokens provide the middle ground:

This is one of the reasons tokenization is such an important part of modern LLM architecture.

---

Token IDs Are Still Not What the Transformer Processes

There is another important distinction.

Suppose tokenization produces:

These numbers are token IDs.

The Transformer doesn’t interpret 43291 as meaning “Redis.”

It’s essentially an identifier pointing to an entry in the model’s vocabulary.

Before the Transformer can process the sequence, each token ID is mapped to a learned numerical vector called an embedding.

Conceptually:

These vectors are the numerical representations the neural network actually operates on.

---

Where Tokenization Fits in the LLM Pipeline

Let’s put everything together.

When you send:

the simplified processing pipeline looks like this:

At a high level:

1. Tokenization

Your text is broken into tokens and mapped to token IDs.

2. Embeddings

Those token IDs are mapped to learned numerical vectors.

3. Transformer processing

The Transformer processes these representations in context and produces probabilities for what token should come next.

4. Next-token generation

The model selects a next token, then repeats the process autoregressively to generate additional tokens.

5. Decoding

The generated token IDs are converted back into text that you can read.

So the model never directly sees the sentence the way humans do.

It operates on numerical representations derived from tokens.

---

Why Context Windows Are Measured in Tokens

You may have seen models described as supporting context windows such as:

Why tokens instead of words?

Because tokens are the actual discrete input units presented to the model before embedding.

Suppose you send:

• Your system prompt
• Conversation history
• A large document
• Source code
• Your latest question

All of this content must first be tokenized.

Those tokens consume the model’s available context window.

That’s why two pieces of text containing the same number of words don’t necessarily consume the same amount of context.

Their token counts can be different.

---

Why LLM APIs Charge Per Token

This also explains something developers encounter immediately when using LLM APIs:

token-based pricing.

Why don’t providers simply charge per word or character?

Because the model’s workload is tied much more directly to the tokens it processes and generates.

Your request generally involves input tokens:

And the model produces output tokens:

Processing and generating more tokens requires additional computation, memory, and time.

That’s why LLM APIs commonly express usage and pricing in terms of tokens rather than human-visible word counts.

---

One Concept Explains Several LLM Behaviors

Once you understand tokenization, several things about LLMs start making more sense.

Context windows

They are measured in tokens because tokens are the model’s input units.

API pricing

Providers charge based on input and output tokens because those tokens correspond to computational work.

Unknown words

New words don’t necessarily require new vocabulary entries because they can often be decomposed into existing token pieces.

Prompt size

Two prompts with similar word counts can have different token counts depending on how the tokenizer splits them.

LLM processing

Before the Transformer can process your text, it must first be converted into tokens and then numerical representations.

---

The Mental Model to Remember

If you remember only one thing from this article, remember this:

Complete words would make the vocabulary difficult to scale.

Individual characters would make sequences unnecessarily long for many kinds of text.

Tokens provide a practical middle ground.

They allow LLMs to represent enormous amounts of language with a manageable vocabulary while still being able to encode text they haven’t encountered as a single vocabulary item before.

And that’s why, before an LLM can generate even a single word of its answer, your prompt first becomes tokens.

---

What’s Next?

Tokenization solves the first problem:

How do we convert text into units a model can work with?

But it creates the next question.

Once we have tokens, how does the model actually understand the relationship between them?

Why does the meaning of a word change depending on the words around it?

And how can a Transformer determine which parts of a sentence matter most?

That takes us into embeddings, positional information, and attention—the next pieces in understanding how LLMs actually work.