What Is a Token in AI?

In plain language

An AI model does not always process text as one unit per word or one unit per character. The system first breaks content into tokens and then processes those pieces.

According to OpenAI, a token can be a character, part of a word, a full word or punctuation. The number of tokens can vary with the language, model and encoding. (OpenAI Help Center)

That is why token count is not simply another name for word count.

An everyday example

An analogy

The model processes the pieces rather than swallowing an entire passage as one indivisible object. This analogy explains the basic idea; it does not describe every tokenizer's exact rules.

Where you will encounter it

You will see tokens discussed in model limits, context windows, API usage, long-document workflows and comparisons of AI tools. Whenever someone gives an exact token figure, remember that the count depends on the tokenizer and content.

Should you care

• All AI users: useful to understand.
• People working with long documents: worth understanding.
• API developers: essential.
• Heavy agent or MCP users: useful when assessing how much information a workflow adds.

How it differs

Token vs word: A token can be smaller or larger than a word, and punctuation can be tokenized too.

Token vs character: They do not map one-to-one. A token may contain one character, several characters or part of a word.

Token vs context: Tokens are units of content. Context is the information currently available to the model, measured within a context window.

What to read next

Read What is AI context? to see how tokens relate to the information a model can work with at one time.