What Are Tokens — And Why Should You Care?

The hidden unit of measurement that shapes every conversation you have with Claude. You type a message to Claude. You hit send. A response flows back in seconds. Simple, right? But beneath that seamless exchange, something interesting is happening — your words are being sliced into tiny linguistic units called tokens before Claude ever "reads" them. Tokens are the atomic unit of language for large language models. They're not characters, and they're not always full words. They sit somewhere in between — and understanding them unlocks a clearer picture of how AI language models actually work, why they have limits, and how to work with those limits instead of against them. So what exactly is a token? Think of tokenisation as breaking text into the most useful chunks for a model to learn from. Common words like "the" or "and" are usually one token. Longer or rarer words might get split into two or three pieces. Punctuation, spaces, and newlines all count too. Example — how this sentence gets tokenised Claude under stands language through tok en isation . As a rough rule of thumb: 100 tokens is about 75 words, or a short paragraph. A typical novel runs around 100,000 words — that's roughly 133,000 tokens. Claude's extended context window can hold the equivalent of several books at once. The context window: Claude's working memory Every conversation with Claude happens inside a context window — a fixed-size buffer that holds everything Claude can "see" at once. This includes your entire conversation history, any documents you paste in, system instructions, and Claude's own responses. Once the window fills up, older content scrolls out of view. Claude doesn't forget it in the human sense — it simply can't read past what fits. This is why very long conversations can occasionally feel like Claude loses track of something said much earlier.