Token (LLM)

Before a model sees text, a tokenizer splits it into tokens drawn from a fixed vocabulary: common words become single tokens, rarer words split into pieces ("optimization" might become "optim" and "ization"). English averages roughly four characters per token; other languages and unusual strings tokenize less efficiently.

Everything downstream is counted in tokens: the context window is a token budget, API pricing is per token, and generation speed is tokens per second. Tokens are the currency of the entire LLM economy.

When an AI engine retrieves your page during RAG, it does not pass the whole page to the model, it chunks content into token-sized passages and injects only the most relevant few hundred tokens. If your key claim is spread across a meandering 800-word section, it may never fit cleanly into a retrieved chunk.

Dense, front-loaded passages survive chunking: the answer in the first sentence, support after. Token-efficiency is the unglamorous reason concise expert writing outperforms padded content in AI answers.

You do not need to count tokens, but you should write as if every retrieved chunk must justify its budget: one idea per section, no preamble, concrete facts early. Unusual brand spellings and stylized names tokenize into fragments, one more reason consistent naming across the web helps models recognize your entity reliably.

Token costs also explain platform behavior: engines summarize rather than reproduce, cite a handful of sources rather than dozens, and truncate long pages, all budget management you can design content around.