In short. Tokenization splits text into tokens and maps them to numeric IDs using a vocabulary and rules matched to the model.

Definition: Tokenization splits text into tokens, units such as words, word fragments or punctuation. A tokenizer maps these units to numeric identifiers in a vocabulary, its inventory of tokens, so a model can process them.

Key facts

As of 17 October 2026, Hugging Face's tokenization algorithms documentation describes subword methods including byte pair encoding, Unigram and WordPiece.
Hugging Face's tokenization pipeline documentation separates normalization, pre-tokenization, the tokenization model and post-processing.
The Hugging Face Tokenizers course chapter specifies that tokenization rules and vocabulary must match those used when the model was pretrained.
The Tokenizers Quicktour exposes token strings through Encoding.tokens and their vocabulary indices through Encoding.ids.

How does text become model input?

Hugging Face's pipeline documentation describes normalization as operations on the input string, such as lowercasing or removing accents. Pre-tokenization divides that string into smaller spans. Its Whitespace example splits on spaces and punctuation, separating the apostrophe in a contraction.

The tokenization model then applies learned rules to split those spans into tokens and map them to vocabulary IDs. Here, “model” refers to the trained tokenizer component. Post-processing can add special tokens, such as [CLS] and [SEP], around a sentence. Decoding converts vocabulary IDs back into text, with a decoder handling the way subword pieces join together.

Why can a word contain several tokens?

A subword is a piece of a word represented in a tokenizer's vocabulary. Hugging Face's algorithm guide explains that common words can remain whole while rare words split into familiar pieces. Its example annoyingly can become annoying plus ly, or annoy plus ing plus ly, depending on the vocabulary.

Byte pair encoding, abbreviated BPE, learns merges by repeatedly combining the most frequent adjacent pair. Unigram starts with candidate subwords and removes those whose removal least increases the training loss. These learned vocabularies help explain why a word count differs from a token count.

How can website text be inspected?

Hugging Face's course example shows the intermediate tokens and IDs for bert-base-cased:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
sequence = "Using a Transformer network is simple"
tokens = tokenizer.tokenize(sequence)
ids = tokenizer.convert_tokens_to_ids(tokens)

# tokens: ['Using', 'a', 'transform', '##er', 'network', 'is', 'simple']
# ids: [7993, 170, 11303, 1200, 2443, 1110, 3014]

This is the course's documented example; inspecting a website passage requires its text and the target model's tokenizer. The course also shows that calling the tokenizer directly performs the combined encoding operation.

For tracing a token back to its source text, the Quicktour documents Encoding.offsets, the character spans associated with tokens. It also shows how post-processing adds special tokens and how padding adds [PAD] tokens to align sentence lengths in a batch. Those additions matter when interpreting an encoded sequence's length.

FAQ

Can I use any tokenizer to count a page's tokens?

For a model-specific count, use that model's tokenizer. Hugging Face's course requires the rules and vocabulary used during pretraining, so the example above represents a particular tokenizer.

How can I see where a token comes from?

If you use Hugging Face Tokenizers, inspect the encoding's offsets attribute. The Quicktour shows how these spans identify the corresponding portion of the original sentence.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer