Tokenization and Embeddings Interview Questions, Explained With a Runnable Demo

Interviewers ask about tokenizers because it's where "the model reads text" stops being true. The model reads integers. Once you can explain what those integers are, you can explain bad arithmetic, uneven pricing across languages, odd whitespace behaviour in code, and why you can't swap one embedding model for another without rebuilding your index.

These are practice questions I wrote, not leaked ones, and they don't come from any particular company. Each has a strong answer outline, the follow-up that tends to come next, and a common wrong answer. I ran the demos below on 2026-10-08 with tiktoken 0.14.0 and NumPy. For other rounds, see AI engineer interview questions on RAG, agents and evals and prompt engineering interview questions with answers. For the cost side of tokens, the LLM token counting guide goes further.

What is the difference between BPE, WordPiece and Unigram tokenization?

Strong answer outline

• BPE (byte-pair encoding), applied to subword units in Sennrich et al. 2015: start from characters, repeatedly merge the most frequent adjacent pair, stop at a target vocabulary size. Encoding replays the merges in order. Byte-level BPE (GPT-2 style) starts from bytes, so any string can be encoded and there's no unknown token.
• WordPiece: also bottom-up, but selects merges by a likelihood-style score instead of plain frequency, and marks continuation pieces (BERT shows them with a ## prefix). At inference it typically does greedy longest-match-first against the vocabulary. It's associated with BERT-family models.
• Unigram (Kudo 2018): top-down. Begin with a large candidate vocabulary, assign each piece a probability, and prune pieces whose removal costs the least likelihood. A string can have several valid segmentations, and the paper uses that ambiguity as a regulariser (sampling segmentations during training).
• SentencePiece (Kudo and Richardson 2018) is a library, not a fourth algorithm: it trains BPE or Unigram directly on raw text without a pre-tokenization step, treating whitespace as a symbol. That helps languages without spaces.

A toy BPE run shows the mechanism. I trained eight merges on a tiny corpus (low ×5, lower ×2, newest ×6, widest ×3) with an end-of-word marker:

merge 1: ('t', '</w>') (count 9)
merge 2: ('s', 't</w>') (count 9)
merge 3: ('e', 'st</w>') (count 9)
merge 4: ('o', 'w') (count 7)
merge 5: ('l', 'ow') (count 7)
merge 6: ('w', 'est</w>') (count 6)
merge 7: ('n', 'e') (count 6)
merge 8: ('ne', 'west</w>') (count 6)
final segmentation: low</w>, low e r </w>, newest</w>, w i d est</w>

Notice "est" got learnt because it appears in two words, and "widest" still ends up as w i d est</w> since there was too little data to merge the rest. Learned vocabularies reflect the training corpus. That's the root of most tokenizer problems.

Follow-up: Why byte-level? To guarantee coverage: any text, including emoji or a script absent from training data, falls back to bytes. The cost is that rare scripts decompose into many tokens.

Common wrong answer: "WordPiece and BPE are the same thing." They share the bottom-up shape but differ in how merges are chosen, and in the continuation marker and encoding procedure.

Why do tokenizers hurt arithmetic and letter counting?

Strong answer outline

• The model operates on token IDs. It never directly sees that "strawberry" has three r's. It has to learn it from how tokens co-occur.
• Numbers are chunked by the tokenizer's merge rules, not by place value. Here is what the two OpenAI encodings in tiktoken did with the inputs below:

== cl100k_base
number 1       tokens=  3 ['123', '456', '7']
number 2       tokens=  3 ['123', '456', '78']
sum            tokens= 10 ['123', '456', ' +', ' ', '654', '321', ' =', ' ', '777', '777']
leading space  tokens=  1 [' strawberry']
no space       tokens=  3 ['str', 'aw', 'berry']
Capital        tokens=  3 ['Str', 'aw', 'berry']
== o200k_base
no space       tokens=  3 ['st', 'raw', 'berry']

Look at 1234567 versus 12345678. Both split left to right into chunks of three digits, and the leftover lands at the end: 123, 456, 7 versus 123, 456, 78. The chunk boundaries follow the string, not place value, so the units, tens and hundreds digits don't sit in a consistent position inside a token. Column arithmetic needs exactly that alignment. Also note that " strawberry" (with a leading space) is one token while "strawberry" at the start of a string is three, and capitalisation changes the first piece. Same word, different inputs.

• Honest caveat: tokenization is one contributing factor. Models also lack reliable multi-step carry logic without reasoning steps. Don't claim tokenization is the sole cause.
• Fixes: let the model use a calculator or code tool; have it spell out digits or write them in a structured form; use reasoning-enabled models for multi-step math; for letter tasks ask it to write the word one character per line first.

Follow-up: How would you test whether a prompt change helps? Build 50 to 100 arithmetic cases with known answers across digit lengths, run baseline and change, compare exact-match accuracy by length bucket. Don't judge from three examples.

Common wrong answer: "LLMs are bad at math because they're just predicting words." It's true-ish but unspecific. A good answer names the token chunking and the missing algorithmic carry steps, and offers a tool-based fix.

Why does non-English text cost more tokens?

Strong answer outline

• Vocabularies are learned from a training corpus dominated by some languages. Scripts that are underrepresented get fewer merges, so the same sentence breaks into more pieces.
• More tokens means more cost per request, less context room, and slower generation per sentence.
• Measured with tiktoken, the same question ("Where is the nearest railway station?") in four languages:

On the older cl100k_base encoding the Hindi sentence took 37 tokens against 7 for English. On the newer o200k_base (about 200k vocabulary entries, against about 100k) it takes 11. One sentence per language is a demonstration, not a benchmark, so say that if you quote it. The practical point: newer encodings with larger, more multilingual vocabularies reduce the gap. They don't remove it.

In the raw output, the cl100k Hindi pieces print as broken characters. That's because some tokens are partial UTF-8 byte sequences, which is the byte-level fallback showing itself.

Follow-up: What would you do for a Hindi chatbot's cost model? Count tokens with the actual tokenizer of the model you'll call, on real user text, not on English estimates. Re-measure when you change models, because encodings differ. For rough rupee conversion across models, AI model pricing 2026 is the place to start.

Common wrong answer: "Tokens are roughly four characters or three quarters of a word." That heuristic is for English prose. Don't apply it to other scripts or code.

How does tokenization affect code?

Strong answer outline

• Identifiers split into subwords, so getUserAccountBalance costs more than a plain word.
• Whitespace matters. In the demo, def add(a, b):\n return a + b\n came to 12 tokens, with the four-space indent split as a three-space piece plus the fourth space attached to return. Languages and files with deep indentation can be costly, and tokenizers vary on how well they merge runs of whitespace.
• Leading-space variants (" return" versus "return") are different tokens, which is why prompts that end mid-word or without the expected trailing space can bias completions.

Follow-up: How would you shrink the token count of a large code prompt? Strip comments you don't need, drop blank lines, send diffs rather than files, and measure with the real tokenizer before and after. See prompt compression for the wider technique.

Common wrong answer: "Minifying code always saves tokens." Minified code can tokenize badly, since unusual character sequences have fewer merges. Measure it.

Run the tokenizer demo yourself

Install and run (Python 3; the first call downloads the encoding files, so it needs internet access):

import tiktoken

for name in ("cl100k_base", "o200k_base"):
    enc = tiktoken.get_encoding(name)
    for s in ("1234567", "strawberry", " strawberry", "सबसे नज़दीकी रेलवे स्टेशन कहाँ है?"):
        ids = enc.encode(s)
        print(name, len(s), len(ids), [enc.decode([i]) for i in ids])

Decode single IDs one at a time and you'll see the pieces. If some print as replacement characters, that's a token that holds part of a multi-byte character.

How do you choose an embedding model for retrieval?

Strong answer outline

• Don't pick from a leaderboard alone. MTEB covers 8 task types across 58 datasets and 112 languages, which makes it a useful starting filter, but your corpus and queries are not MTEB. Take 3 or 4 candidates and measure recall@k on a labelled set of your own questions.
• The checklist: language coverage (is your Hindi or code actually in training data?), maximum input length versus your chunk size, dimension and storage cost, price or hosting, latency, and whether the model needs query and document prefixes. Many retrieval models expect a different instruction for queries than for documents. Read the model card.
• Domain mismatch beats small score differences. A general model that gets 2 points more on a public benchmark can lose on your legal or medical text.
• Plan for change: keep raw text, store the model name and version with each vector, and budget a full re-embed when you switch. For a comparison of the current options, see embedding models comparison 2026, and for index choices, vector store guide.

Follow-up: What is Matryoshka representation learning, and why would you use it? Models trained this way (Kusupati et al. 2022) pack coarse information into the first dimensions, so you can truncate vectors to fewer dimensions to save storage and search time, at some quality cost. Verify that your specific model supports truncation, and re-normalise after cutting (see below).

Common wrong answer: "Use the highest MTEB score." Or: "Just use the biggest dimension." Neither tests your data.

What are the cosine similarity pitfalls?

Strong answer outline

1. Cosine ignores magnitude, dot product doesn't. For unit vectors they rank identically. Otherwise they can disagree. A toy case I computed, with the query at [1, 0]:

A (same direction, long)     dot= 10.00 cos=1.000 L2=  9.00
B (same direction, short)    dot=  0.50 cos=1.000 L2=  0.50
C (45 degrees, long)         dot=  8.00 cos=0.707 L2= 10.63

By cosine, A and B tie. By dot product A wins by a mile. By Euclidean distance B is nearest. Three metrics, three rankings. Use the metric the model was trained with, and normalise once if the index uses dot product.

2. Scores aren't comparable across models, queries or corpora. A cosine of 0.78 means nothing absolute. Don't hard-code a threshold like "above 0.8 is relevant" without calibrating on labelled pairs from your own data.

3. Truncation breaks normalisation. Taking the first 256 dimensions of a unit-length 1024-dimensional random vector gave a norm of 0.505, not 1. Re-normalise after truncating before using dot product.

4. High dimensions make random vectors nearly orthogonal. Random unit vectors, 1,000 pairs each:

d=    8 mean=+0.016 std=0.344 max|cos|=0.947
d=   64 mean=-0.011 std=0.123 max|cos|=0.394
d=  384 mean=+0.001 std=0.053 max|cos|=0.158
d= 1536 mean=+0.000 std=0.025 max|cos|=0.092

The spread shrinks roughly as one over the square root of the dimension. Real embeddings aren't random and often sit in a narrower cone, so unrelated texts can still score well above zero. The takeaway is to compare scores relative to each other within one model, not against an absolute scale.

5. Embeddings capture topical similarity, not logical relations. "The contract is valid" and "The contract is not valid" can be close. If negation or exact identifiers matter, add a reranker or hybrid keyword search. See how RAG works and the RAG lesson.

Follow-up: Your retriever returns near-duplicates with high scores. What do you do? Deduplicate at ingestion, apply maximal marginal relevance or a diversity step, and inspect whether chunk overlap is causing it.

Common wrong answer: "Cosine similarity of 1 means the texts mean the same thing." It means the vectors point the same way, within one model's space.

Practice checklist

• Run the tiktoken snippet on your own language and your own code. Write down two surprises.
• Compute a real cost: take 20 production-like prompts and count tokens under two encodings.
• Be able to explain, in two sentences each, BPE, WordPiece and Unigram, and name one place each is used. I'm confident about BERT for WordPiece and SentencePiece as the usual Unigram implementation. For specific current models, check the model card instead of guessing.
• Build a 30-question retrieval test for your own data before you argue about which embedding model is best.

What I could not verify

I ran the demos with tiktoken only, so the claims about other vendors' tokenizers (Claude, Gemini, Llama) are not tested here and each has its own vocabulary. The WordPiece description is from general knowledge of the BERT lineage, not from the original paper, which I did not open for this post. The cosine behaviour on real embedding models was not measured, only on random and toy vectors.