The vocabulary tax: the token tables your users never touch, and how to trim them safely
How much of a small LLM is vocabulary a Thai deployment never uses, why frequency trimming breaks BPE tokenizers, and how to trim safely.
A Thai and English chatbot built on Gemma 4 E2B downloads 1.93 GB of rows it can never use. Those rows sit in the token tables, the input embedding, the output head and the per-layer embeddings, and they belong to writing systems the deployment never sees. My new paper, The Vocabulary Tax, measures that cost across the Hugging Face Hub without downloading any weights. It also shows that the usual way of trimming it breaks BPE tokenizers, fixes that, and adds a trim certificate: a number, computed from your own text before you touch the model, that says how much of your future input the trimmed tokenizer will split differently.
It continues the work behind my workshop at AI Expo Thailand 2026. This post walks through the idea with diagrams, the numbers that matter and where I think it can go next.
สรุปภาษาไทย: หลังจบงาน AI Expo Thailand 2026 ผมต่อยอดคำถามจาก workshop “Squeeze Until It Breaks” ว่าจะบีบโมเดลตรงไหนก่อน เปเปอร์ใหม่ชื่อ The Vocabulary Tax วัดว่าโมเดล AI เก็บตาราง token ของภาษาที่ผู้ใช้ไม่เคยใช้ไว้มากแค่ไหน วัดจาก header ของไฟล์โดยไม่ต้องดาวน์โหลด weights ครอบคลุม 1,244 โมเดลบน Hugging Face ถ้าใช้งานแค่ภาษาไทยกับอังกฤษ ส่วนที่ไม่ได้ใช้คิดเป็นค่ากลาง 4.4% ของขนาดโมเดลที่เล็กกว่า 0.5B และ 2.1% ของโมเดลที่ใหญ่กว่า 8B ใน Gemma 4 E2B มีถึง 1.93 GB โมเดลเล็กที่ใช้บนมือถือจึงเสียมากที่สุด การตัดตามระบบตัวเขียน (อักษรไทย + ละติน) ปลอดภัยแน่นอน ส่วนการตัดตามข้อความจริงของผู้ใช้ตัดได้มากกว่ามาก แต่วิธีที่ใช้กันทั่วไปทำให้ tokenizer แบบ BPE ตัดคำผิดไปจากเดิม ผมจึงแก้ด้วย merge closure และเพิ่ม trim certificate ที่บอกล่วงหน้าว่าข้อความใหม่จะถูกตัดคำต่างไปเท่าไร และต้องใช้ข้อความเพิ่มอีกกี่บรรทัด ทดสอบกับ 6 โมเดลขนาด 270M–3B ใน 11 ภาษา เมื่อ certificate ผ่าน ตัดแถวออกได้ค่ากลาง 89.7% โดยคุณภาพแทบไม่เปลี่ยน ใช้ได้ทั้งบนเว็บ AI Model Explorer (หัวข้อ Where to cut) และผ่าน pip install ai-model-explorer
From the workshop to a second paper
My session at AI Expo Thailand 2026 was Squeeze Until It Breaks: A Hands-On Edge Inference Contest (Saturday 7 November, Builder Stage). The idea was simple: take a model, compress it for a device, and keep going until it fails. Before you can squeeze well, you need to know where the bytes are. That question produced AI Model Explorer and my first paper, which reads a model’s anatomy from its weight headers.
Once you can see the parts, one part stands out in small multilingual models. The vocabulary is huge, and every table indexed by it grows with it. So the follow-up question was obvious: how much of a model is vocabulary that a given deployment never uses, and can you remove it without breaking anything?
- Before the expoAI Model ExplorerBuilt so workshop participants could see the parts of a speech model before squeezing it.
- 2 and 6 NovPaper 1 and the Hub censusAnatomy from headers alone; 12,974 repositories read.
- 7 NovWorkshop at AI Expo ThailandSqueeze Until It Breaks, Builder Stage.
- 15 NovPaper 2: the vocabulary taxWhich rows your languages never use, and how to remove them safely.
What the vocabulary tax is
Open models now ship vocabularies of up to 262k tokens. They have to: one checkpoint is meant to serve dozens of languages. But every table indexed by the vocabulary has one row per token:
- the input embedding, which turns a token id into a vector;
- the output head, which scores every token at each step, when it isn’t tied to the input embedding;
- per-layer embeddings, used by recent on-device models such as Gemma 3n and Gemma 4, which keep one extra table per layer.
A deployment that serves Thai and English still downloads the rows for Cyrillic, Arabic, Devanagari, Hangul and the rest. I call those bytes the vocabulary tax. Formally, it’s the bytes of the rows a deployment’s keep rule removes, as a share of the stored model.
Tap the screenshot to open it at full size.
- 1Pick the languages you serve. Thai and English are on here.
- 2Optional: paste your own text to trim further. It is tokenized in your browser and never uploaded.
- 3The script rule removes 964.3 M weights, 19% of the model. The paper reports 18.8% of stored bytes (1.93 GB) for the same model.
- 4Why it is so large: the per-layer embedding alone is 2.35 B weights, 46% of the model, and it has one row per token.
The tax can be computed without downloading weights. The tensor manifest in the safetensors header gives the shape and storage type of every token table, and the tokenizer file says which piece each row belongs to. That’s exactly what the census from the first paper already reads, so the paper measures the tax for 1,244 generative and speech-recognition models in one pass.
Who pays it: small models, and the languages they serve
The tax is regressive. For a Thai and English deployment, the median tax is:
| Model size | Median tax (share of model bytes) |
|---|---|
| below 0.5B parameters (116 models) | 4.4% |
| 0.5–2B | 4.3% |
| 2–8B | 2.8% |
| above 8B (628 models) | 2.1% |
The reason is structural. Token tables grow with the vocabulary times the width. Transformer layers grow with the width squared times the depth. So in a small model the tables are a much bigger slice: a median 41.2% of model bytes below 0.5B, against 13.1% across all 1,244 models. The 90th percentile below 0.5B pays 22.1%.
At Hub scale this adds up. Qwen3 0.6B was downloaded 23 million times in the month before the census, and each copy carried 0.18 GB of rows that a Thai and English deployment never uses.
- Gemma 3 270M0.54 GB21.9%21.9% → 55.4%
- Qwen3 0.6B1.50 GB11.9%11.9% → 35.8%
- Gemma 3 1B2.00 GB10.6%10.6% → 26.7%
- Qwen3 1.7B4.06 GB8.8%8.8% → 26.5%
- Gemma 4 E2B10.25 GB18.8%18.8% → 47.5%
- Ministral 3 3B4.67 GB4.7%
- Whisper large-v3-turbo1.62 GB1.2%
script rule + 15,000 lines of your text
A memory tax or a compute tax: the tokenizer decides
There’s a trade-off hiding here. A tokenizer with a small, English-centred vocabulary wastes almost no memory, but it splits Thai into many more tokens, and every extra token costs compute and context. GPT-2’s vocabulary leaves only 0.5% of rows unused by Thai and English, but a Thai sentence needs 8.29 times as many tokens as its English translation. Gemma’s leaves 35.0% unused at 1.53 times. GPT-OSS’s o200k sits between them at 27.1% and 1.84 times. No family in the census avoids both. A Thai user pays a compute tax or a memory tax, and the tokenizer picks which.
The tax barely depends on which language you keep: below 0.5B it is 4.4–4.5% for 35 of 40 languages. The exceptions are telling. For Lao and Punjabi the script rule keeps no more rows than for English alone, because the reference vocabularies hold hardly any pieces in those scripts. Their speakers don’t pay the memory tax. They pay the compute tax in full.
Rule 1: keep your writing systems. It’s exact
The safest trim keeps every piece whose characters all belong to your scripts, plus Latin, plus the shared character classes (the Unicode Common and Inherited scripts, and format characters such as zero-width spaces). Special tokens, byte fallbacks, digits and punctuation are always kept.
Text in those scripts then tokenizes exactly as before, as long as the tokenizer is rebuilt consistently. The paper checks this:
- Over 189 language and tokenizer pairs, at most 0.008% of held-out tokens fall in removed rows, and those are rare symbols outside any declared script.
- After a full rebuild and trim to Thai and English for five tokenizer families, 100% of sentences tokenize identically and round-trip back to the input.
Latin is not optional. Thai text is full of Latin: names, numbers, URLs, English words. Trimming to the Thai script alone removes 91–95% of rows instead of 12–35%, but 3.5–6.6% of the tokens in Thai text are then lost. With Latin kept, none are.
Each inclusion in the rule fixes a failure I actually hit. The Japanese long-vowel mark and the Arabic tatweel belong to the Common script. Combining marks belong to Inherited. Zero-width spaces are format characters.
Rule 2: keep what your users write. It removes far more, and the usual way breaks
If you have text from your users, you can go much further: keep only the pieces that appear when you tokenize it. Several papers and tools do exactly this. With 15,000 lines of Thai, the median removable share below 0.5B rises from 7.9% to 29.0%, and Gemma 4 E2B can lose 4.87 GB of its 10.25 GB.
But there’s a trap, and it’s specific to BPE, the tokenizer type most open models use. In BPE, every piece except a single character is built by merging two smaller pieces. Many of those smaller pieces never appear as tokens in real text, because they always get merged further. A rule that keeps “pieces seen as tokens” removes them. Every piece built on them becomes unreachable.
The word “there” after rebuilding the tokenizer
The orphan’s row is still in the model, so it still costs memory, but the rebuilt tokenizer can never select it. Words that used it fall apart into smaller pieces. The usual check misses this. It counts held-out tokens whose row was removed (the nominal rate), not how much text actually gets split differently after the rebuild (the real rate).
Measured across 22 languages and four BPE families (Gemma, Qwen, Llama 3 and GPT-2):
- A median 828 kept rows per setting (5.0%) are orphans.
- The real re-split rate is a median 1.9 times the nominal rate, and more than 1.2 times in 95 of 101 measurable settings.
- For German on Qwen, the nominal rate is 2.4% but the real rate is 7.1%. For English web text, the real rate is 23–27% against a nominal 9%, and sequences grow by 49–62%.
The fix: merge closure
The repair is to close the keep set under merges: for every kept piece, also keep the two pieces it was merged from, recursively. The parents come from the merge list in tokenizer.json or GGUF. For tiktoken and Tekken vocabularies, which store ranks instead of merges, they come from the last merge BPE performs when it encodes the piece on its own.
Closure costs a median 5.9% more rows, a small price next to the 80–90% that the corpus rule removes. With it, the real re-split rate falls to a median 0.94 of the nominal rate, and sequences grow by a median 0.9% instead of 2.5%. Against the keep rules of prior work, at the same compression, the others re-split 1.30–1.32 times as much new text, and only ours keeps no unreachable rows.
The trim certificate: know before you deploy
Merge closure makes the trim correct. The remaining question is whether you had enough text. If your sample missed pieces your users will need, their words get split into smaller pieces. The model still works, but on input it saw less often.
Holding out part of your text to test this costs text and, as it turns out, overestimates the error. Instead, the certificate estimates it from the trimming text itself, using an old idea from statistics: the Good–Turing estimate. It says that the chance the next token is something you’ve never seen is about the share of tokens you’ve seen exactly once. With the default minimum count of one, the certificate is simply:
estimated re-split rate = (tokens of pieces seen exactly once) ÷ (all tokens)
To say how much more text a target needs, it computes the same estimate on the first half of your text and on all of it, fits the decay (the form Heaps’ law gives for Zipf-like text) and extrapolates. It runs in the time it takes to tokenize your text.
How well it works:
- Across 300 settings (22 languages, four families, 1k–15k lines), the estimate agrees with the real rate after rebuilding with correlation 0.996, and falls on the right side of a 1% threshold in 297 of 300.
- Its median relative error is 4.5–6.6%. A 20% hold-out gets 22.6–25.9%, and five-fold cross-validation about 25–27%.
- The deployed comparison is the one that matters. Users of prior rules check the nominal rate on held-out test text and keep the most aggressive setting that passes. At a 2% target, they overshoot in 9 to 13 of 44 settings. Choosing by the certificate, with no test text at all, overshoots in 0 of 44 while removing about as many rows (68.2% against 68.1–68.5%).
How much text you need depends on the language, by two orders of magnitude:
- Arabic
- Persian
- Hebrew
- Hindi
- Korean
- Russian
- Ukrainian
- Thai
- Japanese
- Spanish
- Polish
- Czech
- Italian
- Portuguese
- German
- Turkish
- Indonesian
- Dutch
- French
- English (web)
The order follows how much of a language’s vocabulary it shares with English. Pieces in a script used by few languages are few and frequent, so a few thousand lines see nearly all of them. Latin pieces are numerous and spread across every Latin-script language. English, whose pieces are the bulk of most vocabularies, is the extreme.
Needs only the model id
- 1Read the headersTensor manifest and tokenizer file, a few kilobytes.
- 2Price the taxRows and bytes your languages never use, per table.
- 3Script ruleKeep your scripts plus Latin. Exact.
Uses your own text
- 4Corpus ruleKeep the pieces your text actually uses.
- 5Merge closureAdd back the parents of every kept piece.
- 6Trim certificatePredicted re-split rate, and how many lines you still need.
Before you ship
- 7Rebuild and sliceNew tokenizer, kept ids in order, every table sliced with one index.
- 8Check 200 real linesMeasure on text from the deployment itself, to catch domain shift.
Does the trimmed model still work?
Tokenizer experiments are only half the story, so the paper also runs six edge-sized models with different tokenizers (Gemma 3 270M and 1B, Qwen3 0.6B and 1.7B, Llama 3.2 1B, SmolLM3 3B) in 11 languages. A trimmed model can be evaluated exactly without editing any weights: map the trimmed tokenizer’s ids to the original rows and take every softmax over the kept rows only.
- Script rule: bits per character change by between −0.71% and +0.55%, at least 93% of greedy continuations are identical to the original model’s (median 100%), and Belebele answers are identical in 61 of 66 cases.
- Corpus rule, where the certificate passes (68 settings): a median 89.7% of rows removed, bits per character change by a median −0.65% and at most +1.99%, and a median 91% of greedy continuations are identical.
- Corpus rule, where it asks for more text (54 settings): the median change is +3.41%, and the worst case, English with 4,000 lines, reaches +23.3%. This is the certificate doing its job.
The change in bits per character follows the re-split rate almost linearly, roughly 2.5% per percent of re-split tokens. So the certificate, computed before any model is run, also predicts the quality cost.
One caution I didn’t expect. SmolLM3 shares Llama 3’s tokenizer, yet where the re-split rate is under 0.2% its bits per character drop by 8–14% after trimming, against at most 2.3% for Llama 3.2 1B. Trimming renormalises the model over the kept rows. SmolLM3 put a lot of probability on pieces that never occur in the target language, so removing them looks like an improvement. It isn’t: Belebele accuracy doesn’t rise. Read a drop in bits per character together with the re-split rate.
Does it change what fits on a device?
For weight files, yes. Of the 182 edge-sized models (under 4B parameters) in the paper’s population that exceed a 1 GB budget at their stored precision, 3 fit after the script rule and 25 after the corpus rule. At INT4 and 1 GB, 10 and 21 of 54. Qwen3 0.6B (1.50 GB) fits in 1 GB only after the corpus rule, and so does Qwen3 1.7B at INT4.
These are bytes on disk. Runtimes that memory-map weights only page in the embedding rows they touch, so for them the input embedding costs download size more than RAM. The output head is different: it’s read in full at every decoding step, so trimming an untied head saves memory bandwidth and compute too.
Which rule should you use?
| Your situation | Use | What the paper measured |
|---|---|---|
| No text from your users | Script rule | Exact for your scripts; bits per character moved by at most 0.71% |
| Text, certificate below your target | Corpus rule + merge closure | A median 89.7% of rows removed; bits per character at most +1.99% |
| Text, certificate above your target | Add the lines it asks for, or fall back to the script rule | Uncertified trims: median +3.41%, worst +23.3% |
| Any trim, before release | Check 200 lines of real deployment text | Domain shift raised the real rate a median 1.7× |
The last box matters. The certificate describes new text from the same source as your trimming text. When the paper trained the keep set on one domain and tested on another, the real rate was a median 1.7 times higher, up to 3.4 times for Korean. The cheap guard is to check the nominal rate on just 200 lines of real deployment text. With a merge-closed keep set, that number is honest: a median 0.86 of the real rate on 1,000 lines. Without closure, it would understate the damage about twofold.
Most real deployments are mixed. For Thai with Chinese and English, the script-rule tax below 0.5B falls to 3.3%, because Chinese pieces are a big part of most vocabularies. The pooled certificate can hide a weak language (French reaches 8.1% next to Arabic’s 0.5% on Gemma), so it’s computed per language too, and it predicts each language’s real rate with a median ratio of 1.03.
The potential: where I think this goes
The paper measures. This section is my view of what it opens up, so read it as direction rather than result.
- Regional on-device AI. Southeast Asian apps usually serve one or two local languages plus English. The tax hits exactly the small models those apps can afford to run, and Thai needs only about 4,000 lines of real text for a certified trim. A phone app, a kiosk or an in-car assistant can ship a smaller model with no retraining.
- Faster downloads and updates. Even where RAM is not the limit, a quarter to half off the file (26–55% for the small reference models above, with 15,000 Thai lines) means faster installs and over-the-air updates, and less storage on cheap devices.
- Call centres and on-prem servers. A server that handles a fixed set of languages doesn’t need to stream an untied output head over the whole vocabulary at every step. That is bandwidth and compute on the hottest path.
- A pre-flight check for model choice. Because the tax is read from headers, you can compare candidate models before downloading any of them. The memory tax against compute tax trade-off becomes a number you can read off, per language.
- Draft models for speculative decoding. Smaller vocabularies already help draft models (Ben Shoham, 2026). Merge closure and the certificate apply there unchanged.
- The other side: low-resource scripts. Lao, Punjabi, Amharic, Burmese and Khmer pay almost no memory tax because their scripts are barely in the vocabularies. They pay in sequence length instead. The same census can show which models treat which language fairly, which is useful data for anyone building or extending a vocabulary.
Try it on your own model
On the web, open AI Model Explorer and use Where to cut. It tokenizes text you paste in the browser with the model’s own tokenizer, without uploading it. The census page lists the tax for 2,061 models in any of 41 languages.
Tap the screenshot to open it at full size.
- 1Choose your language. English is always kept alongside it.
- 2Switch the keep rule: script rule, or your own text at 4,000 or 15,000 lines.
- 3Share of the model you never use. Qwen3-0.6B: 37.3% with 4,000 lines of text.
- 4The price: share of words the trimmed tokenizer splits differently at that setting.
From the command line:
pip install ai-model-explorer
ai-model-explorer tax Qwen/Qwen3-0.6B --lang th --corpus my_text.txt --export out/
From Python:
from ai_model_explorer import explore
explore("Qwen/Qwen3-0.6B").vocab_tax(["th"], corpus=["my_text.txt"]) # rows and bytes you never use, plus the certificate
--export writes the trimmed tokenizer and the list of kept ids, so the token tables can be sliced with one indexing operation. Every table behind the paper is in the ai-vocabulary-tax dataset.
What it doesn’t cover yet
- Model size and quality. Quality is measured without retraining on models up to 3B, on text from the same source as the trimming text. Larger models, human judgement of generation and fine-tuning after trimming are not covered, and Belebele has little power at this scale.
- Domain shift. The certificate is for in-domain text. Keep the 200-line check.
- Newest tokenizers. Families distributed only through the Hub (Qwen3.5 and later, Nemotron-H, GLM-5, DeepSeek-V3.2 and V4) are not in the census populations. The package computes their tax directly from their own tokenizer files.
- Other tokenizer types. The corpus experiments cover four BPE families. The package also closes tiktoken, Tekken and SentencePiece-BPE vocabularies, but those extensions aren’t part of the measured experiments.
- Encoders and multimodal towers are outside the populations studied.
If you use this in research, please cite the paper.
References
- Singkul, S. The Vocabulary Tax: Safely Trimming Unused Tokens from LLMs for On-Device Use. Preprint, 2026. doi:10.13140/RG.2.2.10665.15200. All figures in this post come from this paper.
- Singkul, S. Reading the Architecture off the Weights: Manifest-Only Model Anatomy and a Census of 12,974 Hugging Face Repositories. Preprint, 2026. doi:10.13140/RG.2.2.19473.19048.
- Singkul, S. AI Model Explorer, Python package, source code and the ai-vocabulary-tax dataset. 2026.
- AI Expo Thailand. Agenda, Day 3, Builder Stage. 2026.
- Sennrich, R., Haddow, B. and Birch, A. Neural Machine Translation of Rare Words with Subword Units. ACL 2016.
- Kudo, T. and Richardson, J. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. EMNLP 2018 System Demonstrations.
- Good, I. J. The population frequencies of species and the estimation of population parameters. Biometrika 40(3–4):237–264, 1953.
- Gale, W. A. and Sampson, G. Good-Turing frequency estimation without tears. Journal of Quantitative Linguistics 2(3):217–237, 1995.
- Heaps, H. S. Information Retrieval: Computational and Theoretical Aspects. Academic Press, 1978.
- Ushio, A., Zhou, Y. and Camacho-Collados, J. Efficient Multilingual Language Model Compression through Vocabulary Trimming. Findings of EMNLP 2023.
- Bogoychev, N., Chen, P., Haddow, B. and Birch, A. The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language and by Frequency. Workshop on Insights from Negative Results in NLP, 2024.
- SqueezeBits. Vocabulary trimming: an easy and effective method for SLM acceleration. 2025.
- Petrov, A., La Malfa, E., Torr, P. H. S. and Bibi, A. Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2023.
- Ahia, O. et al. Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models. EMNLP 2023.
- Land, S. and Bartolo, M. Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models. EMNLP 2024.
- Ben Shoham, O. Balancing coverage and draft latency in vocabulary trimming for faster speculative decoding. arXiv:2603.05210, 2026.
- Bandarkar, L. et al. The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants. ACL 2024.
- The Unicode Consortium. Unicode Standard Annex #24: Unicode Script Property.
- Hugging Face. safetensors. 2022.

