Research · Preprint, 2026
The Vocabulary Tax: Safely Trimming Unused Tokens from LLMs for On-Device Use
Paper (DOI) Plain-language explainerTax for 2,061 models Try AI Model Explorer
Abstract
Open large language models (LLMs) and speech models ship vocabularies of up to 262k tokens, and every user downloads the tables those tokens index (the input embedding, an untied output head and, in recent on-device models, per-layer embeddings), whatever languages they need. We call the bytes in rows a deployment never uses the vocabulary tax and measure it, without downloading weights, for 1,244 generative and speech-recognition models on the Hugging Face Hub. For a Thai and English deployment it is a median 4.4% of model bytes below 0.5B parameters and 2.1% above 8B, and 1.93 GB in Gemma 4 E2B: it falls hardest on the small models that edge devices run. Keeping only the rows of the deployment's writing systems is exact. Keeping the tokens seen in the user's own text removes much more, a median 29.0% below 0.5B, but the usual way of doing it breaks BPE tokenizers: it drops the intermediate tokens that merges build on, so new text is split differently 1.9 times as often as the standard check reports. We repair this with merge closure, at a cost of 5.9% more rows, and add a trim certificate: a Good–Turing estimate, from the trimming text alone, of how much new text the trimmed tokenizer will split differently (correlation 0.996 over 300 settings in 22 languages) and of how much more text a target needs. Without test data our method never exceeds a 2% target, while prior keep rules checked on test data exceed it in 9–13 of 44 settings. On six models of 270M to 3B parameters in 11 languages, trimmed models keep their predictions: where the certificate passes, a median 89.7% of rows is removed and bits per character rise by at most 2.0%. The census, the trimming and the certificate are released as an open Python package, and the per-model tax for 41 languages as an open dataset.
Key findings
- For a Thai and English deployment the unused rows are a median 4.4% of model bytes below 0.5B parameters and 2.1% above 8B; 1.93 GB in Gemma 4 E2B.
- Trimming with the user's own text removes a median 29.0% below 0.5B, but the usual way breaks BPE tokenizers: new text is split differently 1.9 times as often as the standard check reports.
- Merge closure repairs it for 5.9% more rows; the trim certificate predicts the real re-split rate from the trimming text alone (correlation 0.996, 300 settings, 22 languages).
- On six models from 270M to 3B in 11 languages, when the certificate passes, a median 89.7% of rows go and bits per character rise by at most 2.0%.
Code, data and tools
- In the browser: Where to cut in AI Model Explorer tokenizes text you paste with the model's own tokenizer, without uploading it.
- Python package and CLI:
pip install ai-model-explorer(PyPI), thenai-model-explorer tax MODEL --lang th --corpus my_text.txt --export out/. - The vocabulary tax section of the Hub census: rows and bytes per model for any of 41 languages.
- Dataset on Hugging Face: ai-vocabulary-tax.
- Source code (MIT).
Where it comes from
This paper follows Reading the Architecture off the Weights, which reads a model's anatomy from its weight headers, and takes its token tables and model sizes from that paper's census of 12,974 Hugging Face repositories. Both grew out of the workshop “Squeeze Until It Breaks: A Hands-On Edge Inference Contest” at AI Expo Thailand 2026.
Cite
@misc{singkul2026vocabulary,
title = {The Vocabulary Tax: Safely Trimming Unused Tokens from LLMs for On-Device Use},
author = {Singkul, Sattaya},
year = {2026},
note = {Preprint},
howpublished = {ResearchGate},
doi = {10.13140/RG.2.2.10665.15200},
url = {https://doi.org/10.13140/RG.2.2.10665.15200}
}