Research · Preprint, 2026

The Vocabulary Tax: Safely Trimming Unused Tokens from LLMs for On-Device Use

Sattaya Singkul (สัตยา สิงห์กุล), sole author · DOI 10.13140/RG.2.2.10665.15200

Paper (DOI) Plain-language explainerTax for 2,061 models Try AI Model Explorer

Abstract

Open large language models (LLMs) and speech models ship vocabularies of up to 262k tokens, and every user downloads the tables those tokens index (the input embedding, an untied output head and, in recent on-device models, per-layer embeddings), whatever languages they need. We call the bytes in rows a deployment never uses the vocabulary tax and measure it, without downloading weights, for 1,244 generative and speech-recognition models on the Hugging Face Hub. For a Thai and English deployment it is a median 4.4% of model bytes below 0.5B parameters and 2.1% above 8B, and 1.93 GB in Gemma 4 E2B: it falls hardest on the small models that edge devices run. Keeping only the rows of the deployment's writing systems is exact. Keeping the tokens seen in the user's own text removes much more, a median 29.0% below 0.5B, but the usual way of doing it breaks BPE tokenizers: it drops the intermediate tokens that merges build on, so new text is split differently 1.9 times as often as the standard check reports. We repair this with merge closure, at a cost of 5.9% more rows, and add a trim certificate: a Good–Turing estimate, from the trimming text alone, of how much new text the trimmed tokenizer will split differently (correlation 0.996 over 300 settings in 22 languages) and of how much more text a target needs. Without test data our method never exceeds a 2% target, while prior keep rules checked on test data exceed it in 9–13 of 44 settings. On six models of 270M to 3B parameters in 11 languages, trimmed models keep their predictions: where the certificate passes, a median 89.7% of rows is removed and bits per character rise by at most 2.0%. The census, the trimming and the certificate are released as an open Python package, and the per-model tax for 41 languages as an open dataset.

Key findings

Code, data and tools

Where it comes from

This paper follows Reading the Architecture off the Weights, which reads a model's anatomy from its weight headers, and takes its token tables and model sizes from that paper's census of 12,974 Hugging Face repositories. Both grew out of the workshop “Squeeze Until It Breaks: A Hands-On Edge Inference Contest” at AI Expo Thailand 2026.

Cite

@misc{singkul2026vocabulary,
  title  = {The Vocabulary Tax: Safely Trimming Unused Tokens from LLMs for On-Device Use},
  author = {Singkul, Sattaya},
  year   = {2026},
  note   = {Preprint},
  howpublished = {ResearchGate},
  doi    = {10.13140/RG.2.2.10665.15200},
  url    = {https://doi.org/10.13140/RG.2.2.10665.15200}
}