摘要：现代多语言 tokenizer 往往会比处理英语时更严重地切分乌克兰语及其他代表性不足的西里尔字母语言，从而造成成本和上下文容量方面的差异。我们量化了这一开销，覆盖 9 种生产环境 tokenizer、5 种具有标准化西里尔字母和拉丁字母表示形式的语言，以及 837 万个词形。在语料库基准测试中，依据 BrUK 和 Brown 语料库上的全文 fertility 测量，乌克兰语在现代 tokenizer 中表现出 68%—121% 的 token 开销，在较旧的 cl100k 中则为 220%。在拥有独立验证英语基线的子集中，该开销与西里尔字母词表分配呈负相关，但这种关联不具有统计显著性（Spearman rho = -0.536，p = 0.215，n = 7）。我们评估了两种缓解策略。在包含 1,536 件商品和 145 个查询的电商 RAG 基准中，LLMLingua-2 可将乌克兰语输入长度减少 47%—49%，且在 80 个可检索案例中未出现由压缩导致的价值损失。一种采用 20 万词表上限训练、实际收敛至 158,184 个词条的均衡字节级 BPE tokenizer，可将留出集上的乌克兰语/英语 token 比从 2.22 倍降至 1.30 倍。对于大多数 tokenizer，罗马化会使乌克兰语 token 数增加 2%—19%。在这 5 种语言中，分词效率更偏向于网络数据中更常见的文字系统。这些发现表明，训练数据分配会造成西里尔字母分词开销，并且可在推理和 tokenizer 设计阶段采取缓解措施。

Abstract:Modern multilingual tokenizers often fragment Ukrainian and other underrepresented Cyrillic-script languages more heavily than English, creating disparities in cost and context capacity. We quantify this overhead across nine production tokenizers and five languages with standardized Cyrillic and Latin representations, covering 8.37 million word forms. On a corpus benchmark, Ukrainian shows 68-121% token overhead on modern tokenizers and 220% on the older cl100k, measured through full-text fertility on the BrUK and Brown corpora. Overhead is negatively associated with Cyrillic vocabulary allocation in the subset with independently verified English baselines, although the association is not statistically significant (Spearman rho = -0.536, p = 0.215, n = 7). We evaluate two mitigation strategies. LLMLingua-2 reduces Ukrainian input length by 47-49% on an e-commerce RAG benchmark of 1,536 products and 145 queries, with no compression-induced value losses among 80 retrievable cases. A balanced byte-level BPE tokenizer trained with a 200K vocabulary cap, converging at 158,184 actual entries, reduces the held-out UK/EN ratio from 2.22x to 1.30x. Romanization increases Ukrainian token counts by 2-19% on most tokenizers. Across the five languages, tokenization efficiency favors the script more prevalent in web data. These findings indicate that training data allocation contributes to Cyrillic tokenization overhead and that mitigation is possible at both inference and tokenizer-design stages.