穿越越南高地、三角洲与海岸的回响：面向占语、高棉语和岱依-侬语的多语言语料库
英文标题：Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nung
越南少数民族语言在NLP领域近乎空白，该工作推出了首个涵盖占语、高棉语和岱依-侬语的CKTN语料库与基准（44,367篇文档、2,400万子词标记）。研究发现现有多语言编码器对这些语言碎片化严重，且常用适配指标可能产生误导：模型可降低语言建模损失或在词形重叠检索上表现优异，却仍无法跨文档语义泛化。论文提出一种脚本感知适配方案（词汇增强+校准替换标记预训练），有效减少碎片化并取得最强分类性能，揭示了仅依赖词形重叠检索作为评估信号的局限性。
2.6 KB
2.1 KB
640 B
612 B
380 B
9.1 KB
32.7 KB
22.8 KB
24.2 KB
34.7 KB
中文
[提交于 2026年7月9日]
标题：穿越越南高地、三角洲与海岸的回响：面向占语、高棉语和岱依-侬语的多语言语料库
查看本文 PDF：Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nung，作者：Anh Trac Duc Dinh 等 4 位作者
摘要：越南的少数民族语言在自然语言处理（NLP）领域几乎完全缺席，其挑战不仅在于数据稀缺：占语（Cham）、高棉语（Khmer）和岱依-侬语（Tay-Nung）在书写系统、与越南语接触程度以及标准化程度上差异显著，这导致标准的多语言适配可能学到错误信号。我们推出 CKTN，这是首个针对这些语言的语料库与基准（44,367 篇文档，2,400 万子词标记），涵盖持续预训练、类别分类以及摘要-文档检索三项任务。我们发现现有多语言编码器对这些语言存在严重碎片化，且常见的适配指标可能产生误导：模型或许能降低语言建模损失，或在词形重叠检索中表现优异，但仍无法跨文档实现语义泛化。我们通过一种脚本感知的适配方法来解决这一问题——将词汇增强与校准后的替换标记预训练相结合——以此防止判别器利用浅层的脚本不匹配。最终得到的编码器碎片化显著降低，并在所评估模型中取得最强的分类性能，由此暴露了词形重叠检索作为评估信号的局限性。
提交历史
来自：Khang Vo Nhat Hoang [
]
[v1] 2026年7月9日 星期四 11:20:17 UTC（3,754 KB）
English / 原文
[Submitted on 9 Jul 2026]
Title:Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nung
View a PDF of the paper titled Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nung, by Anh Trac Duc Dinh and 4 other authors
Abstract:Vietnam's ethnic minority languages are almost absent from the field of Natural Language Processing (NLP), and the challenge goes beyond data scarcity: Cham, Khmer, and Tay-Nung differ sharply in script, Vietnamese contact, and standardization, conditions under which standard multilingual adaptation can learn the wrong signals. We introduce CKTN, the first corpus and benchmark for these languages (44,367 documents, 24M subword tokens), spanning continued pretraining, category classification, and summary-document retrieval. We show that existing multilingual encoders severely fragment these languages, and that common adaptation metrics can mislead: models may lower language-modeling loss or excel at lexical-overlap retrieval while still failing at semantic generalization across documents. We address this with a script-aware adaptation recipe - vocabulary augmentation combined with calibrated replaced-token pretraining - that prevents the discriminator from exploiting trivial script mismatches. The result is an encoder with substantially less fragmentation and the strongest classification performance among evaluated models, exposing the limits of lexical-overlap retrieval as an evaluation signal.
Submission history
From: Khang Vo Nhat Hoang [
]
[v1]
Thu, 9 Jul 2026 11:20:17 UTC (3,754 KB)