EEG 与文本的语义对齐:用于想象语音解码的对比学习框架
英文标题:Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech
该研究提出将 EEG 想象语音试次与冻结的预训练文本编码器嵌入空间进行对齐,以支持更具语义灵活性的脑机接口解码。研究者结合语义重加权的 EEG—句子对齐目标和基于文本原型的类别判别目标,在 39 个意图类别之间兼顾语义结构与可分性。该方法在 Chisco 想象语音数据集的 5 名受试者上取得了与任务优化的闭集分类器相当的解码表现。MaxCosine 指标能够筛选出 Top-1 和 Top-5 准确率更高的留出试次,而语义轴评估和 Semantic Gallery Probe 则表明,该框架可支持未见候选句子的相似度检索与更广泛的语义结构分析。
612 B
380 B
9.1 KB
32.7 KB
22.8 KB
24.2 KB
34.7 KB
中文
[提交于 2026 年 10 月 14 日]
标题:EEG 与文本的语义对齐:用于想象语音解码的对比学习框架
查看由 Shahryar G. Namdari 及另外 3 位作者撰写的论文《EEG 与文本的语义对齐:用于想象语音解码的对比学习框架》的 PDF
摘要:基于 EEG 的想象语音解码为构建更自然的脑机接口提供了一条很有前景的路径。传统的闭集分类器将语义类别视为彼此独立的标签,未能明确利用语言意图之间的关系,因此限制了输出的灵活性。我们将一个多模态对比学习框架应用于想象语音任务,在冻结的预训练文本编码器嵌入空间中对齐 EEG 试次与语言表征。研究训练了一个 EEG 编码器,其复合优化目标结合了经语义重加权的 EEG—句子对齐,以及基于文本原型的类别判别,从而在保留 39 个意图类别可分性的同时促进语义结构的形成。我们在涵盖 5 名受试者的 Chisco 想象语音数据集上评估该方法,报告 Top-k 准确率、宏平均 F1 值以及平衡两两匹配准确率(2v2)。我们还研究了最大 EEG—文本余弦相似度(MaxCosine)能否作为解码可靠性的内在指标,并通过 5 个评估时语义轴,以及使用此前未见候选句子的定性语义画廊探测(Semantic Gallery Probe,SGP),考察超越类别标签的语义结构。该对比学习框架在将 EEG 映射到冻结语言嵌入空间的同时,实现了与面向任务优化的闭集分类器相当的解码性能。MaxCosine 在不同受试者中识别出了 Top-1 和 Top-5 准确率更高的留出试次子集。评估时语义轴在全部 5 个维度上的平均一致率均高于 50% 的随机参考水平,SGP 则定性展示了与未见候选句子进行直接比较的能力。综合来看,这些结果表明,EEG—语言对齐能够在保持可比的固定意图解码能力的同时,支持基于相似度的检索、内在可靠性排序,以及对更广泛语义结构的评估。
提交历史
来自:Shahryar Namdari Ghareghani [
]
[v1]
2026 年 10 月 14 日,星期三,18:14:36 UTC(478 KB)
English / 原文
[Submitted on 14 Oct 2026]
Title:Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech
View a PDF of the paper titled Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech, by Shahryar G. Namdari and 3 other authors
Abstract:Imagined speech decoding from EEG offers a promising route toward naturalistic brain-computer interfaces. Traditional closed-set classifiers treat semantic categories as independent labels and do not explicitly exploit relationships among linguistic intents, limiting output flexibility. We adapt a multimodal contrastive learning framework to imagined speech, aligning EEG trials with language representations in the embedding space of a frozen pretrained text encoder. An EEG encoder is trained with a composite objective combining semantically reweighted EEG-sentence alignment and text-prototype-based class discrimination, encouraging semantic structure while preserving separability among 39 intent categories. We evaluate the approach on the Chisco imagined speech dataset across five subjects, reporting Top-k accuracy, macro-F1, and balanced pairwise matching accuracy (2v2). We also investigate maximum EEG-text cosine similarity (MaxCosine) as an intrinsic indicator of decoding reliability and probe semantic structure beyond category labels using five evaluation-time semantic axes and a qualitative Semantic Gallery Probe (SGP) with previously unseen candidate sentences. The contrastive framework achieved decoding performance comparable to a task-optimized closed-set classifier while mapping EEG into a frozen language embedding space. MaxCosine identified subsets of held-out trials with higher Top-1 and Top-5 accuracy across subjects. Evaluation-time semantic axes showed mean agreement above the 50% chance reference across all five dimensions, and the SGP qualitatively illustrated direct comparison with unseen candidate sentences. Together, these findings suggest that EEG-language alignment can retain comparable fixed-intent decoding while supporting similarity-based retrieval, intrinsic reliability ranking, and evaluation of broader semantic structure.
Submission history
From: Shahryar Namdari Ghareghani [
]
[v1]
Wed, 14 Oct 2026 18:14:36 UTC (478 KB)