摘要：扑克 Agent 可能会做出符合规则的合法出牌，但在策略上却可能非常糟糕。我们探究了紧凑型 AI Agent 是否能在隐藏信息环境下展现出策略性行为，以及偏好优化能否在静态和零样本策略的基础上取得进一步突破。我们首先评估了一个静态基线模型和三个冻结语言模型：SmallThinker、Qwen3 和 Gemma-3，它们在精确博弈与六人对局评估中的表现差异显著。为此，我们提出了结合 KL 锚定均衡的策略偏好优化（Strategic Preference Optimization with KL-anchored Equilibrium，简称 SPOKE），该方法融合了教师相对价值惩罚、策略锚定、风险约束以及结构化推理监督，并将其与 SFT、DPO、GRPO 和 GSPO 进行了对比。在 SmallThinker-4B-A0.6B 策略的三次运行中，SPOKE 在 Kuhn（0.213）和 Leduc（1.081）扑克中均实现了最低的平均可利用度，并在六人局 Top-1 一致性上达到最高水平（0.775），同时保持了与 SFT 相当的教师 KL 散度（0.522 对比 0.523）。这些结果提供了初步证据，表明策略偏好优化可以在保持与教师模型行为一致的同时，显著提升策略质量。

Abstract:A poker agent can make a legal move and still make a strategically poor one. We ask whether compact AI agents can exhibit strategic behavior under hidden information and whether preference optimization can improve beyond static and zero-shot policies. We first evaluate a static baseline and three frozen language models: SmallThinker, Qwen3, and Gemma-3 whose performance varies substantially between exact-game and six-player evaluation. We therefore propose Strategic Preference Optimization with KL-anchored Equilibrium (SPOKE), combining teacher-relative value penalties, policy anchoring, risk constraints, and structured rationale supervision, and compare it with SFT, DPO, GRPO, and GSPO. Across three runs of a SmallThinker-4B-A0.6B policy, SPOKE achieves the lowest mean exploitability in Kuhn (0.213) and Leduc (1.081) and the highest six-max top-1 agreement (0.775), while maintaining teacher KL comparable to SFT (0.522 vs 0.523). These results provide preliminary evidence that strategic preference optimization can improve strategic quality while preserving teacher-consistent behavior.