🔶 Show HN：Gemini 4 Argon 玩《星际争霸》
英文标题：🔶 Show HN: Gemini 4 Argon plays StarCraft
StarSkirmish Bench v0.2 测试 LLM 在一小时内为《星际争霸：母巢之战》编写 C++ 神族 Bot 的能力，并以其对人类 Bot 和演示 Bot 的胜率评分。GPT-6 Astra 以最高分领跑，GPT-6.1 Sol 位居第二且单次运行成本仅略高于前者的十分之一；Gemini 4 Argon、Claude Opus 5.5 和 Claude Fable 5.1 则以 37 至 39 分基本并列。即使是最强模型 GPT-6 Astra，对 Stardust、PurpleWave 和 BananaBrain 三个顶尖人类 Bot 的胜率也仅为 7%，显示人类编写的强 Bot 仍明显领先。测试发现，模型在一小时内越早、越频繁地与强对手练习，通常最终表现越好；该基准与主流通用评测的相关性则在下降。
550.6 KB
1.7 KB
1.6 KB
2.8 KB
5.9 KB
342.1 KB
中文
TL;DR
- StarSkirmish 是一项基准测试，要求 LLM 为《星际争霸：母巢之战》编写 Bot
- 每个 LLM 有一小时实际墙钟时间，用 C++ 编写一个神族 Bot；随后该 Bot 将与其他 LLM 的 Bot，以及成熟的人类编写 Bot 对战评估
- StarSkirmish 测试长时程推理和 Agent 式编程能力，因为 LLM 必须吸收练习赛中的经验，以改进其《星际争霸》Bot
StarSkirmish Bench 评估每个 LLM 在 1 小时实际墙钟时间内能够编写出多强的《星际争霸》Bot。v0.2 在 StarSkirmish Bench v0.1 的基础上更新，纳入了 OpenAI、Anthropic、Google 和 Meta 的旗舰模型；每个模型均在其所属实验室的第一方 CLI 运行框架中运行。
LLM 的分数反映其对一组具有竞争力的人类编写 Bot 和演示 Bot 的胜率。
StarSkirmish Bench v0.2
在各自第一方 CLI 运行框架中进行 1 小时运行。分数为每个模型的 5 次运行所产生 Bot，对一组具有竞争力的人类编写 Bot 的平均胜率（0–100）。
Gemini 4 Argon 以 high 运行，这是它可用的最高推理强度；其他所有模型均以 max 运行。
GPT-6 Astra 是 StarSkirmish Bench v0.2 中得分最高的 LLM，明显领先于位列第二的 GPT-6.1 Sol。Gemini 4 Argon、Claude Opus 5.5 和 Claude Fable 5.1 紧随其后，实际可视为并列，得分均在 37 至 39 之间。
StarSkirmish Bench 与成本
StarSkirmish Bench 分数与单次 1 小时运行的平均 API 成本之间的关系。成本采用对数刻度。
Muse Wonder Full 的定价基于 Muse Wonder 的 API 定价。
GPT-6.1 Sol 提供了独特的高性价比：它是唯一位于最具吸引力象限的 LLM，分数排名第二，而每次运行成本仅略高于 GPT-6 Astra 的十分之一。
最强的人类编写 Bot 仍然远远领先于所有模型。面对 Stardust、PurpleWave 和 BananaBrain 时，最强模型 GPT-6 Astra 的胜率仅为 7%。
对顶尖人类编写 Bot 的胜率
每个模型的 5 个 Bot 对阵顶尖人类编写 Bot Stardust、PurpleWave 和 BananaBrain 的平均胜率。
1 小时的推理限制是一个影响因素，但这些比赛是在三张双人地图上进行的。真实的人类编写 Bot 锦标赛条件要困难得多。
Google 和 Meta 的新旗舰模型替代了 v0.1 中的 Gemini 3.8 Flash 和 Muse Spark 1.3：Gemini 4 Argon 位列第 3，Muse Wonder Full 位列第 7。GPT-6 Astra 和 Claude Opus 5.5 已在 v0.1 中介绍。
每次运行的 Agent 轨迹都会记录它如何使用这一小时。模型得分最强的预测指标，是它在这一小时内与参赛阵容中 7 个最强 Bot 进行了多少练习。
胜率与对强 Bot 的练习情况
StarSkirmish Bench 分数与模型中位数运行在一小时内同参赛阵容 7 个最强 Bot 进行的练习赛数量之间的关系。虚线表示对 9 个模型进行的线性拟合。
GPT-6 Astra、GPT-6.1 Sol 和 Gemini 4 Argon 会在这一小时较早阶段就开始与这些 Bot 对战，当时它们自己的 Bot 仍会输给对手。Muse Wonder Full、Claude Haiku 5.5 和 GPT-6 Luna 很少或从不这样做。Claude Fable 5.1 是例外：它在几乎只与较弱 Bot 练习的情况下仍获得 37 分。
运行间方差
每个模型 5 次运行的胜率。每个点代表一次运行最终产出的 Bot；线段跨越最低和最高运行结果，横条标记均值（即该模型的 Bench 分数），阴影带则表示一个标准差（σ）。
同一模型的不同运行之间，结果也存在差异。Gemini 4 Argon 的波动远大于其他实验室的旗舰模型：其最佳和最差运行相差 18 个百分点，而 GPT-6 Astra 为 9 个百分点，Claude Opus 5.5 为 8 个百分点。
每个 LLM 均使用 C++、基于 BWAPI 4.4.0 编写一个神族 Bot，并在 OpenBW 上进行对局。比赛为神族对神族，在三张天梯地图 Heartbreak Ridge、Benzene 和 Destination 上进行。当一方所有建筑均被摧毁时，比赛结束。达到 60 分钟上限（86,400 帧）的比赛，按 BASIL 天梯规则判定：击杀数 + 摧毁数得分较高的一方获胜。
LLM 从一个只会采矿的骨架 Bot 开始，同时获得三个演示 Bot 的源代码，以及两个标准 Bot 库：战斗模拟器 FAP 和地图分析器 BWEM。1 小时结束时其 Bot 文件夹中的所有内容即为最终 Bot，因此它必须始终让 Bot 保持可编译状态。
LLM 拥有编译 Bot、进行练习赛，以及读取每场比赛摘要、对局记录和自身 Bot 日志的工具。练习对手按等级分组：D 级（三个演示 Bot）、C 级和 B 级（各有三个中游排名的人类编写 Bot）、A 级（BananaBrain、Locutus）以及 S 级（Stardust、PurpleWave）。
LLM Bot
对手
比赛
每组配对的比赛数
每个模型最终产出的 5 个 Bot，都会与参赛阵容中的每个 Bot 对战：该阵容包括 10 个具有竞争力的人类编写 Bot 和 3 个演示 Bot。每对 Bot 会在每张地图上进行 6 次比赛，并使用全新的随机种子。与 v0.1 不同，LLM Bot 之间不再互相比赛。
StarSkirmish Bench v0.1 让所有模型在同一运行框架中以“high”推理强度运行。v0.2 则让每个模型在其所属实验室的第一方 CLI 运行框架中，以该框架提供的最高推理设置运行。
运行框架
v0.1 在一个中立的第三方运行框架中运行所有模型；v0.2 则在各实验室的第一方 CLI 运行框架中运行其模型。两个版本都给予每个模型 1 小时。
CLI 版本：Claude Code 2.1.296、Codex 0.162.1、Antigravity CLI 1.3.5 和 Muse Code 1.4.7。
推理强度
v0.1 让所有模型以高推理强度运行；v0.2 则让它们以最高强度运行。
v0.1
Low Medium High XHigh Max
High
v0.2
Low Medium High XHigh Max
Max
Gemini 4 Argon 以 high 运行，这是其支持的最高强度。
每个 v0.2 运行框架都会在模型上下文达到约 25 万 token 时进行压缩。当对话接近这一阈值时，CLI 会将其浓缩，模型随后继续工作。例如，Codex——OpenAI 的 CLI——为 GPT 模型提供默认的 27.2 万 token 上下文窗口，并在窗口填充至 90% 时触发压缩。
CLI 的网页搜索和网页抓取工具均被关闭，Agent 也没有其他互联网访问权限。CLI 在计时开始前已完成安装，1 小时从 Agent 启动时开始计算。不存在提交工具：时间耗尽时，运行框架直接收集 Bot。
v0.2 用直接胜率取代了 v0.1 的 Elo 评分。一个 Bot 的分数是它在锦标赛比赛中的胜率；一个模型的 StarSkirmish Bench 分数，则是其 5 个 Bot 分数的平均值，范围从 0（输掉每一局）到 100（赢下每一局）。
由于每个 Bot 都与相同的固定阵容比赛，其比赛结果不依赖于场内其他 LLM Bot。它们全部并行运行，因此可以为新模型的 Bot 评分，而无需重新进行其他人的比赛。固定阵容也能使评判标准长期保持一致，因此不同模型代际的分数，比在同一批参赛者内部拟合的 Elo 评分更能直接比较。
StarSkirmish 不再与最广泛被报道的评测保持同步。它与 Artificial Analysis Intelligence Index 的相关性很低（ρ 0.26）；在该指数中，本文最高分模型 GPT-6 Astra 在这 9 个模型中仅并列第 4。它也已偏离曾在 v0.1 中与之高度一致的 Terminal-Bench 4.0。Claude Sonnet 5.5 和 Muse Wonder Full 在 Terminal-Bench 4.0 中均接近榜首，但在此处却接近末位。
基准相关性
每个模型的 StarSkirmish Bench 分数与其 Vals IOI 及 WeirdML v3 分数之间的关系，以及它们之间的 Spearman ρ。虚线表示最佳直线拟合。
WeirdML v3 对 9 个模型中的 7 个进行了评分：截至撰写时，Gemini 4 Argon 和 Muse Wonder Full 尚未被纳入。
StarSkirmish 仍与 Vals IOI——以 C++ 解答的奥林匹克编程题——以及 WeirdML——一项针对非常规机器学习编程任务的独立基准——保持一致。一种解读是，受测最广泛的评测已成为模型被专门调优的目标，而针对长时程、高难度编程任务、较少被定向优化的测试仍会共同变化。
English / 原文
TL;DR
- StarSkirmish is a benchmark where LLMs write bots for StarCraft: Brood War
- Each LLM gets one hour to write a Protoss bot in C++, which is then evaluated against other LLMs' bots and established human written bots
- StarSkirmish tests long horizon reasoning and agentic coding as LLMs must incorporate learnings from practice games to improve their StarCraft bot
The StarSkirmish Bench evaluates how strong of a StarCraft bot can each LLM write given 1 hour of wall clock time. v0.2 updates StarSkirmish Bench v0.1 with flagship models from OpenAI, Anthropic, Google and Meta, each running in its lab's 1st party CLI harness.
An LLM’s score reflects its win rate against a roster of competitive human written bots and demo bots.
StarSkirmish Bench v0.2
1 hour runs, in its 1st party CLI harness. Scores are average win rates (0–100) of each model's 5 runs against a roster of competitive human written bots.
Gemini 4 Argon runs at high, the highest reasoning effort available to it; every other model runs at max.
GPT-6 Astra is the top scoring StarSkirmish Bench v0.2 LLM, a clear step ahead of GPT-6.1 Sol in second. Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 are functionally tied behind them, all scoring between 37 and 39.
StarSkirmish Bench vs. Cost
StarSkirmish Bench score against the average API cost of one 1 hour run. Cost is on a log scale.
Muse Wonder Full's pricing is based on Muse Wonder's API pricing.
GPT-6.1 Sol offers uniquely good value: it is the only LLM in the most attractive quadrant, scoring second at just over a tenth of GPT-6 Astra's cost per run.
The strongest human written bots are still well ahead of every model. Against Stardust, PurpleWave and BananaBrain, the best model, GPT-6 Astra, wins only 7% of its games.
Win rates vs the top human written bots
Average win rate of each model's 5 bots against top human written bots Stardust, PurpleWave, and BananaBrain.
1 hour reasoning limits are a factor, but these matches are played on three 2 player maps. Real human written bot tournament conditions are significantly harder.
Google and Meta's new flagships replace v0.1's Gemini 3.8 Flash and Muse Spark 1.3: Gemini 4 Argon places 3rd, and Muse Wonder Full 7th. GPT-6 Astra and Claude Opus 5.5 were covered in v0.1.
Each run's agent trace records how it spent its hour. The strongest predictor of a model's score is how much it practiced against the 7 strongest bots on the roster.
Win Rates vs. Practice Against Strong Bots
StarSkirmish Bench score against the number of practice games a model's median run played against the 7 strongest roster bots during the hour. The dashed line is a linear fit across the 9 models.
GPT-6 Astra, GPT-6.1 Sol and Gemini 4 Argon start playing these bots early in the hour, while their own bots still lose to them. Muse Wonder Full, Claude Haiku 5.5 and GPT-6 Luna rarely or never do. Claude Fable 5.1 is the outlier: it scores 37 while practicing almost entirely against weaker bots.
Run-to-Run Variance
Win rates of each model's 5 runs. Each dot is one run's final bot; the line spans the lowest to the highest run, the bar marks their mean (the model's Bench score) and the band one standard deviation (σ).
Results also vary between runs of the same model. Gemini 4 Argon varies far more than the other labs' flagships: its best and worst runs are 18 points apart, against 9 for GPT-6 Astra and 8 for Claude Opus 5.5.
Each LLM writes a Protoss bot in C++ against BWAPI 4.4.0, played on OpenBW. Games are Protoss versus Protoss on three ladder maps: Heartbreak Ridge, Benzene and Destination. A game ends when one side's buildings are all destroyed. Games that reach the 60 minute cap (86,400 frames) are decided as on the BASIL ladder: the higher kills + razings score wins.
The LLM starts from a skeleton bot that only mines, with the source code of the three demo bots and two standard bot libraries: FAP, a combat simulator, and BWEM, a map analyzer. Whatever is in its bot folder when the hour ends is its final bot, so it has to keep the bot compiling.
The LLM has tools to compile its bot, play practice games, and read each game's summary, transcript and its bot's own log. Practice opponents are grouped into tiers: D (the three demo bots), C and B (three mid-ranked human written bots each), A (BananaBrain, Locutus) and S (Stardust, PurpleWave).
LLM bots
Opponents
Games
Games / pair
Each model's 5 final bots play each of the 13 roster bots, the 10 competitive human written bots and the 3 demo bots, 6 times on each map, on fresh seeds. Unlike v0.1, the LLM bots don't play each other.
StarSkirmish Bench v0.1 ran every model in the same harness at 'high' reasoning. v0.2 runs each model in its lab's 1st party CLI harness at the highest reasoning setting it offers.
Harness
v0.1 ran every model in one neutral 3rd party harness; v0.2 runs each lab's models in its 1st party CLI harness. Both versions give each model 1 hour.
CLI versions: Claude Code 2.1.296, Codex 0.162.1, Antigravity CLI 1.3.5 and Muse Code 1.4.7.
Reasoning Effort
v0.1 ran every model at high reasoning effort; v0.2 runs them at max.
v0.1
Low Medium High XHigh Max
High
v0.2
Low Medium High XHigh Max
Max
Gemini 4 Argon runs at high, the highest effort supported.
Every v0.2 harness compacts the model's context at about 250k tokens. When a conversation nears that point, the CLI condenses it and the model carries on. Codex, OpenAI's CLI, for example, gives the GPT models its default 272k token window and triggers compression once 90% filled.
The CLIs' web search and web fetch tools are turned off, and the agent has no other internet access. The CLI is installed before the clock starts, and the hour counts from when the agent starts. There is no submit tool: the harness collects the bot when the time runs out.
v0.2 replaces v0.1's Elo ratings with plain win rates. A bot's score is its win rate over its tournament games, and a model's StarSkirmish Bench score is the average of its 5 bots' scores, from 0 (every game lost) to 100 (every game won).
Because every bot plays the same fixed roster, its games don't depend on the other LLM bots in the field. They all run in parallel, and a new model's bots can be scored without replaying anyone else's. A fixed roster also keeps the bar the same over time, so scores from different model generations are more directly comparable than Elo ratings fitted within one field.
StarSkirmish no longer tracks the most widely reported evals. It barely moves with the Artificial Analysis Intelligence Index (ρ 0.26), where GPT-6 Astra, the top model here, only ties for 4th of these 9 models. It has also drifted away from Terminal-Bench 4.0, which it tracked closely in v0.1. Claude Sonnet 5.5 and Muse Wonder Full both rank near the top of Terminal-Bench 4.0 but near the bottom here.
Benchmark Correlation
Each model's StarSkirmish Bench score against its Vals IOI and WeirdML v3 scores, with the Spearman ρ between them. The dashed line is the best straight-line fit.
WeirdML v3 scores 7 of the 9 models: Gemini 4 Argon and Muse Wonder Full have not been included at time of writing.
StarSkirmish still tracks Vals IOI, olympiad programming problems solved in C++, and WeirdML, an independent benchmark of unusual machine learning coding tasks. One reading is that the most heavily benchmarked evals have become targets that models are tuned for, while less targeted tests of long horizon, difficult coding tasks still move together.