摘要：计算机使用 Agent 需要在复杂桌面场景中可靠地定位行动目标；在这类场景中，多个应用程序、重叠窗口以及视觉上相似的控件会争夺注意力。现有训练数据很少能将此类场景与密集标注配对，或以可控方式对其进行变化。我们提出 DeskForge：一个可控的桌面环境，通过组合和探索真实应用程序，为计算机使用 Agent 生成大规模监督数据。它能够改变应用状态、内容、窗口布局、外观和分辨率，并将截图、无障碍树和窗口几何信息融合为密集的元素标注，同时记录每次执行操作的结果。利用这一环境，我们构建了 DeskForge-1M 数据集，其中包含 120 万条带标注的桌面观测数据，以及 1.597 亿个元素实例。我们从 DeskForge-1M 中抽取 20 万个 grounding 样本，对四个视觉语言模型进行微调。四个模型均在留出的桌面条件以及全部五项外部 GUI grounding 基准上取得提升；对于 Qwen3.5-4B，其在 ScreenSpot-Pro 上的准确率提高了 11.51 个百分点，在 OSWorld-G 上提高了 10.11 个百分点。这些增益也迁移至长程任务完成能力：在固定规划器条件下，微调后的行动模型能够完成更多 WebArena-Infinity 和 OpenApps 任务；其中，Qwen3.5-4B 在 WebArena-Infinity 的 119 项任务中完成数量从 31 项增至 50 项，在 OpenApps 的 100 项任务中从 3 项增至 15 项。这些结果表明，对真实桌面环境进行可控组合，能够为提升 GUI grounding 与长程计算机使用能力提供可扩展的监督来源。框架代码、数据集和微调模型可在项目页面获取：此 HTTPS 链接

Abstract:Computer-use agents need to reliably ground action targets in complex desktop scenes, where multiple applications, overlapping windows, and visually similar controls compete for attention. Existing training data rarely pair such scenes with dense annotations or vary them in a controlled way. We introduce DeskForge, a controllable desktop environment that composes and explores real applications to generate large-scale supervision for computer-use agents. It varies application states, content, window layout, appearance, and resolution, and fuses screenshots, accessibility trees, and window geometry into dense element annotations while recording the outcome of each executed action. Using this environment, we construct DeskForge-1M, a corpus of 1.2M annotated desktop observations containing 159.7M element instances. We fine-tune four vision-language models on 200K grounding examples drawn from DeskForge-1M. All four improve across held-out desktop conditions and on all five external GUI grounding benchmarks; for Qwen3.5-4B, accuracy increases by 11.51 percentage points on ScreenSpot-Pro and 10.11 points on OSWorld-G. The gains also translate to long-horizon task completion: under a fixed planner, the fine-tuned action models solve more WebArena-Infinity and OpenApps tasks, with Qwen3.5-4B increasing from 31 to 50 of 119 tasks and from 3 to 15 of 100 tasks, respectively. These results show that controllable composition of real desktop environments provides a scalable source of supervision for improving both GUI grounding and long-horizon computer use. The framework code, the dataset, and the fine-tuned model are available from the project page: this https URL