🔶 Launch HN：Vespper（YC F24）——SOTA Docx MCP

引言

Word 文档无处不在。在法律、金融和医疗等领域，Word 文档就是交付物。合同、监管申报材料和审计报告都会在 .docx 中起草、修订并签署，而企业通常也会定期使用大量 Word 模板。

这类工作正越来越多地转向 Agent。Microsoft Copilot 和 Word 中的 Claude 已让文档内 AI 成为主流；与此同时，越来越多的垂直领域 Agent——尤其是法律科技领域的 Agent——需要与 .docx 文件交互。

然而，AI Agent 目前仍难以在 Word 文档中完成高质量工作。

我们与数十名软件工程师交流过，他们大多来自法律科技和医疗健康领域。他们表示，为了可靠地编辑 Word 文档，自己需要花费数周乃至数月调优 Agent harness；如今，他们又不得不维护极其复杂的内部解决方案。

总体而言，如今 Agent 编辑 Word 文档主要有三类方式：

• 让 Agent 编写代码，使用 python-docx、aspose 或 Open XML SDK 等低层 SDK。
• 接入 SuperDoc、Office CLI、safe-docx 或 Adeu 等 MCP；这些 MCP 为 Agent 提供具有明确使用方式的工具。
• 通过有损投影对 Word 文档进行往返处理，即借助 pandoc/mammoth.js 等工具将文档转换为 Markdown/HTML，让 Agent 编辑后再转回 .docx。

当前方案可以处理简单场景，但在复杂场景中仍显不足。

在深入讨论这些方案及其缺点前，先来了解 .docx 文件究竟是什么。

问题

.docx 文件本质上是一个 ZIP 文件，其中包含遵循 OOXML（Office Open XML）规范、具有层级关系的 XML 文件。在该 ZIP 包中：document.xml 包含主体文本，styles.xml 定义可复用样式（有点像 CSS 样式表），numbering.xml 定义列表与编号行为；此外，页眉、页脚、脚注、关系、媒体和文档元数据也分别存储在独立 XML 文件中。

这些 XML 文件非常冗长。例如，在 document.xml 中，即使一段只有 4 至 5 句话的短文本，在加入样式、元数据、格式信息、文本 run 拆分和 XML 样板代码后，也可能变成数千个 token。用户在 Microsoft Word 中看到的文本可能被拆分到多个 XML 节点中，并以极其冗长的方式持久化存储。

这里有一个交互式组件，展示一个简单 .docx 文件在底层的工作方式：

DOCX 编辑的这一特性，使其不同于代码或 HTML 编辑，也更棘手。对于简单文件而言，文本表示基本就是文件本身，修改也通常是局部的。

以 Markdown/简单 HTML 为例，其结构至少是熟悉的，且样式是局部的。

对于垂直领域 AI Agent，这个问题会严重得多。

Harvey 等公司已经遇到了这一问题。当他们重建文档编辑系统时，得出的诊断结论是：他们一直在要求同一个 Agent 同时充当法律助手和 Word 状态机。

像 Harvey 所使用的 Agent，本身已经承担着困难的工作。它需要阅读交易对手的修订内容，应用律所的操作手册，检查第 3 条中的定义术语是否与第 27 条含义一致，并发现刚刚修改的赔偿上限是否与两页前的责任条款相矛盾。这些才是核心工作。拆分文本 run、追踪编号引用并不是，但它们却在竞争同一个上下文窗口。

现状

回到前文提到的解决方案：每一种方案都有不同权衡，但最终都会落到同一个问题上——Agent 将上下文预算耗费在 Word 的格式机制上，而非实际任务本身。

低层库（python-docx、aspose、Open XML SDK）

• 优点： 对 Agent 而言较直观——这些库存在于预训练数据中。它们提供完整表达能力，通常没有什么操作是被限制的。
• 缺点： 速度慢、成本高。处理一份很长的法律文档时，Agent 大部分时间都在编写和调试脚本；而超链接或修订跟踪这类简单功能，都要求 Agent 做出极其复杂的操作。

MCP 服务器（SuperDoc、Office CLI、safe-docx、Adeu）

• 优点： 相较于编写代码，速度更快、成本更低。
• 缺点： 每个 MCP 都是一种新的 DSL，Agent 必须从大量工具和选项构成的接口中即时学习。它们往往只能覆盖常见的 80% 场景——而真正的文档问题恰恰出现在剩余的 20% 中。

往返转换（DOCX ↔ Markdown/HTML）

• 优点： Agent 完全不必思考 Word；它只需编辑文本。
• 缺点： 一个方向的转换是有损的，另一个方向也无法逆转。尤其是 Markdown，无法表达文档从模板继承而来的样式关系。

在这些方案中，我们相信第三种，即往返转换。我们认为，让 Agent 从 Word 的复杂机制中解放出来后，它们能够更好地完成实际任务。

但往返转换有一个重大问题：如何实现无损转换？

解决方案

在解释协调器（reconciler）的工作方式之前，我们想先说明为何如此看好这种方案形态。灵感来自基础设施即代码（Infrastructure as Code，IaC）领域。

在 IaC 工具流行之前，开发者必须进入云账户中四处“点击”。他们过去要花数小时浏览 AWS/GCP 控制台、点击按钮并手动管理基础设施。如果你不幸需要维护多个必须保持同步的环境（预发布、生产、QA、客户环境等），或者需要新建一个环境，事情很快就会变成噩梦。

随后，Terraform 和 Pulumi 出现了。现在开发者只需修改代码，工具就会判断——或者更准确地说，协调（reconcile）——云端需要发生哪些变更。再也不需要花数小时操作控制台。

我们认为，这一理念可以很好地迁移到 Agent 和 Word 文档上。Agent 编辑一种对它来说可读、直观的表示形式，其他系统则负责确定这些修改对实际文件意味着什么。但正如我们所说，所需的工具尚不存在。pandoc 和 mammoth.js 等工具在转换过程中丢失的信息太多，也并不会真正与原始文件进行“协调”，因此这是一种高度有损的方法。

简而言之：我们喜欢这个思路，但现有工具还不够好。因此，我们着手寻找更好的方法。

我们首先要回答的问题是：使用什么表示形式？

Markdown 是一个显而易见的候选项，但我们很快放弃了它，原因是它无法表达样式，或将样式与元素建立关联。最终，我们选择了 HTML：

• HTML 在结构上接近 OOXML：<w:p> → <p>、<w:hyperlink> → <a>、<w:tbl> → <table>
• CSS 将样式与特定元素关联，这与 OOXML 样式的工作方式大致相同

但有损问题依然存在。转换器（pandoc、mammoth.js）能够把 DOCX 转为 HTML，但没有一个能同时产出最小化、整洁且高保真的 HTML，因此我们从零开始构建了自己的转换器。

理论上，Agent 现在可以直接编辑我们的 HTML，剩下的工作就是将其协调回原始 .docx。而这正是棘手之处。

OOXML 的接口范围极大：元素、选项和样式几乎无穷无尽。如果在其上再增加一种专有 HTML 表示形式，就会产生极长的待处理边缘案例。我们没有构建一个庞大的协调引擎、再用手写代码追逐每个边缘情况，而是决定训练一个模型，让它成为协调器。具体而言：向它展示大量 HTML ↔ OOXML 的变更，并训练它根据 HTML 变更预测相应的 OOXML 修改。

该模型处于整个解决方案的中心。它接收 Agent 的 HTML 意图，并为该区块输出有效的 OOXML。接着，我们将结果与原始区块进行 diff，确定性地计算修订跟踪，将修改补丁写入文件，再将其交还给调用方。

这里有一张交互式时序图，展示我们的解决方案如何端到端工作：

下一节将说明我们如何针对替代方案评估该解决方案。

评估

为对方案进行基准测试，我们在内部基准测试集的 279 个 DOCX 编辑任务上，将其与其他 5 种方案进行了比较。每种方案都在两个模型上运行：GPT 5.6 Sol 和 GPT 5.6 Terra，均设置为中等推理强度：

• MCP 服务器
  • Vespper MCP（我们的方案）
  • SuperDoc MCP - v0.18.1
  • Office CLI（通过 MCP）- v1.0.145
  • Adeu MCP - v3.0.2
• Skills/Harnesses
  • Anthropic 的 DOCX skill
  • 纯 python-docx harness

对于 harness，我们使用 LangChain 的 create_agent，它使 Agent 实例化和模型切换更容易，并且能够很好地配合官方 mcp package。我们跳过了内置完整功能的 harness——LangChain 的 deepagents 和 Vercel 的 eve——因为我们希望拥有最大的控制权，并尽量减少后台发生的“魔法”（例如自动压缩上下文、修改系统提示词等）。

关于上述候选方案，还有一些说明：

• 所有方案使用相同的系统提示词—— 不针对任何方案单独进行提示词调优，包括我们的方案。
• 我们从所有候选方案中移除了准备工作—— 几乎每个方案都会要求 Agent 在编辑前完成一些事务性工作：加载 skill、打开文件、跟踪会话，完成后保存和关闭。我们希望“中和”这些部分，因此在所有方案中都将其从 Agent 的工作中剥离：DOCX skill 会预先载入上下文，会话 ID 和文件路径在后台注入，SuperDoc 的 open/save/close 等生命周期工具则完全隐藏。每个候选方案在第一轮开始时就已经能够读取和编辑文件。
• python-docx 是性能下限。 我们希望了解在没有工具设计和调优支持时的性能表现。

下面来看数据集。

数据集

Agentic 任务

为了评估上述方案，我们构建了一个高质量的 Word 编辑任务数据集。该数据集大致经历了三个阶段。

获取原始文档。 从 docxcorp.us 获取原始 DOCX 文件，保留来自政府、医疗健康、金融和法律等不同主题的英文文件，并剔除低置信度分类结果。

合成任务。 对每份文档，我们都需要一组自然语言指令（“任务”）及其评分方法。我们最终构建了一个内部标注工具，用于加载原始文档、在浏览器中快速预览文档，并快速合成编辑任务。我们逐一审查每个任务，并决定批准、丢弃或修改。

关于数据集，还有一些说明：

• 我们必须确保任务清晰明确、直接易懂；我们避免使用“写一个有说服力的引言”这类任务，而是合成“编写一个包含以下内容的引言部分：……”这类任务。
• 真值文档不包含噪声（例如添加未要求的修改、使用错误样式等）。

为保持数据集整洁，我们手工标注了 300 个任务，并对 LLM-as-a-judge 和修复 Agent 进行了对齐，使其能够标记和修复有噪声的任务。

最终数据集：包含 2046 个任务，按文档级别以约 70/15/15 的比例划分（以避免数据泄漏），并根据文档主题、任务领域等元数据进行分层。每个任务包含以下 3 项内容：

• original.docx - 提供给 Agent 作为输入的、未经修改的源文档。
• modified.docx - 展示预期编辑结果的参考“黄金”版本。
• Prompt - 描述所请求编辑内容的自然语言提示词。

以下是数据集中的一个任务示例：

Add a new entry to Schedule 1 (Entities, and extent, to which this Act does not apply) for 'The Western Australian Planning Commission under the Planning and Development Act 2005.' Insert it in alphabetical order, after 'The State Administrative Tribunal established under the State Administrative Tribunal Act 2004.'

其中文含义为：

在附表 1（本法不适用的实体及不适用范围）中，新增一项“The Western Australian Planning Commission under the Planning and Development Act 2005.”。按字母顺序插入在“The State Administrative Tribunal established under the State Administrative Tribunal Act 2004.”之后。

以下是预期输出：

示例任务的预期输出。新增条目带有修订跟踪，缩进与周围列表一致，并且会像上方各行一样将该法案名称设为斜体。

如你所见，样式应当被保留。在这个例子中，Agent 应使用与其他条目相同的缩进，并将法案部分设为斜体。注意：我们不会告诉它新内容需要什么样式；我们期望 Agent 能够隐式理解这一点。

评分

评估时，我们会在上述任务上运行特定 Agent（例如 GPT 5.6 Sol + DOCX Skill）。每次 Agent 运行都会生成一个 output.docx，因此最终会得到：

• original.docx
• modified.docx - “真值”（即请求修改完成后文件应呈现的样子）
• output.docx - Agent 生成的文件

然后，我们执行以下步骤：

1. 将每份 Word 文档转换为一系列区块，大致每个 <w:p>/<w:tbl> 对应一个区块。
2. 对齐 original.docx ↔ modified.docx（真值），以找出实际发生变化的区块；随后对齐 modified.docx ↔ output.docx，以判断 Agent 是否生成了这些区块。对齐使用最长公共子序列算法计算。
3. 按内容和样式比较这些区块。内容是读者所能看到的内容（已应用修订跟踪）；样式是渲染后的计算外观（而非 Word 的格式 XML）。

这里有一幅交互式图示，展示某个具体任务中若干场景的评分流程。

协调器任务

在着手开发协调器前，我们首先定义了两项要求：

• 它必须准确（保留内容，并且擅长处理样式）
• 它必须极其快速

由于第二项要求，Agentic 方案并不可行。处理文档（例如法律文档）时，Agent 往往需要进行数十次、甚至数百次编辑。

Introduction

Word documents are everywhere. In domains such as legal, finance, and healthcare, the Word document is the deliverable. Contracts, regulatory submissions, and audit reports get drafted, redlined, and signed in .docx, and companies usually have libraries of Word templates they work with on a regular basis.

That work is increasingly shifting to agents. Microsoft Copilot and Claude in Word have made in-document AI mainstream, while a growing number of vertical agents; particularly in legal tech—need to interact with .docx files.

However, AI agents are still struggling to do great work in Word documents.

We spoke with dozens of software engineers, mostly in the legal tech and health care spaces, who said they spend weeks and even months tuning their harnesses to edit Word documents reliably, and now are forced to maintain very complex in-house solutions.

In general, there exist several ways for agents to edit Word docs today, which mainly fall into three categories:

• Letting agents write code that uses low-level SDKs such as python-docx / aspose / Open XML SDK.
• Connect MCPs such as SuperDoc, Office CLI, safe-docx or Adeu that provide opinionated tools for the agent.
• Round-trip the Word document through a lossy projection i.e. convert to Markdown/HTML using something like pandoc/mammoth.js, let the agent edit it and convert it back to .docx.

The current solutions work on simple cases, but they fall short when it comes to complex scenarios.

Before diving deep into the solutions and their drawbacks, let’s first understand what a .docx file is.

Problem

A .docx file is essentially a ZIP file with a hierarchy of XML files following the OOXML (Office Open XML) spec. Inside the ZIP, there are the following files: document.xml contains the main text, styles.xml defines reusable styles (kind of like a CSS stylesheet), numbering.xml defines list/numbering behavior, and separate XML files store headers, footers, footnotes, relationships, media, and document metadata.

These XML files are quite verbose. For example, in document.xml , even a short 4–5-sentence paragraph can turn into thousands of tokens once you add styles, metadata, formatting information, run splitting, and XML boilerplate. The text users see in Microsoft Word might be split across many XML nodes and might be persisted in a very verbose manner.

Here's an interactive widget that shows how a simple .docx file works behind the scenes:

This nature of DOCX editing makes it different and trickier from editing code or HTML. With simple files, the text representation is mostly the thing itself and changes are local.

If we take Markdown/simple HTML for example, the structure is at least familiar and styles are local.

This problem gets much worse for vertical AI agents.

Companies like Harvey have already run into it. When they rebuilt their document editing system, the diagnosis they landed on was that they'd been asking one agent to be both a legal assistant and a Word state machine at the same time.

An agent like Harvey's already has a hard job. It has to read the counterparty's redlines, apply the firm's playbook, check that a defined term means the same thing in clause 3 as it does in clause 27, and catch that the indemnification cap it just changed contradicts the liability section two pages up. That's the work. Splitting runs and chasing numbering references is not, but it competes for the same context window.

Status quo

Going back to the solutions above, each of them have different trade-offs, but they all land in the same place: the agent spends its context budget on Word mechanics instead of the actual task.

Low-level libraries (python-docx, aspose, Open XML SDK)

• Good: intuitive for the agent - these libraries are in the pre-training data. They provide full expressiveness; usually nothing is off-limits.
• Bad: slow and expensive. An agent working on a long legal document spends most of its time writing and debugging scripts, and simple things like a hyperlink or a tracked change require the agent to do backflips.

MCP servers (SuperDoc, Office CLI, safe-docx, Adeu)

• Good: fast and cheap compared to writing code.
• Bad: each one is a new DSL the agent has to learn on the fly, from a large surface of tools and options. And they tend to cover the common 80% - the remaining 20% is where real documents live.

Round-tripping (DOCX ↔ Markdown/HTML)

• Good: the agent doesn't have to think about Word at all. It just edits text.
• Bad: the conversion is lossy in one direction and can't be undone in the other. Markdown in particular can't express the style relationships a document inherits from its template.

Of all these approaches, we believe in the third one - round tripping. We believe that by liberating agents from thinking about Word, they can perform better on their tasks.

However, a big problem with round-tripping is - how do you create a lossless conversion?

Solution

Before explaining how the reconciler works, we want to say why we're so bullish on this shape of solution. The inspiration comes from the Infrastructure as Code world.

Before IaC tools became popular, developers had to go to their cloud accounts and "click around". They used to spend hours of navigating the AWS/GCP console, clicking buttons, managing infra by hand. And if you were unlucky enough to have several environments that all had to stay in sync (staging, production, QA, customer envs), or you needed to spin up a new one, that quickly became a nightmare.

Then Terraform and Pulumi showed up. Now a developer just changes code, and the tool figures out (or a better word - reconciles) what needs to change in the cloud. No need to spend hours in the console anymore.

We thought this idea translates nicely to agents and Word docs. The agent edits something readable that is intuitive to it, and something else figures out what that means for the actual file. But like we said, the tools are not there. Tools like pandoc and mammoth.js lose too much on the way, and they don't really "reconcile" anything to the original file, meaning it's a very lossy approach.

So in short: we liked the idea, but the tools were not good enough. We set out to find a better way.

First question for us was what representation are we going to use.

Markdown was an obvious first candidate, but we dropped it quickly. The reason is because it can't express styles or associate them with elements. Instead, we landed on HTML:

• HTML is structurally close to OOXML: <w:p> → <p>, <w:hyperlink> → <a>, <w:tbl> → <table>
• CSS associates styles with specific elements, which is roughly how OOXML styles work too

That still left the lossiness. Converters (pandoc, mammoth.js) will turn a DOCX into HTML, but none of them produce HTML that's minimal, clean and high-fidelity at the same time, so we built our own from scratch.

So in theory the agent can now just edit our HTML, and all that's left is reconciling it back into the original .docx. This is where things get tricky.

OOXML has a huge surface: endless elements, options and styles. If you add a proprietary HTML representation on top of it, you get a very long tail of cases to handle. Instead of building a giant reconciliation engine and chasing every edge case with hand-written code, we decided to train a model to be the reconciler. Namely: show it a lot of HTML ↔ OOXML changes and teach it to predict, given an HTML change, what the OOXML counterpart should be.

That model sits at the center of the solution. It takes the agent's HTML intent and emits valid OOXML for that block. We then diff the result against the original block, compute tracked changes deterministically, patch it into the file and hand it back to the caller.

Here's an interactive sequence diagram that shows how our solution works end-to-end:

The following section explains how we evaluated our solution against alternatives.

Evaluation

To benchmark our approach, we compared it against 5 other solutions on 279 DOCX editing tasks from the test set of our internal benchmark, each run on two models: GPT 5.6 Sol and GPT 5.6 Terra, both at medium reasoning:

• MCP servers
  • Vespper MCP (ours)
  • SuperDoc MCP - v0.18.1
  • Office CLI (via MCP) - v1.0.145
  • Adeu MCP - v3.0.2
• Skills/Harnesses
  • DOCX skill by Anthropic
  • Plain python-docx harness

For the harness we used LangChain's create_agent, which makes agent instantiation and model swapping easy and plays well with the official mcp package. We skipped batteries-included harnesses (LangChain's deepagents, Vercel's eve) because we wanted the most amount of control and the least magic happening behind the scenes (e.g automatic compaction, system prompt modifications, etc).

Few more notes about the candidates above:

• Same system prompt for everyone - No per-solution prompt tuning, including ours.
• We removed the setup work from all candidates - Almost every solution asks the agent to do some housekeeping before it can edit anything: load a skill, open a file, track a session, save and close when it's done. We wanted to "neutralize" these parts, so we took it off the agent's plate across the board: the DOCX skill is pre-loaded into context, session IDs and file paths are injected behind the scenes, lifecycle tools like SuperDoc's open/save/close are hidden entirely. Every candidate starts its first turn already able to read and edit the file.
• python-docx is the floor. We wanted to see what's the performance with no tool design and tuning behind it.

Let's now look at the datasets.

Datasets

Agentic tasks

To evaluate the solutions above, we’ve built a high-quality dataset of Word editing tasks. We built the dataset in roughly three stages.

Get raw documents. Pulled raw DOCX files from docxcorp.us, kept English ones from various topics (government, healthcare, finance, and legal), and dropped low-confidence classifications.

Synthesize tasks. For each doc we needed a set of natural-language instructions ("tasks") and a way to score them. We ended up building an internal annotation tool that helps us load raw documents, preview them quickly in the browser and allows us to synthesize editing tasks quickly. We reviewed each one and approved/discarded/changed things.

A few more notes about the dataset:

• We had to make sure the tasks are unambiguous and straightforward; We avoided tasks such as “Write a compelling introduction” and synthesized tasks such as “Write an introduction section with the following content: …”
• The ground truth document is not noisy (e.g adds unrequested changes, uses correct styling, etc).

To keep our dataset clean, we manually labeled 300 tasks and aligned an LLM-as-a-judge and a fixer agent to flag and fix noisy tasks.

Final dataset: 2046 tasks, split ~70/15/15 at the document level (to avoid data leakage) and stratified on metadata such as document topic, task domain, etc. Each task has 3 things:

• original.docx - the unmodified source document the agent is given as input.
• modified.docx - a reference "gold" version showing the expected edit.
• Prompt - A natural-language prompt describing the requested edit.

Here is an example task from our dataset:

Add a new entry to Schedule 1 (Entities, and extent, to which this Act does not apply) for 'The Western Australian Planning Commission under the Planning and Development Act 2005.' Insert it in alphabetical order, after 'The State Administrative Tribunal established under the State Administrative Tribunal Act 2004.'

And here is the expected output:

Expected output for the example task. The new entry is tracked, indented with the surrounding list, and italicizes the Act the same way as the rows above it.

As you can see, styling is expected to be preserved. In this case, the agent is expected to use the same indentation as the other points and also to italicize the act part. Notice : we don’t tell it what styles that new content needs. We expect agents to understand that implicitly.

Scoring

During evaluation, we run a specific agent (e.g GPT 5.6 Sol + DOCX Skill) on the tasks above. Each agent run produces an output.docx, so we’re ending up with:

• original.docx
• modified.docx - the "ground truth" (i.e. how the file should look like after the requested modifications)
• output.docx - the file that was generated by the agent

Then we do the following:

1. We convert each Word document to a sequence of blocks, roughly one per <w:p>/<w:tbl>.
2. We align original.docx ↔ modified.docx (ground truth) to find the blocks that actually changed, then align modified.docx ↔ output.docx to see if the agent produced them. The alignment is computed using the Longest Common Subsequence algorithm.
3. We compare those blocks by content and style. Content is what a reader would see (tracked changes applied). Style is the computed look after rendering (not Word's formatting XML).

Here is an interactive illustration that shows the scoring flow of a few scenarios on a specific task.

Reconciler tasks

Before we started working on the reconciler, we first defined two requirements:

• It must be accurate (preserve content, be good at styling)
• It must be extremely fast

Due to the second requirement, an agentic solution is not an option. Working on documents (e.g legal) often requires dozens if not hundreds of edits from an agent. If each edit takes 20 seconds to reconcile, that’d create a terrible experience for users.

So we framed it as a translation problem instead. The model gets a triplet - (html_old, html_new, xml_old) - and produces xml_new for that one block. No reasoning, no tool calls, no loop.

Finding xml_old isn't the model's job. We wrote a deterministic locator that maps an HTML block to its OOXML counterpart, tested separately. That keeps the search problem out of the model entirely, so it only has to do the one thing it's good at: write valid OOXML that matches an intended change.

For the model itself, we landed on the 3–8B range. Post-training runs on far smaller datasets than pretraining, so updating every parameter is wasteful. Rather than retraining billions of weights, we use Low-rank adaptation (LoRA) - a lightweight method that freezes the base model and trains a small set of "adapter" matrices alongside it. Early experiments settled us on rank 16, alpha 32, LR 2e-4, AdamW 8-bit.

The dataset is mined self-supervised from the agentic dataset: 4-tuples of (html_old, html_new, xml_old, xml_new) - plus augmentations shaped by what we saw agents are sending us in production.

The final reconciler dataset included ~16k of tasks, split 70/15/15 between train/validation/test.

Once the dataset was ready, the training was just SFT. We used Unsloth as the framework and Modal for GPUs:

Reconciler SFT training loss on Unsloth / Modal.

We also wanted, during training, to score the model on a metric we can understand (since loss is not very interpretable).

For that, we used a simplified version of the scoring method we mentioned above. Instead of calculating alignment between sets of blocks, we simply compare content and styles between just two blocks (the ground truth block and the output block) and compute a “passed” variable which is just a boolean.

We monitored that during training on a subset of the training set and also on our validation set and we made sure they get better over time and that there’s no overfitting:

Train and validation pass rate during reconciler training.

Results

The following shows the main 4 metrics across the 6 different solutions:

Pass rate, cost, latency, and tool calls across the six solutions, for GPT 5.6 Terra and GPT 5.6 Sol.

As you can see, Vespper MCP performed best overall. More specifically, Vespper is 2.7-2.9x cheaper and 2.7-3.5x faster than using the DOCX skill, while being more accurate.

Few other interesting insights:

• Generic tools such as Office CLI turned out to be really expensive and slow. GPT 5.6 Sol + Office CLI yielded ~$0.24/task and ~57s/task, which is ~4.5x more expensive and ~3.8x slower than its Vespper counterpart.
• GPT 5.6 Terra + Vespper MCP get slightly better performance than GPT 5.6 Sol + DOCX skill while being ~ 7x cheaper and ~ 3.7x faster.

We also wanted to see pass rate on different buckets of page ranges:

Pass rate by document length on GPT 5.6 Sol. Vespper stays stable on longer files.

We can see that on longer documents, the Vespper MCP agent achieves stable pass rate, unlike other solutions that experience a drop to ~74% pass rate and lower.

We also wanted to see whether there’s a clear area of tasks where Vespper wins. The following table shows pass rate across 7 domains. Note: there is overlap between them (one task can edit documents, lists and tables):

Pass rate by domain

Bold is the best score in that row. Groups overlap — one task can touch tables, lists, and paragraphs.

GPT 5.6 SolGPT 5.6 Terra

Pass rate by task group for GPT 5.6 SolTask group	Vespper MCP	DOCX Skill	Python-docx	Office CLI MCP	Adeu MCP	SuperDoc MCP
Tables (n=90)	77.8%	71.1%	72.2%	54.4%	27.8%	25.8%
Lists (n=96)	71.9%	70.8%	64.6%	51.0%	45.8%	34.4%
Paragraphs (n=159)	75.5%	72.3%	71.7%	60.4%	40.3%	39.6%
Headings (n=28)	60.7%	46.4%	60.7%	35.7%	21.4%	17.9%
Header/Footer (n=4)	50.0%	75.0%	50.0%	50.0%	50.0%	0.0%
Hyperlinks (n=8)	25.0%	25.0%	25.0%	12.5%	12.5%	12.5%
Images/Embedded Objects (n=4)	50.0%	50.0%	50.0%	25.0%	50.0%	25.0%
Tracked-change management (n=26)	57.7%	34.6%	42.3%	34.6%	34.6%	0.0%

Looking at the tables, the categories with the biggest gaps are tables, tracked changes and headings.

Limitations

There are still things that Vespper MCP doesn’t support yet:

• Creating/replying to comments
• Attach images/videos
• Support for latent styles i.e styles that are not defined in styles.xml

We are working hard to support them in upcoming versions.

Conclusions

We have shown that our approach of letting agents manipulate high-fidelity HTML abstraction, together with a good reconciler, delivers a big improvement in latency, cost and accuracy on our Word editing benchmark. These improvements unlock advanced Word editing use-cases such as fast live-editing experiences, long-horizon workloads and more. You can review Vespper pricing or follow the quickstart to connect an agent.

If you’re interested in trying Vespper DOCX MCP, you can sign up here. We’d love to work with agent builders and help their agents to edit Word documents. Our mission is to be the bridge between agents and Word.

Feel free to also send us any questions at founders@vespper.com 🙂