Rails 测试进入自动驾驶模式：构建一个编写开发者不愿写测试的智能体

在大多数大型 Rails 单体应用中，组织总是优先开发新功能，而不是为新功能编写测试。久而久之，越来越多的代码未经测试，团队不得不花费更多时间调试棘手的问题。

我们构建了一个自主智能体来填补这一空白。它可以读取 Rails 源文件，生成或改进 RSpec 测试，根据风格规则和覆盖率目标进行验证，并在 CI/CD 流水线中运行，无需任何人工干预。为了在这种规模的代码库上运行，它采用并行模式：多个实例同时处理不同的文件。

从智能体的视角看 RSpec

Ruby 是动态类型语言：没有编译步骤，因此错误会在运行时才暴露。对我们的智能体来说，这意味着验证测试语法的唯一方法就是执行它。RSpec 是标准的 Rails 测试框架，它让测试表达性强、可读性高，但其领域特定语言很容易用错。

当智能体读取一个 Ruby on Rails 代码库时，它会处理五种主要文件类型（models、serializers、controllers、mailers、helpers），每种类型的结构各不相同（因此测试方式也不同）。智能体需要为每种类型获取不同的指令。

一个好处是：源文件到测试文件的映射几乎是 1:1。一般约定如下：

不过，这条规则也有一些例外，比如 app/controllers/ 有时会映射到 spec/requests/，或者单个源文件可以有多个测试文件，这种情况下的约定是：

这种直观的映射使得定位任何文件的测试，或找出完全没有测试的文件变得很容易。

让智能体感到困难的地方在于，为了避免重复代码，RSpec 严重依赖共享上下文：工厂、固件、数据库模式……

• 工厂（Factories）： 可复用的模板，用于创建具有预定义属性的测试对象，方便生成一致的测试数据。

• 固件（Fixtures）： 静态数据文件，用于预加载测试数据库记录，为测试提供固定的基线。

如果工厂文件不存在，智能体会创建它；如果已存在，智能体会复用。因为工厂文件跨多个测试共享（与测试文件不同），草率的改动很容易破坏其他地方的测试，因此对这些文件的更新必须谨慎。

使用 Vibe 构建智能体

我们基于 Vibe 构建了此智能体，这是 Mistral 的开源编码助手。默认的系统提示对本项目已经足够，因此我们专注于三个调控点：仓库级别的上下文、专门化技能和自定义工具。

上下文工程

上下文工程是我们方法的核心。Vibe 支持仓库级别的 AGENTS.md 文件：当在仓库根目录包含该文件的情况下运行时，其内容会自动附加到系统提示中。

我们使用的 AGENTS.md 提供了目标仓库的基本信息，但最重要的是，它为智能体提供了一个逐步执行计划：

2. 阅读文档（如果存在）

3. 检查是否已有测试文件

4. 根据源文件的位置，只选择一个技能文件并读取

5. 查找现有的模式、工厂和辅助方法

6. 执行技能（提取 → 工厂 → 生成测试）

7. 使用 RuboCop 工具验证

8. 使用 SimpleCov 工具验证

每个步骤都包含了要做什么以及成功的标准。我们还在我们认为重要的领域加入了一些 RSpec 最佳实践，以便引导智能体。示例：

- **永远不要**使用 `be_present`、`be_truthy`、`be_between` 或 `include(:key)`，

这些都很模糊。始终使用 `eq(exact_value)`。

我们发现智能体有时会跳过某些方法或遗漏边缘用例：它会生成一个_看起来_完整的测试，但却悄悄忽略了源文件中的几个公开方法。为了解决这个问题，AGENTS.md 结尾强制进行自我审查：智能体必须重新读取源文件，并在结束前明确问自己：“我测试了每一个公开方法吗？数一下。”如果有遗漏，就返回重新生成。

凭借这个通用的 AGENTS.md 文件强制智能体遵循严格的计划，我们的质量评分从 0.68 提高到 0.74，而这仅仅来自一个包含框架层面指导的 Markdown 文件。

使用技能文件：

回顾 AGENTS.md 的第 4 步：

4. 根据源文件的位置，只选择一个技能文件并读取

单一通用技能会产生平庸的效果：对测试 model 文件足够精确的指令，对 controller 文件来说却是错误的指令。

有效的方法是，为每个类别创建一个独立的技能文件，外加一个用于普通 Ruby 文件的技能文件。

下面是一个用于测试 controllers 的基本技能文件示例：

name: "Generate Request Spec"

description: "为 Rails 控制器生成 RSpec 请求测试。当源文件位于 app/controllers/ 时使用。"

- `spec/requests//_spec.rb` — 从文件名中去掉 `_controller`

- `spec/factories/.rb` — 如果需要，创建或更新

## Controllers 测试示例

# frozen_string_literal: true

describe 'Admin::Users', type: :request do

  let(:user) { create(:user, :admin) }

  # 未授权访问 — 每个 action 一个测试

  describe '#authorized?' do

    let(:user) { create(:user) }

    it 'GET /admin/users 重定向' do

      expect(response).to have_http_status(:redirect)

  # 每个 action: 正常路径 + 异常路径

  describe 'POST /admin/users' do

    let(:valid_params)   { { user: attributes_for(:user) } }

    let(:invalid_params) { { user: { email: '' } } }

    context 'with valid params' do

        expect { post '/admin/users', params: valid_params }.to change(User, :count).by(1)

        expect(response).to have_http_status(:created)

    context 'with invalid params' do

      it '返回不可处理实体并包含错误' do

        post '/admin/users', params: invalid_params

        expect(response).to have_http_status(:unprocessable_entity)

        json = JSON.parse(response.body, symbolize_names: true)

        expect(json[:errors]).to include("Email can't be blank")

- **断言内容，而非仅检查状态码：** 务必解析 JSON 并验证确切的值

- **测试精确的错误消息：** 如 `include("Email can't be blank")`，而不是 `be_present`

- **验证状态变化：** 使用 `.by(1)` 并检查所创建记录的属性

- **覆盖所有 action：** index、show、create、update、destroy 以及任何自定义 action

- **对每个 action 测试鉴权：** 确保未授权用户获得正确的状态码

- 只使用 `let`，永远不要用 `@instance_variables`；在 `follow_redirect!` 之前再次 `sign_in user`

添加自定义工具

Vibe 内置了读写文件、运行 bash 命令和编辑代码的工具。但它也支持自定义工具和 MCP 来扩展其操作空间。对本项目而言，自定义工具是关键。

经过一些实验，我们选择了：

• 一个 RuboCop 代码检查工具，运行 rubocop 命令检测风格违规，以便智能体返回代码并修复它们。

• 一个 SimpleCov 工具，与 RSpec 集成，可以检查代码覆盖率和测试正确性。将此步骤放在最后很关键：它确保智能体编写的每一个测试真正运行起来。

下面是实现 RuboCop 代码检查工具的简化版本：

from pydantic import BaseModel

from vibe.core.tools.base import BaseTool, ToolError

class RubocopLintResult(BaseModel):

    exit_code: int | None = None

class RubocopLint(BaseTool):

    description = "使用 Rubocop 对测试文件进行代码检查。"

    async def run(self, spec_path: str):

        spec_path = Path(spec_path)

        cmd = ["rubocop", "--format", "simple", str(spec_path)]

        proc = await asyncio.create_subprocess_exec(

            *cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE

            stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=60)

            raise ToolError("Rubocop 在 60 秒后超时")

        output = stdout.decode("utf-8", errors="ignore")

            success=proc.returncode == 0,

            exit_code=proc.returncode

SimpleCov 工具遵循相同的模式，但做得更多：它使用 RSpec 运行测试文件，收集每个测试的通过/失败结果，并报告源文件的覆盖率百分比。智能体能够获取哪些测试失败了（以及错误消息），以及源文件的哪些行从未被执行。这赋予了它在必要时进行自我校正的能力。

当我们首次接入这个工具时，只有大约三分之一的生成测试在第一次执行时通过。智能体在几次迭代内便自行修正了所有失败。如果没有这一步，其中一些测试根本就不会运行起来。

---

衡量测试质量

一个好的单元测试遵循 Arrange-Act-Assert 模式：设置状态，执行一个行为，验证结果。它应该使用精确的断言，覆盖正常路径和错误路径，并产生能够清晰表明崩坏位置的失败信息。

基于工具的指标

现有的 Ruby on Rails 工具可以让我们方便地获取多项指标：

• RSpec: 最流行的 Ruby 测试框架。表达性语法用于编写可读、结构良好的测试。

• RuboCop: 一个静态 Ruby 代码分析和格式化工具，帮助强制执行风格指南并保持代码一致性。

• SimpleCov: 一个针对 Ruby 的代码覆盖率工具，衡量测试套件执行了多少代码。

从这些工具中，我们可以提取一些有用的量化指标：测试通过的比例、每个文件的 RuboCop 违规数量、单元测试套件覆盖的代码比例，以及执行速度。这些指标为衡量测试套件的整体性能提供了坚实的基线。

但这些数字本身无法确定一个测试文件是否真正优秀。比如，两个文件都有零 RuboCop 违规、100% 覆盖率且所有测试通过，该如何比较？量化指标可能看起来完全一样，但测试质量仍然可能存在质的差异。

用 LLM 充当评判员

LLM-as-a-judge 的想法是向 LLM 提问：“所有错误条件都测试到了吗？”或“这是一个好测试吗？”，然后根据明确的标准返回一个介于 0 到 1 之间的分数。

示例提示：

精确的值：`eq(100)`，而不是 `be_present` 或 `be_a(Float)`。

状态变化：`change { User.count }.by(1)`。

错误路径：`raise_error(ArgumentError, "message")`。

边界情况：在阈值处测试精确的值（例如，`page(0)`）。

公共接口：不要使用 `.send(:private_method)`。

模糊的断言：`be_truthy`、`be_an(Array)`。

“不会崩溃”测试：`not_to raise_error`。

过度模拟（测试的是模拟对象，而不是行为）。

对返回值没有断言。

- 1.0: 对返回值/状态有精确的断言 + 所有关键路径都已覆盖（正常、错误、边界）。

- 0.8: 所有已测路径有精确断言，但缺少 1–2 个边缘用例。

- 0.6: 所有关键路径都已覆盖，但有 1–2 个测试使用了模糊断言（例如 `be_truthy` 而不是 `eq(expected)`）或轻微过度模拟。

- 0.4: 部分测试有精确断言，其他则模糊、过度模拟或跳过了返回值检查。

- 0.2: 大多数测试缺乏有意义的断言：模糊匹配器、桩化的逻辑、没有验证实际输出。

- 0.0: 测试结构上已崩坏：测试私有方法、没有真正的断言、或验证的是模拟对象而不是行为。

这里的“评分规则”部分尤为重要，因为它强制使每次运行的评分更加一致。如果没有这一部分，智能体将缺乏对评分严格或宽松程度的明确定义。

这样的分数不如工具的输出或彻底的开发者评审那样有权威性。但它能很好地表明当前版本智能体的表现，特别是当在仓库的每个文件上大规模运行时。

LLM-as-a-judge 评估的局限性：

任何可以表示为文本的东西都可以用 LLM-as-a-judge 评分，这使得它非常通用。但评分输出并不是确定性的。即使设置 temperature=0，由于内核操作顺序的原因，仍会存在差异。实际上，针对同一评估的输出总是很接近，但这种不确定性的缺失对某些用例来说可能是个问题。

就我们的用例而言，因为我们并不是只给一个测试评分，而通常是一次几十到几百个，所以这是一个可以接受的局限性。聚合后的分数保持了一致。

缺失括号问题

LLM-as-a-judge 还有另一个核心问题：一个测试可能质量很高，但却无法运行，或者其背后的逻辑完全错误。

示例：

RSpec.describe User, "#full_name" do

  it "用空格组合名和姓" do

    user = User.new(first_name: "Ada", last_name: "Lovelace"

    expect(user.full_name).to eq("Ada Lovelace")

这个测试看起来很棒。它只测试一个行为，描述清晰，读起来像文档，遵循……（原文在此截断）

In most large Rails monoliths, organizations prioritize writing new features over writing tests for them. Over time, more and more code goes untested, forcing teams to spend more time debugging painful bugs.

We built an autonomous agent that closes that gap. It reads Rails source files, generates or improves RSpec tests, validates them against style rules and coverage targets, and runs inside a CI/CD pipeline with no human intervention. To operate on codebases at this scale, it runs in parallel: multiple instances working on different files simultaneously.

RSpec through an agent's eyes

Ruby is dynamically typed: there is no compilation step, so errors surface at runtime. For our agent, this means the only way to verify test syntax is to execute it. RSpec, the standard Rails testing framework, makes tests expressive and readable, but its domain-specific language is easy to get wrong.

When the agent reads a Ruby on Rails codebase, it reads five main file types (models, serializers, controllers, mailers, helpers), each structured differently (therefore tested in different ways). The agent needs distinct instructions for each type.

One benefit: the mapping from source file to spec file is nearly 1:1. The general convention is:

There are a few exceptions to that rule however, like that app/controllers/ are sometimes mapped to spec/requests/, or that sometimes a single source file can have multiple spec files, in which case the convention is:

This straightforward mapping makes it easy to locate the tests for any given file, or to identify files that lack tests entirely.

Where it gets harder for our agent is that to avoid duplicating code, RSpec relies heavily on shared context: factories, fixtures, database schemas...

• Factories: Reusable templates for creating test objects with predefined attributes, making it easy to generate consistent test data.

• Fixtures: Static data files that preload test database records, providing a fixed baseline for tests.

If a factory file doesn’t exist, the agent creates it; if it does, the agent reuses it. Because factories are shared across many tests (unlike spec files), careless changes can easily break tests elsewhere, so updates to these files must be made with caution.

Building the agent with Vibe

We built the agent on top of Vibe, Mistral's open-source coding assistant. The default system prompt was sufficient for this project, so we focused on three levers: repository-level context, specialized skills, and custom tools.

Context engineering

Context engineering was central to our approach. Vibe supports a repository-level AGENTS.md file: when running on a repository with this file at its root, its contents are automatically appended to the system prompt.

The AGENTS.md we used provided basic details about the target repositories, but mostly, it provided the agent with a step-by-step execution plan:

2. Read the documentation (if it exists)

3. Check if a spec already exists

4. Choose and read exactly one skill based on the source file location

5. Find existing patterns, factories, and helpers

6. Execute the skill (Extract → Factory → Generate tests)

7. Validate with Rubocop tool

8. Validate with SimpleCov tool

Each step included details about what to do and what the success criteria are. We also included some best practices of RSpec on areas where we felt it was important to orient the agent. Example:

- **NEVER** use `be_present`, `be_truthy`, `be_between`, or `include(:key)`

These are vague. Use `eq(exact_value)` always

We found the agent would sometimes skip methods or leave edge cases untested: it would generate a spec that looked complete but quietly ignored a few public methods from the source file. To counter this, the AGENTS.md ends with a forced self-review: the agent must re-read the source file and explicitly ask itself "Did I test every public method? Count them." before finishing. If anything is missing, it goes back.

With this generic AGENTS.md file forcing the agent to follow strict planning, our quality score went from 0.68 to 0.74 , all from a single markdown file with framework-level instructions.

Using SKILLS files:

Recall step 4 of our AGENTS.md:

4. Choose and read exactly one skill based on the source file location

A single generic skill would produce mediocre results: the instructions precise enough for testing a model file are the wrong instructions for a controller file.

What worked was creating a separate skills file for each category, plus one for plain Ruby files.

Here is an example of a basic skills file for testing controllers:

name: "Generate Request Spec"

description: "Generate RSpec request tests for a Rails controller. Use when the source file is in app/controllers/."

- `spec/requests//_spec.rb` — drop `_controller` from the filename

- `spec/factories/.rb` — create or update if needed

## Example tests for Controllers

# frozen_string_literal: true

describe 'Admin::Users', type: :request do

  let(:user) { create(:user, :admin) }

  # Unauthorized access — one test per action

  describe '#authorized?' do

    let(:user) { create(:user) }

    it 'GET /admin/users redirects' do

      expect(response).to have_http_status(:redirect)

  # Each action: happy + sad paths

  describe 'POST /admin/users' do

    let(:valid_params)   { { user: attributes_for(:user) } }

    let(:invalid_params) { { user: { email: '' } } }

    context 'with valid params' do

        expect { post '/admin/users', params: valid_params }.to change(User, :count).by(1)

        expect(response).to have_http_status(:created)

    context 'with invalid params' do

      it 'returns unprocessable entity with errors' do

        post '/admin/users', params: invalid_params

        expect(response).to have_http_status(:unprocessable_entity)

        json = JSON.parse(response.body, symbolize_names: true)

        expect(json[:errors]).to include("Email can't be blank")

- **Assert content, not just status:** always parse JSON and verify exact values

- **Test exact error messages:** `include("Email can't be blank")`, not `be_present`

- **Verify state changes:** use `.by(1)` and check the created record's attributes

- **Cover all actions:** index, show, create, update, destroy, and any custom actions

- **Test auth on every action:** verify unauthorized users get the correct status code

- Use `let` only, never `@instance_variables`; `sign_in user` again before `follow_redirect!`

Adding custom tools

Vibe has built-in tools for reading and writing files, running bash commands, and editing code. But it also supports custom tools & MCPs to expand its action space. For this project, custom tools were key.

After some experimentation, we went for:

• A RuboCop linting tool that runs the rubocop command to detect style violations, so that the agent goes back to the code and fixes them.

• A SimpleCov tool integrated with RSpec that checks both code coverage and test correctness. Placing this as the final step was key: it guarantees every test the agent writes actually runs.

Below is a stripped-down version of how we implemented the Rubocop linting tool:

from pydantic import BaseModel

from vibe.core.tools.base import BaseTool, ToolError

class RubocopLintResult(BaseModel):

    exit_code: int | None = None

class RubocopLint(BaseTool):

    description = "Lint spec files using Rubocop."

    async def run(self, spec_path: str):

        spec_path = Path(spec_path)

        cmd = ["rubocop", "--format", "simple", str(spec_path)]

        proc = await asyncio.create_subprocess_exec(

            *cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE

            stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=60)

            raise ToolError("Rubocop timed out after 60s")

        output = stdout.decode("utf-8", errors="ignore")

            success=proc.returncode == 0,

            exit_code=proc.returncode

The SimpleCov tool follows the same pattern but does more: it runs the spec file with RSpec, collects pass/fail results for each test, and reports the coverage percentage of the source file. The agent gets back which tests failed (with error messages), and which lines of the source file were never exercised. This gives it enough to self-correct when needed.

When we first wired this tool in, only around a third of generated tests passed on first execution. The agent self-corrected all failures within a few iterations. Without this step, some of those tests would not have ran.

---

Measuring test quality

A good unit test follows the Arrange-Act-Assert pattern: set up state, execute one behavior, verify the outcome. It should use precise assertions, cover both happy and error paths, and produce failures that clearly identify what broke.

Tool-based metrics

Existing Ruby on Rails tools allow us to get several metrics easily:

• RSpec: the most popular testing framework for Ruby. Expressive syntax for writing readable, well-structured tests.

• RuboCop: a static Ruby code analyzer and formatter, helping enforce style guidelines and keeping code consistent

• SimpleCov: a code coverage tool for Ruby, measuring how much code is executed by the test suite.

From these tools, we can extract several useful quantitative metrics: the proportion of tests that pass, the number of RuboCop violations per file, the proportion of code covered by the unit test suite, and the speed of execution. These metrics provide a solid baseline for measuring the overall performance of a test suite.

But these numbers alone cannot determine whether a test file is actually good. How should we compare two files that both have zero RuboCop violations, 100% coverage, and all tests passing? The quantitative metrics may look identical, yet qualitative differences in test quality can still exist.

LLM-as-a-judge

The idea behind LLM-as-a-judge is to ask an LLM: “Are all error conditions tested?” or “Is this a good test?” and get back a score from 0 to 1 based on clear criteria.

Example prompt:

Precise values: `eq(100)`, not `be_present` or `be_a(Float)`.

State changes: `change { User.count }.by(1)`.

Error paths: `raise_error(ArgumentError, "message")`.

Boundary cases: Test exact values at thresholds (e.g., `page(0)`).

Public interface: No `.send(:private_method)`.

Vague assertions: `be_truthy`, `be_an(Array)`.

"Doesn't crash" tests: `not_to raise_error`.

Over-mocking (tests mocks, not behavior).

No assertions on return values.

- 1.0: Precise assertions on return values/state + all key paths covered (happy, error, boundaries).

- 0.8: Precise assertions on all tested paths, but 1-2 edge cases missing entirely.

- 0.6: All key paths covered, but 1-2 tests use vague assertions (e.g., `be_truthy` instead of `eq(expected)`) or minor over-mocking.

- 0.4: Some tests have precise assertions; others are vague, over-mocked, or skip return value checks.

- 0.2: Most tests lack meaningful assertions: vague matchers, stubbed-out logic, no verification of actual output.

- 0.0: Tests are structurally broken: testing private methods, no real assertions, or verifying mocks instead of behavior.

The "Scoring Rules" section here is especially important as it enforces the score to be more consistent across runs. Without it, the agent would lack clear definition of how strict or how lax the scoring is.

Such a score is less authoritative than a tool's output or a thorough developer review. But it is a good indication of how well the current version of our agent is doing, especially when run at scale across every file in a repository.

Limitations of LLM-as-a-judge evaluation:

Anything that can be put into text can be scored with LLM-as-a-judge, which makes it versatile. But the score output is not deterministic. Even with temperature=0 , there will be variance due to the order of kernel operations. In practice the outputs for the same evaluation are always close, but this lack of determinism can be an issue for some use cases.

For our use case, since we score not a single test but usually tens to hundreds at a time, this was an acceptable limitation. The aggregated score stayed consistent.

The missing parenthesis problem

There was another core problem with LLM-as-a-judge: a test can be of great quality but not run, or have the logic behind it be wrong altogether.

Example:

RSpec.describe User, "#full_name" do

  it "combines first and last name with a space" do

    user = User.new(first_name: "Ada", last_name: "Lovelace"

    expect(user.full_name).to eq("Ada Lovelace")

This test looks great. It tests one behavior, has a clear description that reads as documentation, follows Arrange-Act-Assert:, and would fail with a useful message if broken.

We ran this test through multiple models using the scoring prompt defined earlier. The average score given was 0.75. A developer taking a quick look would likely agree.

But this test has one core flaw: it will not run. All because of a missing closing parenthesis ) on the User.new(). This is a critical flaw that makes this test definitely not suitable.

This example is easy to mitigate. But there are a lot other possibilities that can lead to syntax error:

• Versioning errors: Using methods or syntax from an incompatible Ruby or Rails version

• Non-existent methods: some methods might have been removed, renamed, or never existed.

• Factory/fixture errors: Referencing missing factories, incorrect attributes, or missing associations

• Missing dependencies: Using gems or modules not included in the test environment

The experiment

We ran an experiment to check the agent could hold up against a real codebase. The target repository contained 275 source files. Half of them already had test coverage. The other half had none.

We pointed the agent at every file, whether it already had tests or not. For uncovered files, it generated specs from scratch successfully. For files that already had tests, it rewrote and improved them.

We scored every spec file before and after with LLM-as-a-judge. The aggregate score for tested files went from 0.49 to 0.74, and coverage reached 100%. There's still room for improvement, but the results validated the approach.

Metric	Result
Files processed	275
Tests passing	100%
Average line coverage (SimpleCov)	100%
RuboCop violations after self-correction	0
LLM-as-a-judge score	0.74

By file type

Not all file types were easy to test. Models scored highest: their business logic is self-contained and the test patterns are predictable. Controllers were harder, HTTP request handling introduced more room for error.

File type	LLM-as-a-judge
Models	0.81
Controllers	0.67
Serializers	0.80

Does it run?

Bundling SimpleCov with RSpec execution as the final step, forcing the agent to run every test it wrote, was the single most impactful decision. Only a third of tests passed on first execution. The agent self-corrected all failures within a few iterations.

Without this step, LLM-as-a-judge would have rated these non-running tests as high quality: the missing parenthesis problem at scale.

---

Vibe is open source. The agent, tools, and skills described here all run on top of it. Try it now.