Show HN：Self-bench——用真实世界软件评测编程 Agent

为你的代码仓库找到最佳模型。
基于你自己的仓库已合并 Pull Request 构建的私有编程 Agent 基准测试。

公开基准测试只能告诉你某个模型在他人代码上的表现。SelfBench 则告诉你它在你的代码上的表现：它会把已合并的 PR 转换为带隐藏测试的任务，在这些任务上运行 Agent 和模型，并绘制准确率与成本的关系。

每一个发布到 selfbench.dev 的开源代码仓库都会获得一个实时排行榜：其中按照每项任务的准确率和成本，对各模型、harness 与推理设置进行排序；帕累托前沿则标示出在这两个维度上均未被其他设置超越的配置。

浏览排行榜： vercel/next.js · supabase/supabase · earendil-works/pi · getsentry/sentry · PostHog/posthog · pingdotgg/t3code · vercel/vercel · 全部代码仓库 →

对于每个已合并的 PR，SelfBench 都会从变更前的提交重建任务：PR 自身的请求将成为指令，创作 Agent 则编写隐藏测试和参考解答。只有在无解答时测试失败、使用原始实现时测试通过、重新运行时再次通过，并且经受独立审查后，任务才会被接受。每个被接受的任务都是原生的 Harbor 任务。

所有操作均在 app.selfbench.dev Web 应用中完成：

1. 使用 GitHub 登录，并连接代码仓库。
2. 批量生成（Batch Generation）：选择要构建的简单、中等和困难任务数量；或者在 Dataset 页面使用 Add PRs，为你选定的每个 Pull Request 构建一个任务。一个批次需要运行数小时；即使关闭页面，它也会继续运行。
3. 数据集（Dataset）：检查每项任务（指令、环境、隐藏测试、参考补丁、流水线产物），并批准或拒绝它。
4. 运行（Run）：选择模型、harness 和沙箱，并在已批准的任务上运行它们。
5. 结果（Results）：比较准确率与成本，并打开任意试运行的记录文本和评分。
6. 发布（Releases）：将公开代码仓库的结果发布至 selfbench.dev。

模型和沙箱会通过 Credentials 使用你所在组织自己的密钥运行。可通过 Public Results API 读取已发布结果，或使用 API key 自动化你的工作区。

SelfBench 的参考部署运行在 GCP 上：Cloud Run 提供 API 服务，GKE Autopilot 运行 Temporal 工作流 Worker 以及由 KEDA 弹性扩缩的 Harbor 任务，并使用 Cloud SQL 和 GCS。Cloud Run Worker 池也可作为替代方案。

1. 配置一个 GCP 项目、计费、Terraform 状态存储桶，以及 GitHub Actions Workload Identity Federation。
2. 配置 Terraform 输入，并将每个运行时密钥值分别存储到独立的 Secret Manager secret 中。
3. 使用 Terraform 应用环境，或者配置受保护的 GitHub dev 和 prod 环境，以通过 Actions 部署。
4. 将你的域名指向已配置的负载均衡器，并为应用 URL 配置 GitHub OAuth。

有关前置条件、精确的 Terraform 命令、运行时配置、GitHub Actions 设置以及 GKE Worker 设置，请参阅自托管与基础设施指南。

需要 Bun 1.3.14+ 及带有 Compose 的 Docker。

bun install --frozen-lockfile
bun run validate

从检出的代码目录运行完整堆栈（提供应用服务的 API、Worker、Temporal、Postgres、本地 Docker 沙箱）：

cp .env.example .env   # GitHub OAuth app, session secret, credential key, managed keys
SELFBENCH_PUBLIC_URL=https://your-tunnel.example docker compose --profile sandbox up -d --build
docker compose port api 8080

Compose 会以检出目录名称命名项目，并发布临时主机端口，因此多个 worktree 可以并行运行。在执行 up 前，将 SELFBENCH_PUBLIC_URL 设置为浏览器打开的源地址（隧道或反向代理），并在 OAuth 应用中注册 <origin>/auth/github/callback。

对于前端开发，bun run dev:site 会运行 API 和启用热重载的 Vite（密钥位于 .env.site；参见 scripts/dev-site.ts）。

MIT © 2026 Mupt AI。

Find the best models for your repo.
Private coding-agent benchmarks built from your repository's own merged pull requests.

Public benchmarks tell you how a model does on someone else's code. SelfBench tells you how it does on yours: it turns your merged PRs into tasks with hidden tests, runs agents and models on them, and plots accuracy against cost.

Every open-source repository released on selfbench.dev gets a live leaderboard: each model, harness, and reasoning setting placed by accuracy and cost per task, with the Pareto frontier drawn through the settings nothing else beats on both.

Browse the leaderboards: vercel/next.js · supabase/supabase · earendil-works/pi · getsentry/sentry · PostHog/posthog · pingdotgg/t3code · vercel/vercel · all repositories →

For each merged PR, SelfBench rebuilds the task from the commit before the change: the PR's own request becomes the instruction, and an authoring agent writes hidden tests and a reference solution. A task is accepted only if the tests fail without a solution, pass with the original implementation, pass again on a rerun, and survive an independent review. Every accepted task is a native Harbor task.

Everything happens in the web app at app.selfbench.dev:

1. Sign in with GitHub and connect a repository.
2. Batch Generation : choose how many easy, medium, and hard tasks to build, or use Add PRs on the Dataset page to build one task from each pull request you pick. A batch takes hours; it keeps running after you close the page.
3. Dataset : inspect each task (instruction, environment, hidden tests, reference patch, pipeline artifacts) and approve or reject it.
4. Run : pick models, harnesses, and a sandbox, and run them on the approved tasks.
5. Results : compare accuracy against cost, and open any trial's transcript and scores.
6. Releases : publish a public repository's results to selfbench.dev.

Models and sandboxes run on your organization's own keys under Credentials. Read released results through the Public Results API, or automate your workspace with an API key.

SelfBench's reference deployment runs on GCP: Cloud Run serves the API, while GKE Autopilot runs the Temporal workflow worker and KEDA-scaled Harbor jobs, with Cloud SQL and GCS. A Cloud Run worker pool remains available as an alternative.

1. Provision a GCP project, billing, Terraform state bucket, and GitHub Actions Workload Identity Federation.
2. Configure Terraform inputs and store each runtime secret value in its own Secret Manager secret.
3. Apply the environment with Terraform, or configure the protected GitHub dev and prod environments to deploy through Actions.
4. Point your domain at the provisioned load balancer and configure GitHub OAuth for the app URL.

For prerequisites, exact Terraform commands, runtime configuration, GitHub Actions setup, and GKE worker setup, see the self-hosting and infrastructure guide.

Requires Bun 1.3.14+ and Docker with Compose.

bun install --frozen-lockfile
bun run validate

Run the whole stack (API serving the app, worker, Temporal, Postgres, local Docker sandboxes) from a checkout:

cp .env.example .env   # GitHub OAuth app, session secret, credential key, managed keys
SELFBENCH_PUBLIC_URL=https://your-tunnel.example docker compose --profile sandbox up -d --build
docker compose port api 8080

Compose names the project after the checkout directory and publishes ephemeral host ports, so worktrees run side by side. Set SELFBENCH_PUBLIC_URL to the origin the browser opens (a tunnel or reverse proxy) before up, and register <origin>/auth/github/callback on the OAuth app.

For frontend work, bun run dev:site runs the API and Vite with hot reload (secrets in .env.site; see scripts/dev-site.ts).

MIT © 2026 Mupt AI.