这是一个厂商中立的起步框架，可用于部署一次性 Kubernetes 测试环境、注入故障、保存任意产品的调查记录，并通过可配置的裁判对已完成的调查进行评分。Python 3.10+ 是唯一的 Python 依赖。Kubernetes 操作还需要 kubectl；本地集群创建需要 Docker 和 kind。

可为集群选择本地 kind 或 AWS EKS，然后选择包含 21 个场景的完整套件或包含 6 个场景的冒烟测试环境。集群类型和场景套件是相互独立的选择。

mermaid flowchart LR A[Deploy cluster and application] --> B[Connect your product] B --> C[Inject and verify a fault] C --> D[Let your product investigate] D --> E[Save investigation records] E --> F[Score with your chosen judge] F --> G[Compare results]

加载中

请在 Project Arena 目录中运行命令。集群设置、产品集成、场景执行和评分是独立步骤；请按下文顺序操作。

我们比较了 Edge Delta 的原生 AI 调查能力、Grafana 的原生 AI 调查能力，以及 使用各平台可观测性 CLI 的 Claude，覆盖 21 个 Kubernetes 事故场景。edx 提供对 Edge Delta 的访问；gcx 提供对 Grafana 的访问。所有最终调查均基于相同的事故事实和评分量规，使用 GPT-6-Astra 进行评估。

检测与调查结果

Edge Delta 检测并调查了 18 个场景；Grafana 检测并调查了 12 个。对于两个平台上的全部 21 个场景，均从外部启动 Claude：使用 edx 时为 16 个告警和 5 个客户报告，使用 gcx 时为 12 个告警和 9 个客户报告。未对 Claude 的检测能力进行独立测量，因此相应单元格标为 —。

调查评分使用该列中的全部已完成调查。实施就绪度不计入未提出缓解建议的案例。

指标	Edge Delta 原生	Grafana 原生	Claude + edx	Claude + gcx
检测	18/21 (85.7%)	12/21 (57.1%)	—	—
根因分析	15/18 (83.3%)	9/12 (75.0%)	18/21 (85.7%)	19/21 (90.5%)
影响范围	12/18 (66.7%)	8/12 (66.7%)	18/21 (85.7%)	18/21 (85.7%)
有依据的最终缓解方案	8/18 (44.4%)	5/12 (41.7%)	16/21 (76.2%)	15/21 (71.4%)
实施就绪度	9/16 (56.2%)	5/12 (41.7%)	16/21 (76.2%)	15/21 (71.4%)

在相同 12 起事故上的比较

该表使用两个原生产品均调查过的 12 类事故，以及对应的 Claude 调查结果。它控制纳入哪些场景，而不控制启动提示词、时间安排或可用证据的差异。

指标	Edge Delta 原生	Grafana 原生	Claude + edx	Claude + gcx
根因分析	11/12 (91.7%)	9/12 (75.0%)	10/12 (83.3%)	10/12 (83.3%)
影响范围	9/12 (75.0%)	8/12 (66.7%)	10/12 (83.3%)	10/12 (83.3%)
有依据的最终缓解方案	7/12 (58.3%)	5/12 (41.7%)	8/12 (66.7%)	8/12 (66.7%)
实施就绪度	8/12 (66.7%)	5/12 (41.7%)	8/12 (66.7%)	8/12 (66.7%)

查看每个场景的结果 · 下载 CSV

指标	获得分数的条件
检测	产品在观测窗口内检测到事故并启动调查。
根因分析	最终报告正确解释了事故成因。
影响范围	最终报告正确识别受影响的工作负载及下游影响，且不会声称没有证据支持的故障。
有依据的最终缓解方案	最终建议针对该事故给出具体、有依据的修复方案或安全的遏制措施，且不包含仍然存在的错误或不安全建议。
实施就绪度	方案明确说明修正措施及必要细节；常规审查、实施和上线检查可以仍待完成。

缓解方案和就绪度衡量的是提案，而不是已执行的修复或已验证的恢复。

请参阅评分量规了解评分规则，并参阅完整结果和方法论了解判定明细和评估细节。

请在 Project Arena 目录中运行所有命令。完成集群和镜像设置后，两种环境使用相同的基准测试命令。运行器使用你提供的 context；不会自动安装网络组件或配置镜像仓库访问权限。

本地 kind	AWS EKS
集群创建	Docker 和 kind
凭证	无需 AWS 凭证
完整套件镜像	构建并加载到 kind
网络	NetworkPolicy 需要具备强制执行能力的 CNI
清理	删除 kind 集群

集群设置与部署方法是独立的选择：

路径	变更如何到达 Kubernetes	设置
直接方式	bench 使用 kubectl 应用应用和故障清单	按照下方场景命令操作
GitOps（完整套件）	你向自己的仓库提交并推送变更；Argo CD 负责同步	按照 Argo CD 设置和运行指南操作

GitOps 路径使用组件 Application、flagd-values 和 batch-active。应用和故障配置可位于不同仓库中，也可位于同一仓库的不同路径中。请使用自己的仓库并配置其 URL。记录调查产品可以访问哪些仓库。

下文的部署、故障和重置命令使用直接路径。对于由 Argo 管理的应用，请使用 GitOps 指南中的命令；为避免与 Argo 的自愈机制冲突，直接变更会被阻止。调查导入、评分和报告在两条路径中的工作方式相同。

安装 Python 3.10+、kubectl、Docker、kind 和 Helm。按照 kind 设置说明创建启用 Cilium 的集群并加载完整套件镜像。然后选择其 context：

export ARENA_CONTEXT=kind-incident-bench

完整套件使用面向 Linux AMD64 的预构建应用镜像，并要求使用 AMD64 worker。搭载 Apple Silicon 的 Mac 可在原生 ARM64 kind worker 上运行冒烟套件，或操作 AMD64 EKS 集群。仅构建 ARM64 故障镜像并不能使完整应用兼容 ARM64。

对于完整套件，请在 arena.json 中保留 registry: "fixture.local" 和 tag: "v1"。链接中的设置会为 netpol-isolation 启用 NetworkPolicy 强制执行。仅运行冒烟测试时，执行 python3 -m bench cluster create 即可；冒烟测试使用固定版本的公共镜像，不需要构建完整套件镜像。

除 Python、kubectl 和 Docker 外，还需安装 Terraform 和 AWS CLI。配置 AWS 凭证后，按照 AWS 集群设置说明创建集群和 kubeconfig context。将 Docker 认证到你的镜像仓库，并为 worker 架构构建镜像：

export ARENA_CONTEXT=YOUR_CONTEXT kubectl --context "$ARENA_CONTEXT" get nodes -L kubernetes.io/arch

Choose the platform matching the workers shown above.

export ARENA_PLATFORM=linux/amd64 # Required by the full suite’s prebuilt application. python3 -m bench.scenarios build-images --registry YOUR_REGISTRY/bench
--tag YOUR_TAG --platform "$ARENA_PLATFORM" --push

请选择 worker 架构，而不论运行构建的计算机采用何种架构：

Kubernetes worker	构建平台
ARM64 worker	支持冒烟套件；完整套件需要重新构建和验证应用
Intel Mac 本地 kind，或 Intel/AMD worker（包括默认的 EKS m6i.xlarge）	linux/amd64

搭载 Apple Silicon 的 Mac 可使用 linux/amd64 为 Intel/AMD EKS worker 构建镜像；Docker Desktop 通过模拟支持跨平台构建。该命令一次只构建一种目标架构。在混合架构集群中，应用会被调度到 AMD64 worker 上。

在 arena.json 中将 registry 和 tag 设为相同的值。节点必须拥有镜像仓库的拉取权限；请自行配置镜像仓库权限或 Kubernetes 拉取凭证。AWS 资源会产生费用。Terraform 当前会在一个区域内的三个可用区创建子网；可用区数量和 worker 数量是独立设置。

完成任一设置后，请按照产品自己的说明安装其采集器和告警配置。然后继续进行下方的运行配置。

编辑生成的 arena.json：

设置	填写内容
context	设置期间获得的 Kubernetes context；不回退使用环境 context
product	报告列中使用的产品名称
run_id、output_dir	唯一运行名称及其结果目录
suite、scenario	full 或 smoke，以及该套件中的一个场景
registry、tag	构建完整套件镜像时使用的值
judge	API 提供商、模型及可选端点/设置
judge_command	可选的自定义裁判可执行文件；会覆盖内置 API 适配器
truths	用于自定义场景的可选答案键覆盖配置

默认的 full 套件目前包含 21 个场景。较小的 smoke 套件包含 6 个场景，使用公共 Python 镜像，因此不需要构建故障镜像。两者使用不同的应用；请在部署前选择一个套件。请参阅场景表了解要求和差异。

python3 -m bench catalog python3 -m bench deploy

在注入故障之前，完成产品设置，并确认产品能够接收健康应用的遥测数据。

对于完整套件：

python3 -m bench fault python3 -m bench verify python3 -m bench evidence

fault 应用清单；verify 检查是否观察到预期故障。成功应用故障本身并不代表测试有效。请等待产品完成调查后再重置故障。

对于 smoke，请使用 fault 和 evidence，然后自行检查 Pod 状态、事件和服务可用性；自动化 verify 目前仅支持 full。健康的冒烟应用会在 8080 端口提供 HTTP 服务：

kubectl --context "$ARENA_CONTEXT" -n incident-bench port-forward service/api 8080:8080

In another terminal:

curl http://localhost:8080/health

若要再次尝试，请先重置，并使用 --case 选择一个新的案例目录；该参数应置于命令之前：

python3 -m bench --case oom-attempt2 fault python3 -m bench --case oom-attempt2 verify python3 -m bench --case oom-attempt2 evidence

后续导入和评分命令也应使用同一个 --case。它选择的是制品目录，而不是不同的场景。测试另一项故障时，请在运行文件中修改 scenario。

如何连接你的产品

按照产品自身说明安装产品采集器并配置告警。Project Arena 不会安装厂商 Agent、登录产品或自动配置告警。Kubernetes 日志、事件、Pod 状态和应用行为会提供故障信号；请通过所选工具采集指标和追踪数据。

将 arena.json 中的 product 设置为你希望在对比表中显示的名称。调查结束后，将以下文件保存到 <output_dir>/<scenario>/（或你所选的案例目录）下：

文件	内容
final.txt	实际交付的最终回答，保持不变
intermediate.txt	较早期的建议，如可用
actions.txt	工具提交记录及结果，如可用

使用以下方式导入：

该命令会自动使用这些文件名。缺少最终输入会导致错误；缺少中间证据或操作证据时，对应信息将保持缺失状态。在观测窗口和证据支持的情况下，请使用 --detection detected 或 --detection not_detected 明确记录检测情况；默认值为 not_measured。

支持手动导出。对于基于 API 的集成，请依据调查记录格式提供导出器。项目不捆绑托管产品连接器。未检测到的案例仍可拥有一份最终回答为空的记录；但不得将其表示为已完成的调查。

如何为调查评分

裁判是一个 AI 模型，用于将已完成的调查与场景答案键及评分量规进行比较。项目已包含答案键。

1. 在 arena.json 中选择裁判提供商和模型。例如，编辑现有的 judge 部分：

   "judge": { "provider": "openai", "model": "YOUR_MODEL" }

2. 在 shell 中提供该供应商的 API 密钥（本例为 OPENAI_API_KEY）。请勿将密钥写入 arena.json。其他提供商和本地模型同样受支持。

3. 导入调查后，运行：

   python3 -m bench packet # Prepare the investigation, answer key and scoring rules python3 -m bench judge # Send them to your chosen model

结果会以 judgment.json 的形式保存到案例目录中，其中包含分数、解释和支持性引用。对要比较的每个产品，请使用相同的裁判模型和设置。API 调用可能产生费用。

你可以在不重新运行事故的情况下，再次为同一份已保存的调查评分。请参阅[重新评分和已保存的 re

A vendor-neutral starter for deploying a disposable Kubernetes fixture, injecting faults, saving investigation records from any product, and scoring completed investigations with a configurable judge. Python 3.10+ is the only Python dependency. Kubernetes operations additionally need kubectl; local cluster creation needs Docker and kind.

Choose local kind or AWS EKS for your cluster, then select the 21-scenario full suite or six-scenario smoke fixture. Cluster type and scenario suite are separate choices.

flowchart LR
    A[Deploy cluster and application] --> B[Connect your product]
    B --> C[Inject and verify a fault]
    C --> D[Let your product investigate]
    D --> E[Save investigation records]
    E --> F[Score with your chosen judge]
    F --> G[Compare results]

Loading

Run commands from the Project Arena directory. Cluster setup, product integration, scenario execution, and scoring are separate steps; follow the sections below in order.

We compared Edge Delta’s native AI investigations , Grafana’s native AI investigations , and Claude using each platform’s observability CLI across 21 Kubernetes incident scenarios. edx provides access to Edge Delta; gcx provides access to Grafana. All final investigations were evaluated against the same incident facts and scoring rubric using GPT-6-Astra.

Detection and investigation results

Edge Delta detected and investigated 18 scenarios; Grafana detected and investigated 12. Claude was started externally for all 21 scenarios on each platform: 16 alerts and 5 customer reports with edx, and 12 alerts and 9 customer reports with gcx. Detection was not independently measured for Claude, so those cells are shown as —.

Investigation scores use every completed investigation for that column. Implementation readiness excludes cases with no mitigation proposal.

Metric	Edge Delta native	Grafana native	Claude + edx	Claude + gcx
Detection	18/21 (85.7%)	12/21 (57.1%)	—	—
Root cause analysis	15/18 (83.3%)	9/12 (75.0%)	18/21 (85.7%)	19/21 (90.5%)
Blast radius	12/18 (66.7%)	8/12 (66.7%)	18/21 (85.7%)	18/21 (85.7%)
Supported final mitigation	8/18 (44.4%)	5/12 (41.7%)	16/21 (76.2%)	15/21 (71.4%)
Implementation readiness	9/16 (56.2%)	5/12 (41.7%)	16/21 (76.2%)	15/21 (71.4%)

Comparison on the same 12 incidents

This table uses the 12 incident types investigated by both native products, with the corresponding Claude investigations. It controls which scenarios are included, not differences in launch prompts, timing or available evidence.

Metric	Edge Delta native	Grafana native	Claude + edx	Claude + gcx
Root cause analysis	11/12 (91.7%)	9/12 (75.0%)	10/12 (83.3%)	10/12 (83.3%)
Blast radius	9/12 (75.0%)	8/12 (66.7%)	10/12 (83.3%)	10/12 (83.3%)
Supported final mitigation	7/12 (58.3%)	5/12 (41.7%)	8/12 (66.7%)	8/12 (66.7%)
Implementation readiness	8/12 (66.7%)	5/12 (41.7%)	8/12 (66.7%)	8/12 (66.7%)

View results for every scenario · Download CSV

Metric	What earns credit
Detection	The product detected the incident and started an investigation within the observation window.
Root cause analysis	The final report correctly explains what caused the incident.
Blast radius	The final report correctly identifies the affected workloads and downstream impact, without claiming unsupported outages.
Supported final mitigation	The final recommendation gives a concrete, supported fix or safe containment for the incident, with no remaining incorrect or unsafe advice.
Implementation readiness	The proposal specifies the correction and essential details; normal review, implementation, and rollout checks may remain.

Mitigation and readiness measure proposals, not executed repairs or verified recovery.

See the scoring rubric for grading rules and full results and methodology for verdict breakdowns and evaluation details.

Run all commands from the Project Arena directory. Both environments use the same benchmark commands after cluster and image setup. The runner uses the context you provide; it does not automatically install networking or configure registry access.

| Local kind | AWS EKS
---|---|---
Cluster creation | Docker and kind | Terraform in infra/cluster
Credentials | No AWS credentials | Your AWS credentials
Full-suite images | Build and load into kind | Build and push to a registry accessible by the nodes
Networking | NetworkPolicy needs an enforcing CNI | Terraform enables VPC CNI policy enforcement
Cleanup | Delete the kind cluster | Remove workloads, then destroy Terraform resources

Cluster setup and deployment method are separate choices:

Path	How changes reach Kubernetes	Setup
Direct	bench applies application and fault manifests with kubectl	Follow the scenario commands below
GitOps (full suite)	You commit and push changes to your repositories; Argo CD syncs them	Follow the Argo CD setup and run guide

The GitOps path uses component Applications, flagd-values, and batch-active. Application and fault configuration can live in separate repositories or separate paths in one repository. Use your own repositories and configure their URLs. Record which repositories the investigating product can access.

The deployment, fault, and reset commands below use the direct path. For an Argo-managed application, use the GitOps guide's commands; direct mutations are blocked to avoid conflicting with Argo's self-healing. Investigation import, scoring, and reporting work the same way for both paths.

Install Python 3.10+, kubectl, Docker, kind and Helm. Follow the kind setup to create a cluster with Cilium and load the full-suite images. Then select its context:

export ARENA_CONTEXT=kind-incident-bench

The full suite uses prebuilt application images for Linux AMD64 and requires AMD64 workers. Apple Silicon Macs can run the smoke suite on native ARM64 kind workers, or operate an AMD64 EKS cluster. Building ARM64 fault images alone does not make the full application ARM64-compatible.

For the full suite, keep registry: "fixture.local" and tag: "v1" in arena.json. The linked setup enables NetworkPolicy enforcement for netpol-isolation. For smoke alone, python3 -m bench cluster create is sufficient; smoke uses a pinned public image and does not need the full-suite image build.

Install Terraform and the AWS CLI in addition to Python, kubectl and Docker. Configure your AWS credentials, then follow the AWS cluster setup to create the cluster and kubeconfig context. Authenticate Docker to your registry and build images for your worker architecture:

export ARENA_CONTEXT=YOUR_CONTEXT
kubectl --context "$ARENA_CONTEXT" get nodes -L kubernetes.io/arch
# Choose the platform matching the workers shown above.
export ARENA_PLATFORM=linux/amd64  # Required by the full suite’s prebuilt application.
python3 -m bench.scenarios build-images --registry YOUR_REGISTRY/bench \
  --tag YOUR_TAG --platform "$ARENA_PLATFORM" --push

Choose the worker architecture , regardless of which computer runs the build:

Kubernetes workers	Build platform
ARM64 workers	Smoke suite supported; full suite requires rebuilding and validating the application
Intel Mac local kind or Intel/AMD workers, including the default EKS m6i.xlarge	linux/amd64

An Apple Silicon Mac can build for Intel/AMD EKS workers using linux/amd64; Docker Desktop supports cross-platform builds through emulation. This command builds one target architecture at a time. On mixed-architecture clusters, the application is scheduled on AMD64 workers.

Set registry and tag in arena.json to those same values. Nodes must have pull access to the registry; configure registry permissions or Kubernetes pull credentials yourself. AWS resources incur charges. Terraform currently creates subnets in three availability zones within one region; zone count and worker count are separate settings.

After either setup, install your product's collector and alert configuration using its instructions. Continue with the run configuration below.

Edit the generated arena.json:

Setting	What to enter
context	Your Kubernetes context from setup; no ambient-context fallback
product	Product name used in report columns
run_id, output_dir	A unique run name and its result directory
suite, scenario	full or smoke, and a scenario from that suite
registry, tag	Values used when building full-suite images
judge	API provider, model and optional endpoint/settings
judge_command	Optional custom judge executable; overrides the built-in API adapter
truths	Optional answer-key overrides for customized scenarios

The default full suite currently includes 21 scenarios. The smaller smoke suite includes six and uses a public Python image, so it does not need the fault-image build. They use different applications; select a suite before deploying. See the scenario table for requirements and differences.

python3 -m bench catalog
python3 -m bench deploy

Before injecting a fault, complete product setup and confirm the product receives the healthy application's telemetry.

For the full suite:

python3 -m bench fault
python3 -m bench verify
python3 -m bench evidence

fault applies the manifests; verify checks whether the expected failure is observed. Successful application alone does not establish a valid test. Let the product finish investigating before resetting the fault.

For smoke, use fault and evidence, then inspect pod status, events and service availability yourself; automated verify currently supports only full. A healthy smoke app serves HTTP on port 8080:

kubectl --context "$ARENA_CONTEXT" -n incident-bench port-forward service/api 8080:8080
# In another terminal:
curl http://localhost:8080/health

For another attempt, reset first and select a fresh case directory with --case, placed before the command:

python3 -m bench --case oom-attempt2 fault
python3 -m bench --case oom-attempt2 verify
python3 -m bench --case oom-attempt2 evidence

Use that same --case for subsequent import and scoring commands. It selects an artifact directory, not a different scenario. Change scenario in the run file when testing another fault.

How to connect your product

Install your product's collector and configure alerts using its own instructions. Project Arena does not install a vendor agent, log in to a product, or configure alerts automatically. Kubernetes logs, events, pod state and application behavior provide the fault signals; collect metrics and traces through your chosen tooling.

Set product in arena.json to the name you want in comparison tables. After the investigation, save these files under <output_dir>/<scenario>/ (or your selected case directory):

File	Contents
final.txt	The actual delivered final answer, unchanged
intermediate.txt	Earlier advice, if available
actions.txt	Tool submissions and results, if available

Import them with:

The command uses those filenames automatically. Missing final input is an error; missing intermediate/action evidence stays missing. Record detection explicitly with --detection detected or --detection not_detected when supported by an observation window and evidence; the default is not_measured.

Manual export is supported. For API-based integration, provide an exporter using the investigation record format. Hosted-product connectors are not bundled. An undetected case can still have a record with an empty final answer; it must not be represented as a completed investigation.

How to score investigations

The judge is an AI model that compares a completed investigation with the scenario's answer key and the scoring rubric. Answer keys are included.

1. In arena.json, choose the judge provider and model. For example, edit the existing judge section:

   "judge": {
     "provider": "openai",
     "model": "YOUR_MODEL"
   }
   

2. Make the provider's API key available in your shell (OPENAI_API_KEY for this example). Keep the key out of arena.json. Other providers and local models are supported.

3. After importing the investigation, run:

   python3 -m bench packet  # Prepare the investigation, answer key and scoring rules
   python3 -m bench judge   # Send them to your chosen model
   

The result is judgment.json in the case directory: scores, explanations and supporting quotes. Use the same judge model and settings for every product you compare. API calls may incur charges.

You can score the same saved investigation again without rerunning the incident. See rescoring and saved results for details.

python3 -m bench report --summary > summary.csv
python3 -m bench report --details > details.csv

The summary puts metrics in rows and products in columns. Each cell shows successful/applicable (percentage). Illustrative values, not benchmark results:

Metric	Product A	Product B
Detection	8/10 (80.0%)	7/10 (70.0%)
Root cause analysis	6/8 (75.0%)	5/7 (71.4%)
Final mitigation	5/8 (62.5%)	4/7 (57.1%)

The actual report includes every scoring dimension. The detail table lists each scenario's verdict and whether it is included in the denominator. Undetected cases count against detection rate; not-applicable cases are excluded from the relevant scoring denominator. Insufficient evidence stays in the denominator for applicable scored cases. Missing measurements and unscored investigations remain visible in the details. A dash means no applicable scored cases.

Product names come from investigation records, matched to judgments by content hash. Records and judgments are discovered in the configured run directory. Include records for undetected cases too. Use matching scenario cohorts, rubric and judge settings when comparing products; see report options and counting rules for combining runs and interpreting each metric.

For GitOps, follow the Git reset and sync workflow so the repository and cluster return to the same baseline.

For the full suite using direct deployment:

python3 -m bench reset --confirm-disposable

This retires the injected resources, restores the three scenario flags, and clears only that fault’s persistent effects. Healthy services, application data, credentials and the cluster remain in place; reset verifies the baseline before another fault. For smoke, use python3 -m bench reset; it restores the app but retains the storage fault's PVC.

Remove a local kind cluster

python3 -m bench cluster delete --confirm-delete

Remove application resources and any provisioned volumes or load balancers, then follow the AWS teardown instructions. Resetting a fault does not stop AWS charges. Optional Terraform state storage persists separately.

• Use --run path/to/run.json before a command to select another configuration. Explicit flags override saved settings.
• Configured file paths are relative to the run file. Explicit CLI paths and judge executable arguments are relative to the current working directory.
• Use packet --rubric path/to/rubric.md for a custom rubric. Keep the same rubric and judge settings across compared products.
• Product exporters and judge formats describe custom integrations and evidence requirements.
• Scenario implementation guide covers manifests, image builds, verification and reset behavior.

To run the offline tests:

python3 -m unittest discover -s tests -v

Project Arena is licensed under the MIT License. Bundled third-party code retains its own license, including the shop application under Apache-2.0.