商业产业 Hacker News AI 2026-09-29T23:40 正文 ok 1246 words

Inspect：面向大语言模型评测的开源框架

英文标题：🔶 Inspect: An open-source framework for large language model evaluations

Inspect 是由英国 AI Security Institute 与 Meridian Labs 开发的开源前沿 AI 评测框架，可用于代码、Agent 任务、推理、知识、行为和多模态理解等评估。该框架提供可组合的数据集、Solver、Scorer、工具和 Agent 组件，并内置超过 200 个可直接运行的评测实现。Inspect 支持 20 多家模型提供商，以及 HuggingFace、vLLM 和 SGLang 本地推理，并可运行 Claude Code、Codex CLI、Gemini CLI 等外部 Agent。对开发者而言，它还提供 Inspect View、VS Code 扩展、Docker 等沙箱环境及 Python/CLI API，以简化评测编写、运行、调试和结果分析。

打开原文相关 3 · 机会 2

7.2 KB

356.6 KB

307.9 KB

532.8 KB

中文

欢迎使用

Inspect 是由英国 AI Security Institute 和 Meridian Labs 开发的前沿 AI 评测框架。Inspect 可用于广泛的评测场景，衡量代码能力、Agent 任务、推理、知识、行为以及多模态理解。Inspect 的核心功能包括：

可组合的构建模块——数据集、Agent、工具和 Scorer——使评测易于编写和复用。
收录超过 200 个预构建评测，可直接在任意模型上运行。
丰富的工具支持，包括用于监控和可视化评测的 Web 版 Inspect View 工具，以及辅助编写和调试的 VS Code 扩展。
灵活的工具调用支持——自定义工具和 MCP 工具，以及内置的 bash、python、文本编辑、网页搜索、网页浏览和计算机工具。
支持 Agent 评测，包括灵活的内置 Agent、多 Agent 原语，以及运行 Claude Code、Codex CLI 和 Gemini CLI 等任意外部 Agent 的能力。
沙箱系统可通过扩展 API，在 Docker、Kubernetes、Modal、Proxmox、Vagrant 及其他系统中运行不受信任的模型代码。

下文将通过两个简短的“Hello, Inspect”示例进行介绍。了解基础知识后，可阅读关于数据集、Solver、Scorer、工具和 Agent 的文档，学习如何创建更高级的评测。

如果你主要希望运行评测，而非开发新的评测，可查看 Evals 列表，其中提供了超过 200 个热门基准测试的实现。

快速开始

开始使用 Inspect：

通过 PyPI 安装 Inspect：
如果你使用 VS Code，请安装 Inspect VS Code Extension（非必需，但强烈推荐）。

要开发和运行评测，你还需要访问模型；这通常要求安装相应的 Python 包，并确保环境中提供了适当的 API 密钥。例如：

pip install openai
export OPENAI_API_KEY=your-openai-api-key
inspect eval simpleqa.py --model openai/gpt-4o

pip install anthropic
export ANTHROPIC_API_KEY=your-anthropic-api-key
inspect eval simpleqa.py --model anthropic/claude-sonnet-4-6

pip install google-genai
export GOOGLE_API_KEY=your-google-api-key
inspect eval simpleqa.py --model google/gemini-3.5-flash-lite

pip install torch transformers
export HF_TOKEN=your-hf-token
inspect eval simpleqa.py --model hf/meta-llama/Llama-2-7b-chat-hf

Inspect 内置支持超过 20 家模型提供商，也支持通过 HuggingFace、vLLM 和 SGLang 进行本地推理。有关所有受支持提供商的详情，请参阅模型提供商文档。

如果你将编程 Agent 与 Inspect 配合使用，inspect-skills 插件会提供相应技能，使其能够监控正在运行的评测、高效读取日志并分析结果。

Hello, Inspect

Inspect 评测是一个 Task，它将以下三项内容结合起来：

提供带标注样本的数据集——通常是一张包含 input 和 target 列的表，其中 input 是提示词，target 是理想答案或评分指引。
为每个样本生成答案的 Solver。它可以简单到只对模型调用一次 generate()，也可以复杂到成为一个在多轮交互中使用工具的完整 Agent。
对输出进行评估的 Scorer——可使用文本比较、模型评分或其他自定义方案。

下面来看两个简短示例：一个问答基准测试和一个夺旗挑战。

基准测试：SimpleQA

该任务使用 SimpleQA 评估模型；SimpleQA 是一项由简短、事实检索型问题构成的基准测试（点击右侧数字可查看进一步说明）：

from inspect_ai import Task, task
from inspect_ai.dataset import FieldSpec, hf_dataset
from inspect_ai.scorer import model_graded_qa
from inspect_ai.solver import generate

1@task
def simpleqa():
    return Task(
2        dataset=hf_dataset(
            "codelion/SimpleQA-Verified",
            split="train",
3            sample_fields=FieldSpec(
                input="problem",
                target="answer",
            ),
        ),
4        solver=generate(),
5        scorer=model_graded_qa(),
    )

1 @task 装饰器会将函数注册到 Inspect，使 inspect eval 能够按名称发现并运行它。 2 hf_dataset() 会直接从 Hugging Face 加载样本。Inspect 还支持读取 CSV、JSON 和内存中的数据集。 3 FieldSpec 以声明式方式将数据集的 problem 和 answer 列映射至样本的 input 和 target，无需自定义转换函数。 4 generate() Solver 只需将每个 input 发送给模型，并收集其响应。 5 由于答案是自由格式文本，model_graded_qa() 会使用模型，根据 target 对每个响应评分。

在命令行中使用 inspect eval 运行，并通过 --model 选择模型：

inspect eval simpleqa.py --model openai/gpt-5

使用 inspect view 查看结果：

Agent：CTF 挑战

Agent 评测要求模型采取行动，而不仅仅是回答问题。下面是一个夺旗（CTF）任务，其中 react() Agent 使用 bash() 和 todo_write() 工具探索一个沙箱系统，以找到隐藏的 flag：

from inspect_ai import Task, task
from inspect_ai.agent import react
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import includes
from inspect_ai.tool import bash, todo_write

@task
def ctf():
    return Task(
        dataset=json_dataset("challenges.json"),
1        solver=react(
            prompt=(
                "You are a Capture the Flag player.
                Explore the system and find the flag."
            ),
            tools=[bash(), todo_write()],
            attempts=3,
        ),
2        scorer=includes(),
3        sandbox="docker",
    )

1 react() 是一个内置 Agent，会运行“推理—行动—观察”循环，在模型提交答案前持续向其提供指定的 tools（此处最多允许 3 次尝试）。 2 如果目标 flag 出现在 Agent 提交的答案中，includes() Scorer 即判定通过。 3 sandbox="docker" 提供了 bash() 使用的隔离 Docker 容器（由任务旁的 Dockerfile 或 compose.yaml 配置）。

使用 inspect view 查看结果，并更仔细地检查单个交互记录：

请参阅教程，探索展示更多 Inspect 功能和技术的深入示例。

Python API

上文演示了通过 CLI 中的 inspect eval 运行评测——你也可以使用 eval() 函数，直接在 Python 中完成相同操作。例如：

from inspect_ai import eval
from simpleqa import simpleqa

eval(simpleqa(), model="openai/gpt-5")

LLM 辅助

在学习和使用 Inspect 时，我们建议向 LLM 提供其完成辅助工作所需的文档。以下提供两个适合 LLM 使用的 Markdown 文档版本：

llms.txt：文档索引，按需获取文章（约 2,000 tokens）。
llms-guide.txt：全部文档的完整内容（约 185,000 tokens）。

此外，每个页面顶部还提供 Copy Page 按钮，可生成该页面的 Markdown 版本。

了解更多

要进一步了解如何使用 Inspect，请参阅以下文档章节：

教程包含多个带注释的示例，展示各种功能和能力。
组件是评测的构建模块：任务、数据集、Solver 和 Scorer。
模型介绍如何指定模型和提供商，以及缓存、多模态输入、推理、批处理模式和并发。
Agent 将规划、记忆和工具使用结合起来，以处理更长周期的任务；包括内置 ReAct Agent、多 Agent 架构以及与外部框架的桥接。
工具通过自定义和内置工具、MCP 集成、沙箱及工具调用审批扩展模型能力。
运行介绍如何运行更大的评测集合，包括错误处理、限制、并行和提前停止。
分析说明如何阅读评测日志、提取数据框，以及扫描交互记录以发现问题。
扩展展示如何通过新的模型 API、组件、沙箱、审批器、钩子和文件系统扩展 Inspect。

你还可以浏览可直接运行的基准实现 Evals 列表、社区软件包 Extensions 展示页，以及提供完整 Python 和 CLI API 的参考文档。

引用

BibTeX 引用：

@software{UK_AI_Security_Institute_Inspect_AI_Framework_2024,
  author = {AI Security Institute, UK},
  title = {Inspect {AI:} {Framework} for {Large} {Language} {Model}
    {Evaluations}},
  date = {2024-05},
  url = {https://github.com/UKGovernmentBEIS/inspect_ai},
  langid = {en}
}
__

如需署名，请按以下格式引用本作品：

AI Security Institute, UK. 2024.

Inspect AI: Framework for Large Language Model Evaluations

. Released May.

https://github.com/UKGovernmentBEIS/inspect_ai

English / 原文

Welcome

Inspect is a framework for frontier AI evaluations developed by the UK AI Security Institute and Meridian Labs. Inspect can be used for a broad range of evaluations that measure coding, agentic tasks, reasoning, knowledge, behavior, and multi-modal understanding. Core features of Inspect include:

Composable building blocks—datasets, agents, tools, and scorers—that make evaluations easy to write and reuse.
A collection of over 200 pre-built evaluations ready to run on any model.
Extensive tooling, including a web-based Inspect View tool for monitoring and visualizing evaluations and a VS Code Extension that assists with authoring and debugging.
Flexible support for tool calling—custom and MCP tools, as well as built-in bash, python, text editing, web search, web browsing, and computer tools.
Support for agent evaluations, including flexible built-in agents, multi-agent primitives, and the ability to run arbitrary external agents like Claude Code, Codex CLI, and Gemini CLI.
A sandboxing system that supports running untrusted model code in Docker, Kubernetes, Modal, Proxmox, Vagrant, and other systems via an extension API.

We’ll walk through two short “Hello, Inspect” examples below. Read on to learn the basics, then read the documentation on Datasets, Solvers, Scorers, Tools, and Agents to learn how to create more advanced evaluations.

If you are primarily interested in running evaluations rather than developing new ones, see the Evals listing where you’ll find implementations for over 200 popular benchmarks.

Getting Started

To get started using Inspect:

Install Inspect from PyPI with:
If you are using VS Code, install the Inspect VS Code Extension (not required but highly recommended).

To develop and run evaluations, you’ll also need access to a model, which typically requires installation of a Python package as well as ensuring that the appropriate API key is available in the environment. For example:

pip install openai
export OPENAI_API_KEY=your-openai-api-key
inspect eval simpleqa.py --model openai/gpt-4o

pip install anthropic
export ANTHROPIC_API_KEY=your-anthropic-api-key
inspect eval simpleqa.py --model anthropic/claude-sonnet-4-6

pip install google-genai
export GOOGLE_API_KEY=your-google-api-key
inspect eval simpleqa.py --model google/gemini-3.5-flash-lite

pip install torch transformers
export HF_TOKEN=your-hf-token
inspect eval simpleqa.py --model hf/meta-llama/Llama-2-7b-chat-hf

Inspect has built-in support for over 20 model providers as well as support for local inference with HuggingFace, vLLM, and SGLang. See the documentation on Model Providers for details on all supported providers.

If you use a coding agent alongside Inspect, the inspect-skills plugin provides skills that teach it to monitor running evals, read logs efficiently, and analyze results.

Hello, Inspect

An Inspect evaluation is a Task that brings together three things:

Dataset that provides labelled samples—typically a table with input and target columns, where input is the prompt and target is the ideal answer or grading guidance.
Solver that produces an answer for each sample. This can be as simple as a single generate() call to the model, or as sophisticated as a full agent that uses tools over many turns.
Scorer that evaluates the output—using text comparisons, model grading, or other custom schemes.

Let’s look at two short examples: a question-answering benchmark and a capture the flag challenge.

Benchmark: SimpleQA

This task evaluates a model on SimpleQA, a benchmark of short, fact-seeking questions(click on the numbers at right for further explanation):

from inspect_ai import Task, task
from inspect_ai.dataset import FieldSpec, hf_dataset
from inspect_ai.scorer import model_graded_qa
from inspect_ai.solver import generate

1@task
def simpleqa():
    return Task(
2        dataset=hf_dataset(
            "codelion/SimpleQA-Verified",
            split="train",
3            sample_fields=FieldSpec(
                input="problem",
                target="answer",
            ),
        ),
4        solver=generate(),
5        scorer=model_graded_qa(),
    )

1 The @task decorator registers the function with Inspect so that inspect eval can discover and run it by name. 2 hf_dataset() loads samples directly from Hugging Face. Inspect also reads CSV, JSON, and in-memory datasets. 3 FieldSpec declaratively maps the dataset’s problem and answer columns onto the sample’s input and target—no custom conversion function required. 4 The generate() solver simply sends each input to the model and collects its response. 5 Because the answers are free-form text, model_graded_qa() uses a model to grade each response against the target.

Run it from the command line with inspect eval, choosing a model with --model:

inspect eval simpleqa.py --model openai/gpt-5

Use inspect view to view the results:

Agent: CTF Challenge

Agent evaluations require the model take actions rather than just answer a question. Here’s a Capture the Flag (CTF) task where the react() agent explores a sandboxed system using bash() and todo_write() tools to find a hidden flag:

from inspect_ai import Task, task
from inspect_ai.agent import react
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import includes
from inspect_ai.tool import bash, todo_write

@task
def ctf():
    return Task(
        dataset=json_dataset("challenges.json"),
1        solver=react(
            prompt=(
                "You are a Capture the Flag player.
                Explore the system and find the flag."
            ),
            tools=[bash(), todo_write()],
            attempts=3,
        ),
2        scorer=includes(),
3        sandbox="docker",
    )

1 react() is a built-in agent that runs a reason-act-observe loop, giving the model the supplied tools until it submits an answer (here allowing up to 3 attempts). 2 The includes() scorer passes if the target flag appears in the agent’s submitted answer. 3 sandbox="docker" provides the isolated Docker container used by bash() (configured by a Dockerfile or compose.yaml alongside the task).

Use inspect view to view the results and look more carefully at individual transcripts:

See the Tutorial to explore more in-depth examples that demonstrate additional Inspect features and techniques.

Python API

Above we demonstrated using inspect eval from CLI to run evaluations—you can perform all of the same operations from directly within Python using the eval() function. For example:

from inspect_ai import eval
from simpleqa import simpleqa

eval(simpleqa(), model="openai/gpt-5")

LLM Assistance

As you learn and use Inspect we recommend you provide an LLM with the documentation required for it to assist. There are two versions of LLM friendly markdown documentation available:

llms.txt: Documentation index, articles fetched as required (~2k tokens).
llms-guide.txt: Full contents of all documentation (~185k tokens).

There is also a Copy Page button at the top of every page that provides a markdown version of the page.

Learning More

To learn more about using Inspect see the following documentation sections:

Tutorial includes several annotated examples demonstrating various features an capabilities.
Components are the building blocks of an evaluation: tasks, datasets, solvers, and scorers.
Models covers specifying models and providers, along with caching, multimodal input, reasoning, batch mode, and concurrency.
Agents combine planning, memory, and tool use for longer-horizon tasks, including the built-in ReAct agent, multi-agent architectures, and bridges to external frameworks.
Tools extend models with custom and built-in tools, MCP integrations, sandboxing, and tool-call approval.
Running covers running larger eval sets, with error handling, limits, parallelism, and early stopping.
Analysis explains how to read eval logs, extract data frames, and scan transcripts for issues.
Extensions shows how to extend Inspect with new model APIs, components, sandboxes, approvers, hooks, and filesystems.

You may also want to explore the Evals listing of ready-to-run benchmark implementations, the Extensions gallery of community packages, and the Reference for the complete Python and CLI API.

Citation

BibTeX citation:

@software{UK_AI_Security_Institute_Inspect_AI_Framework_2024,
  author = {AI Security Institute, UK},
  title = {Inspect {AI:} {Framework} for {Large} {Language} {Model}
    {Evaluations}},
  date = {2024-05},
  url = {https://github.com/UKGovernmentBEIS/inspect_ai},
  langid = {en}
}
__

For attribution, please cite this work as:

AI Security Institute, UK. 2024.

Inspect AI: Framework for Large Language Model Evaluations

. Released May.

https://github.com/UKGovernmentBEIS/inspect_ai