Show HN：Jevcrawl——无需在爬取循环中使用 LLM 的 AI Agent 网页爬虫

浏览网站的 Agent 面临两种糟糕选择：它们可以阅读爬虫抓取的一切内容，但答案会淹没在导航和样板内容中，并消耗大量 token；或者可以让 LLM 选择每一个链接，但这会很慢，而且每个问题要花费数美元。jevcrawl 两者都不采用：

• 🎯 更频繁找到答案页面。 任务中的答案页命中率为 54%，而 Claude 选择链接时为 33%；命中率也是 Crawl4AI 目标导向爬取的 2 倍。
• 💸 每项任务约便宜 8 倍，每个正确答案约便宜 10 倍。 爬取决策成本约为一美分。
• ⚡ 比 LLM 驱动的爬取快 2 倍。 每次并行处理四个页面，每页只发起一次决策请求。
• 📜 逐字返回，并附带来源。 每段内容均从页面原文复制，并带有 URL、标题路径和字符偏移。决策模型仅负责评分，因此无法编造文本。
• 🛡️ 从设计上抵御提示注入。 页面文本只会被评分，绝不会被当作指令执行。
• 🔌 **适用于 MCP 支持的所有环境：**Claude Code、Claude Desktop、Cursor 或 CLI。

你需要从 console.typesafe.ai 获取 Jev API key，并安装 uv。

Claude Code

claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx jevcrawl

Claude Desktop、Cursor 及其他 MCP 客户端（客户端配置文件中的 mcpServers）

{
  "mcpServers": {
    "jevcrawl": {
      "command": "uvx",
      "args": ["jevcrawl"],
      "env": { "TYPESAFE_API_KEY": "<your key>" }
    }
  }
}

然后向你的 Agent 提出类似的问题：“使用 jevcrawl 从 docs.python.org 开始，查找是哪个 Python 版本加入了海象运算符。” 可通过 uvx jevcrawl doctor 检查配置。能够加载技能的 Agent 还可以使用 skills/jevcrawl/SKILL.md。

每个系统都从一个网站首页开始，拥有相同的 30 页预算和相同的问题。每个系统收集到的内容都会交给同一个回答 LLM，并由同一个评判器对答案评分，因此唯一变化的因素就是爬虫本身。

Crawl4AI 是最受欢迎的开源 LLM 爬虫之一。其带有关键词评分器的 BestFirstCrawlingStrategy，是它在不使用 LLM 的情况下向问题答案方向爬取的方法，因此我们比较了两者能否抓取包含答案的页面。

数据来源

基准测试代码及复现所有数据的方法见：bench/。

{"blocks": [{"url": "https://docs.python.org/3/whatsnew/3.8.html",
             "heading_path": "What’s New In Python 3.8 > New Features > Assignment expressions",
             "text": "There is new syntax := that assigns values to variables as part of a larger expression. ...",
             "char_start": 307, "char_end": 580, "relevance": 0.97}],
 "uncertain": [],
 "pages": [{"url": "https://docs.python.org/3/", "title": "3.15.0 Documentation", "lang": "en",
            "text_sha256": "b624ce93..."}],
 "stats": {"pages_fetched": 6, "tokens_raw": 80034, "tokens_returned": 1873, "jev_calls": 11,
           "cache_hits": 0, "jev_input_tokens": 164139, "jev_cost_usd": 0.0069,
           "model": "jev-1.13.0", "degraded": false, "degraded_reason": null,
           "needs_js_urls": [], "stopped_because": "answered", "elapsed_seconds": 8.3}}

被裁剪为一个内容块和一个页面。80,034 个页面 token 被转化为 1,873 个答案 token，成本为 0.0069 美元，耗时 8.3 秒。

• blocks 是相关段落，按相关性从高到低排列；uncertain 是边界情况，在判定缺失信息前值得阅读的内容。
• char_start/char_end 是页面文本中的索引，对应文本的 sha256 位于 pages 中，因此引用可以在之后被核验。
• stopped_because 的值可以是 answered、frontier_empty、max_pages、max_seconds、budget 或 auth_error。
• 预期内的失败会以 {"error": "<code>", "message": "<what to do>"} 返回；工具绝不会向 Agent 会话抛出异常。

同样的操作也可以通过命令行执行（uv tool install jevcrawl 或 pip install jevcrawl 会将 jevcrawl 加入你的 PATH）：

jevcrawl crawl https://docs.python.org/3/ --goal "What does the walrus operator do?" [--json]
jevcrawl scrape https://example.com/help/returns [--goal "..."] [--json]
jevcrawl map https://example.com --goal "..." [--limit 20]
jevcrawl doctor

• Jev 负责决策，而不负责写作。 Jev 针对每个是/否问题返回概率，因此不能生成文本。你收到的一切内容均复制自原始页面。
• 页面文本是数据，而不是指令。 它只会在问题的数据字段中传递；每一句问题文本都是由 jevcrawl 编写的固定字符串。注入的“忽略之前的指令”最多只能影响评分结果。
• 低于优先级下限的链接会被保留备用，而不是被丢弃，因此只要仍有页面预算，爬取就不会提前停止。
• 故障会降级，但不会被隐藏。 在三次 Jev 请求失败后，爬取将继续以广度优先方式进行，将所有未评分内容块放入 uncertain 返回，并设置 stats.degraded。
• 重复操作免费。 Jev 的回答会被本地缓存（~/.jevcrawl/cache），因此针对相同页面执行相同目标不会产生费用。

其他所有设置（阈值、每页链接数、并发数）均在 jevcrawl/config.py 中采用经过测量的默认值，Python 调用方也可以传入自己的 Config。

Jev 仅针对输入 token 收费，jev-1.13.0 的价格为每百万 token 0.042 美元；在基准测试中，一次爬取的平均成本为 0.011 美元。max_decision_usd 是硬性限制：任何可能超出该额度的请求都不会被发送，爬取会以 budget 停止。每个结果均报告 jev_cost_usd，以及 tokens_raw（普通爬取会交给你的 LLM 的内容量）与 tokens_returned。

页面文本会被发送给 TypeSafe（内容块、链接标签及 URL）以进行评分。API key 从环境变量中读取，绝不会写入磁盘或日志。

jevcrawl 不读取 robots.txt，除了每次并发抓取四个页面外，也不自行进行速率限制。你需要自行对所爬取的网站、其服务条款及适用于你的法律负责。它不会登录，不会尝试绕过付费墙或 CAPTCHA，也不运行 JavaScript（需要 JavaScript 的页面会列在 stats.needs_js_urls 中）。每次调用均受 max_pages、max_seconds、max_decision_usd 和 return_tokens 的限制。

uv sync
uv run ruff check . && uv run ruff format --check . && uv run mypy --strict jevcrawl
uv run pytest                                    # offline: all HTTP is mocked
TYPESAFE_API_KEY=... uv run pytest -m live -s    # one tiny real request, a fraction of a cent

测试绝不会访问网络或花费金钱；任何会这样做的测试都会标记为 live。README 中的图表由 assets/build.py 根据测量数据绘制。欢迎提交 Issue 和 Pull Request；请按照 SECURITY.md 中的说明报告安全问题。

Apache-2.0

Agents that browse a site have two bad options. They can read everything a crawler fetches, which buries the answer in navigation and boilerplate and burns tokens. Or they can let the LLM pick each link , which is slow and costs dollars per question. jevcrawl does neither:

• 🎯 Finds the answer page more often. 54% of tasks vs 33% when Claude picks the links, and 2x as often as Crawl4AI's goal-directed crawl.
• 💸 About 8x cheaper per task, 10x per correct answer. The crawl decisions cost about a cent.
• ⚡ 2x faster than an LLM-driven crawl. Four pages at a time, one decision request per page.
• 📜 Verbatim, with sources. Every passage is copied from the page with its URL, heading path and character offsets. The decision model only scores, so it cannot invent text.
• 🛡️ Safe against prompt injection by design. Page text is only ever scored, never followed.
• 🔌 Works everywhere MCP does: Claude Code, Claude Desktop, Cursor, or the CLI.

You need a Jev API key from console.typesafe.ai and uv.

Claude Code

claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx jevcrawl

Claude Desktop, Cursor and other MCP clients (mcpServers in the client's config file)

{
  "mcpServers": {
    "jevcrawl": {
      "command": "uvx",
      "args": ["jevcrawl"],
      "env": { "TYPESAFE_API_KEY": "<your key>" }
    }
  }
}

Then ask your agent something like "Use jevcrawl to find which Python version added the walrus operator, starting from docs.python.org". Check the setup with uvx jevcrawl doctor. Agents that load skills can also use skills/jevcrawl/SKILL.md.

Every system starts on a site's home page with the same 30-page budget and the same question. What each one collects goes to the same answering LLM , and the same judge grades the answer, so the crawler is the only thing that changes.

Crawl4AI is one of the most popular open-source crawlers for LLMs. Its BestFirstCrawlingStrategy with a keyword scorer is its way to crawl toward a question without an LLM, so we compared the two on whether they fetch the page that holds the answer.

Where the numbers come from

|
---|---
Tasks | WebWalkerQA from WebWalker: Benchmarking LLMs in Web Traversal (Wu et al., Alibaba Tongyi Lab, ACL 2025): questions whose answers sit several clicks deep in real websites (conferences, universities, organizations, games), in English and Chinese.
Task selection | Each task was re-checked live before the run: the site and answer page still load and still contain the reference answer. Tasks looked at while developing are excluded (bench/tasks/seen-ids.txt).
Accuracy run | 50 tasks, 2026-10-12. Answering model claude-opus-5-5, judge claude-sonnet-5-5.
Crawl4AI run | 20 different tasks, 2026-10-13, Crawl4AI 0.9.4. Crawl only, no LLM: did the crawler fetch the gold page? In the accuracy run, whether a system fetched the gold page closely tracked whether it answered correctly.
Baselines | Claude picks every link : Claude reads each page and chooses the next link. Breadth-first : every page to depth 2. All systems share jevcrawl's fetcher and remove navigation and footers.
Costs | Jev cost is billed and exact. LLM costs are token counts at list price.
Caveats | 50 and 20 tasks are small samples: the cost gap is large and stable, the accuracy gap less so. Crawl4AI drives a real browser, jevcrawl does not run JavaScript.

The benchmark code and how to reproduce every number: bench/.

{"blocks": [{"url": "https://docs.python.org/3/whatsnew/3.8.html",
             "heading_path": "What’s New In Python 3.8 > New Features > Assignment expressions",
             "text": "There is new syntax := that assigns values to variables as part of a larger expression. ...",
             "char_start": 307, "char_end": 580, "relevance": 0.97}],
 "uncertain": [],
 "pages": [{"url": "https://docs.python.org/3/", "title": "3.15.0 Documentation", "lang": "en",
            "text_sha256": "b624ce93..."}],
 "stats": {"pages_fetched": 6, "tokens_raw": 80034, "tokens_returned": 1873, "jev_calls": 11,
           "cache_hits": 0, "jev_input_tokens": 164139, "jev_cost_usd": 0.0069,
           "model": "jev-1.13.0", "degraded": false, "degraded_reason": null,
           "needs_js_urls": [], "stopped_because": "answered", "elapsed_seconds": 8.3}}

Trimmed to one block and one page. 80,034 tokens of pages became 1,873 tokens of answer, for $0.0069, in 8.3 seconds.

• blocks are the relevant passages, best first; uncertain are borderline ones worth reading before concluding something is missing.
• char_start/char_end index into the page text whose sha256 is in pages, so a quote can be checked later.
• stopped_because is answered, frontier_empty, max_pages, max_seconds, budget or auth_error.
• Expected failures come back as {"error": "<code>", "message": "<what to do>"}; the tools never raise into the agent's session.

The same operations work from the command line (uv tool install jevcrawl or pip install jevcrawl puts jevcrawl on your path):

jevcrawl crawl https://docs.python.org/3/ --goal "What does the walrus operator do?" [--json]
jevcrawl scrape https://example.com/help/returns [--goal "..."] [--json]
jevcrawl map https://example.com --goal "..." [--limit 20]
jevcrawl doctor

• Jev decides, it does not write. Jev returns a probability per yes/no question, so it cannot produce text. Everything you get back is copied from the page.
• Page text is data, never instructions. It travels only in data fields of a question; every question sentence is a fixed string written by jevcrawl. An injected "ignore previous instructions" can at worst bias a score.
• Links below the priority floor are kept in reserve , not dropped, so a crawl never stops early while it still has page budget.
• Failures degrade, they do not hide. After three failed Jev requests the crawl continues breadth-first, returns every block unscored in uncertain, and sets stats.degraded.
• Repeats are free. Jev's answers are cached locally (~/.jevcrawl/cache), so the same goal on the same pages costs nothing.

Everything else (thresholds, links per page, concurrency) has a measured default in jevcrawl/config.py, and Python callers can pass their own Config.

Jev charges for input tokens only, $0.042 per million for jev-1.13.0; in the benchmark a crawl cost $0.011 on average. max_decision_usd is a hard limit: a request that could pass it is never sent, and the crawl stops with budget. Every result reports jev_cost_usd, and tokens_raw (what a plain crawl would hand your LLM) against tokens_returned.

Page text is sent to TypeSafe (blocks, link labels and URLs) to be scored. The API key is read from the environment and never written to disk or logs.

jevcrawl does not read robots.txt and does not rate-limit itself beyond fetching four pages at a time. You are responsible for the sites you crawl, their terms of service and the law that applies to you. It never logs in, does not try to get past paywalls or CAPTCHAs, and does not run JavaScript (pages that need it are listed in stats.needs_js_urls). Every call is bounded by max_pages, max_seconds, max_decision_usd and return_tokens.

uv sync
uv run ruff check . && uv run ruff format --check . && uv run mypy --strict jevcrawl
uv run pytest                                    # offline: all HTTP is mocked
TYPESAFE_API_KEY=... uv run pytest -m live -s    # one tiny real request, a fraction of a cent

Tests never touch the network or spend money; anything that does is marked live. The README charts are drawn by assets/build.py from the measured numbers. Issues and pull requests are welcome; please report security problems as described in SECURITY.md.

Apache-2.0