AI News
商业产业 Hacker News AI 2026-09-25T17:16 正文 ok 1569 words

🔶 Jev 对决 Kev：开源 Jev 替代方案并排实测

英文标题：🔶 Jev vs. Kev: open-source Jev alternative tested side by side

TypeSafe 于 9 月 15 日推出 Jev，这是一类可读取状态并针对类型化问题返回各选项概率的新模型；在其架构和训练数据未公开的情况下，一周内已出现约 30 个基于 API 契约复现的开源版本。测试将开源模型 Kev 4B 与 Jev 接入同一 Opper API 端点，在 362 道发布时间晚于两者发布日的新题目上比较，二者各项准确率相差均不超过 2 个百分点。Jev 的概率校准整体更好，但每次请求会额外计入约 257 个输入 token，导致短请求成本约为 Kev 的 12 倍；随着输入变长，成本差距收窄。两者延迟接近，Kev 并发 8 路请求时中位延迟约 220 毫秒，Jev 约 275 毫秒，说明开源复现已可在直接分类任务上接近闭源原版，但在校准、细微语义和长文档处理上仍有差距。
打开原文相关 2 · 机会 2
media95.7 KB

中文

TypeSafe 于 9 月 15 日发布了 Jev。这是一种新型模型：它不生成文本，而是读取一段状态，并针对类型化问题（是/否、单选、按量表评分）作答，同时为每个选项返回概率。TypeSafe 尚未公开其架构或训练数据。一周之内，已经出现约 30 个仅依据 API 契约重建的开源复现版本。

我们想知道，其中一个复现版本究竟能接近原版到什么程度。因此，我们选择了 Jared Palmer 开发、讨论度最高的复现版本之一 Kev 4B，将其自行托管在欧盟，并接入 Jev 已在 Opper 上使用的同一端点。随后，我们向两个模型提出了相同的问题。

简要结论如下：

  • 准确率很接近。 在两个模型发布后编写的 362 道题目上，两个模型在每项任务上的差距都不超过 2 个百分点；对于这一样本规模而言，这处于噪声范围内。
  • 两者对 token 的计数方式截然不同。 同一请求、相同标价下，Jev 会为每次请求固定额外计入约 257 个输入 token。
  • 速度相近。 通过 Opper，Kev 的响应时间约为 220 毫秒，Jev 约为 275 毫秒。

一个端点，只改一个字符串

Kev 使用与 Jev 相同的线传输格式，因此进行对比只需更换模型名称：

curl https://api.opper.ai/v3/compat/v1/systemone \
  -H "Authorization: Bearer $OPPER_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "opper/kev-4b",
    "state": {"message": "I was charged twice for my order and nobody has replied."},
    "questions": {
      "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
      "team":   {"type": "choice", "instructions": "Which team should handle this?",
                 "criteria": {"billing": "Billing", "support": "Tech support", "sales": "Sales"}}
    }
  }'

将 opper/kev-4b 替换为 typesafe/jev-latest，其他内容完全不变：请求、响应结构、概率以及每次调用的成本响应头。

Kev 是什么

Kev 4B（Hugging Face 模型卡、GitHub 代码）是 Jared Palmer 基于阿里巴巴 Qwen3.5-4B-Base 进行的微调模型，并增加了一个可通过单次前向传播读取答案的小型头部。它采用 Apache-2.0 许可证，并为每个选项返回概率。我们将其托管在欧盟。

为什么我们构建了自己的测试集

基准测试的自然选择本应是知名公共数据集。但问题在于，Kev 已在其中许多数据集上训练过：AG News、Banking77、BoolQ、MNLI、SST-5、Yelp reviews、IMDb、Amazon reviews、DBpedia、TREC、ARC、OpenBookQA 和 CommonsenseQA。

尽管如此，我们仍尝试了其中几个；正如预期，Kev 在大多数数据集上与 Jev 持平或优于 Jev。

因此，我们使用在两个模型发布后发表的文本构建了一个新的小型数据集：新的 arXiv 论文、Stack Exchange 问题和 GitHub issue。两个模型都不可能见过其中任何内容。

方法

同一端点，每次请求一个问题。 每个条目均通过 Opper 的 POST /v3/compat/v1/systemone，以完全相同的请求体发送给两个模型。只有 model 字段不同：typesafe/jev-latest 或 opper/kev-4b。每次请求只携带一个问题，同时有 8 个请求在飞行中，我们记录每次调用返回的答案、延迟和 Opper 成本响应头。

两个模型都不可能见过的数据。 标题数据来自 362 个发表于 2026-09-20 之后的条目，此时 Jev 1.13（发布于 2026-09-15）和 Kev 4B 均已发布。真实标签来自原始来源，而非任何模型：

任务 来源 问题 答案来源 n
arXiv 分类 arXiv API，8 个分类中的最新论文 “这篇论文的主要主题属于哪个 arXiv 分类？”（8 个选项） 论文的主要分类 160
Stack Exchange 站点 Stack Exchange API，6 个站点中的最新问题 “这个问题发布在哪个问答社区？”（6 个选项） 其发布所在的站点 120
GitHub bug 或功能 VS Code、Godot、Rust、Windows Terminal、Ollama、Flutter 中的新 issue “该 issue 是否为 bug 报告（而非功能请求）？”（是/否） 维护者标签 82

分类和站点均做了平衡处理（每类 20 个条目）；GitHub bug 被下采样，以匹配 41 个功能请求；测试集采用固定随机种子冻结，因此每个模型看到的条目完全相同。以下是两个真实示例：

{"model": "opper/kev-4b",
 "state": {"title": "Is there any function you can integrate with respect to sqrt(dx) and get a finite nonzero result?",
           "body": "Messing around trying to build intuition for manipulating differentials, I had the idea to ..."},
 "questions": {"q": {"type": "choice", "instructions": "Which Q&A community was this question posted on?",
   "criteria": {"math": "Mathematics", "physics": "Physics", "askubuntu": "Ask Ubuntu (Ubuntu Linux)",
                "superuser": "Super User (general computing)", "electronics": "Electrical engineering and electronics",
                "diy": "Home improvement (DIY)"}}}}

该问题发布于 Mathematics。Jev 给出 math 的概率为 1.00，Kev 给出 math 的概率为 0.93。对于一个标题为“f16b.rs fails on LoongArch”的 Rust issue，维护者将其标为 bug；Jev 判断为 bug 的概率是 0.97，而 Kev 的判断为 0.48，恰好落在错误一侧。

评分。 准确率按最高概率答案计算（是/否问题以 0.5 为阈值）。校准误差将声明的置信度与 10 个分箱中的实际命中率进行比较；越低越好。每 1,000 次调用成本为 Opper 每次调用成本响应头的平均值乘以 1,000，按测试时（2026 年 9 月）的标价计算：两个模型的输入价格均为每百万 token 0.042 美元，输出免费。

次要验证使用 Kev 未训练过的五个较旧数据集：emotion、offensive tweets、QNLI、PAWS、SciQ。

准确率与置信度

任务 Jev 准确率 Kev 准确率 Jev 校准误差 Kev 校准误差 Jev 每 1,000 次调用成本 Kev 每 1,000 次调用成本
arXiv 分类（8 个选项，n=160） 96.9% 95.0% 0.032 0.044 $0.030 $0.015
Stack Exchange 站点（6 个选项，n=120） 97.5% 98.3% 0.027 0.125 $0.026 $0.012
GitHub bug 或功能（是/否，n=82） 95.1% 93.9% 0.049 0.137 $0.025 $0.013

任一方向上的准确率差异都仅为几个百分点，处于这一样本规模的噪声范围内。更明显的差异在于校准：Jev 的声明概率与其实际答对频率更一致，而 Kev 的偏差更大，尤其是在 Stack Exchange 和 GitHub 任务中。

在五个较旧的公共数据集上，Jev 在释义检测（PAWS，87.0% 对 74.5%）上明显领先，在 QNLI 和 SciQ 上也略有优势；Kev 则在 emotion 和 offensive tweets 上领先。Kev 从未在这些数据集上训练，但其作者曾使用这些数据集选择最佳训练运行，因此我们将其视为次要信号。

Token 计费带来的意外

测试时（2026 年 9 月），两个模型的标价相同：每百万输入 token 0.042 美元，输出免费，并且接收完全相同的请求。差异在于计入了什么。我们将相同请求直接发送给 TypeSafe 和通过 Opper 发送，得到的计数完全一致，因此这是 Jev 自身的计费方式，而不是我们的计数问题。文本增长时，两个模型增加的 token 数相同；但 Jev 会为每次请求固定增加约 257 个输入 token 的计费，最可能是其内部提示词。一条单行消息加一个问题，在 Jev 上计为 280 个 token，在 Kev 上则为 23 个。

因此，每次调用的成本差距完全取决于输入长度：

请求 Jev token 数 Kev token 数 Jev 成本相对 Kev
单行消息，1 个问题 280 23 12 倍
arXiv 摘要，1 个问题 约 720 约 360 2 倍
长消息（约 40 句话），1 个问题 960 703 1.4 倍
长消息，单次请求中 5 个问题 1,030 777 1.3 倍

这部分开销按请求计算，而非按问题计算，因此在同一请求中围绕同一文本提出多个问题可以摊薄开销。我们的基准每次请求只发送一个问题，这正是对 Jev 最不利的情况。

速度

从斯德哥尔摩通过 Opper 逐个发送请求时，Kev 的响应时间约为 180 至 200 毫秒，Jev 约为 265 毫秒。在同时有 8 个请求在飞行中时，Kev 的中位延迟约为 220 毫秒，Jev 约为 275 毫秒。对于两个模型而言，大部分时间花在网络往返而非模型本身。Kev 在空闲时会缩容，因此静默一段时间后的首次调用会更慢。

我们的结论

一个发布仅一周的开源复现版本，在没有获取 Jev 架构或训练数据的前提下，已经能在直接分类任务上达到令人意外的接近程度。它的不足也正是预期中一个基于少量公共数据集训练的小模型会落后的地方：校准后的置信度、细微语义和长文档。两者的对比还揭示了一项原本可能不会发现的情况：Jev token 计数中存在按请求收取的固定开销。

两个模型均已在 Opper 上通过同一端点提供服务，因此你可以用自己的问题同时测试二者，以判断哪一个更适合你的需求。

注意事项

  • 每个测试套件仅运行一次，且只有数百个条目。低于约 5 个百分点的差异处于噪声范围内。
  • 新任务对两个模型都相对容易（准确率均为 95% 及以上），这压缩了差异。更困难的新任务会更明显地区分二者。
  • 新数据并不等于新任务：尽管 Kev 从未见过这些条目，但它训练过主题、意图和是/否分类任务。Jev 的训练数据尚未公开。
  • 在较旧数据集上，Kev 的作者曾使用 emotion、offensive tweets、QNLI、PAWS 和 SciQ 在训练运行之间进行选择，因此 Kev 在这些数据集上的结果并非完全独立。
  • Kev 使用短输入训练（状态最多 384 个 token），容易遗漏深埋在长文档中的细节。
  • 延迟仅从一个地点、通过 Opper 测量。

链接

复现方法

基准代码、测试条目列表和我们的结果均位于 github.com/opper-ai/jev-vs-kev。使用 Opper API key，运行 python3 fetch_texts.py，然后运行 SUITE=fresh python3 jev_vs_kev.py。

English / 原文

TypeSafe launched Jev on 15 September. It is a new kind of model: instead of writing text, it reads a piece of state and answers typed questions (yes/no, pick one, score on a rubric) with a probability for every option. TypeSafe has not published the architecture or the training data. Within a week there were around 30 open reproductions, rebuilt from the API contract alone.

We wanted to know how close one of those copies actually gets. So we took Kev 4B, an open model by Jared Palmer and one of the most discussed reproductions, hosted it ourselves in the EU, and put it behind the same endpoint Jev already has on Opper. Then we asked both models the same questions.

What we found, in short:

  • Accuracy is close. On 362 questions written after both models were released, the two land within 2 points of each other on every task, which is inside the noise for samples this size.
  • They count tokens very differently. Same request, same list price, but Jev counts a fixed ~257 extra input tokens on every request.
  • Speed is similar. Through Opper, Kev answered in about 220 ms and Jev in about 275 ms.

One endpoint, one string changed

Kev speaks the same wire format as Jev, so the comparison needs nothing more than a different model name:

curl https://api.opper.ai/v3/compat/v1/systemone \
  -H "Authorization: Bearer $OPPER_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "opper/kev-4b",
    "state": {"message": "I was charged twice for my order and nobody has replied."},
    "questions": {
      "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
      "team":   {"type": "choice", "instructions": "Which team should handle this?",
                 "criteria": {"billing": "Billing", "support": "Tech support", "sales": "Sales"}}
    }
  }'

Swap opper/kev-4b for typesafe/jev-latest and nothing else changes: the request, the response shape, the probabilities and the per-call cost header.

What Kev is

Kev 4B (model card on Hugging Face, code on GitHub) is a fine-tune by Jared Palmer of Alibaba's Qwen3.5-4B-Base with a small head that reads the answer from a single forward pass. It is Apache-2.0 and returns a probability for every option. We host it in the EU.

Why we built our own test set

The natural choice for a benchmark would have been well-known public datasets. The problem is that Kev was trained on many of them: AG News, Banking77, BoolQ, MNLI, SST-5, Yelp reviews, IMDb, Amazon reviews, DBpedia, TREC, ARC, OpenBookQA and CommonsenseQA.

We tried a few of them anyway, and as expected, Kev matched or beat Jev on most.

So we built a small new dataset from text published after both models were released: new arXiv papers, Stack Exchange questions and GitHub issues. Neither model could have seen any of it.

Methodology

Same endpoint, one question per request. Every item was sent to both models through POST /v3/compat/v1/systemone on Opper with an identical body. Only model differed: typesafe/jev-latest or opper/kev-4b. Each request carried a single question, 8 requests were in flight at a time, and we recorded the answer, the latency and the cost header Opper returns on every call.

Data neither model could have seen. The headline numbers come from 362 items published after 2026-09-20, after Jev 1.13 (released 2026-09-15) and Kev 4B were both out. The ground truth comes from the source, never from a model:

Task Source Question Answer from n
arXiv category arXiv API, newest papers in 8 categories "Which arXiv category is this paper's primary subject?" (8 options) The paper's primary category 160
Stack Exchange site Stack Exchange API, newest questions on 6 sites "Which Q&A community was this question posted on?" (6 options) The site it was posted on 120
GitHub bug or feature New issues in VS Code, Godot, Rust, Windows Terminal, Ollama, Flutter "Is this issue a bug report (as opposed to a feature request)?" (yes/no) The maintainers' label 82

Categories and sites were balanced (20 items each), GitHub bugs were sampled down to match the 41 feature requests, and the set was frozen with a fixed seed so every model saw the same items. Two real examples:

{"model": "opper/kev-4b",
 "state": {"title": "Is there any function you can integrate with respect to sqrt(dx) and get a finite nonzero result?",
           "body": "Messing around trying to build intuition for manipulating differentials, I had the idea to ..."},
 "questions": {"q": {"type": "choice", "instructions": "Which Q&A community was this question posted on?",
   "criteria": {"math": "Mathematics", "physics": "Physics", "askubuntu": "Ask Ubuntu (Ubuntu Linux)",
                "superuser": "Super User (general computing)", "electronics": "Electrical engineering and electronics",
                "diy": "Home improvement (DIY)"}}}}

Posted on Mathematics. Jev answered math at 1.00, Kev math at 0.93. On a Rust issue titled "f16b.rs fails on LoongArch", labelled a bug by the maintainers, Jev said bug at 0.97 and Kev hedged at 0.48, just on the wrong side of the line.

Scoring. Accuracy counts the top answer (yes/no at 0.5). Calibration error compares stated confidence with the actual hit rate across 10 bins; lower is better. Cost per 1,000 calls is the average of Opper's per-call cost header times 1,000, at list price when we ran it (September 2026): USD 0.042 per million input tokens on both models, output free.

The secondary check uses the five older datasets Kev did not train on (emotion, offensive tweets, QNLI, PAWS, SciQ).

Accuracy and confidence

Task Jev accuracy Kev accuracy Jev calibration error Kev calibration error Jev cost per 1k calls Kev cost per 1k calls
arXiv category (8 options, n=160) 96.9% 95.0% 0.032 0.044 $0.030 $0.015
Stack Exchange site (6 options, n=120) 97.5% 98.3% 0.027 0.125 $0.026 $0.012
GitHub bug or feature (yes/no, n=82) 95.1% 93.9% 0.049 0.137 $0.025 $0.013

Accuracy is within a couple of points either way, which is inside the noise for samples this size. The clearer difference is calibration: Jev's stated probabilities track how often it is actually right, while Kev's drift further, most on the Stack Exchange and GitHub tasks.

On the five older public datasets, Jev led clearly on paraphrase detection (PAWS, 87.0% vs 74.5%) and slightly on QNLI and SciQ; Kev was ahead on emotion and offensive tweets. Kev never trained on these, but its authors used them to pick their best training run, so we read them as a secondary signal.

The token accounting surprise

Both models listed at the same price when we ran this (September 2026), USD 0.042 per million input tokens with free output, and receive the identical request. The difference is in what gets counted. We sent the same requests to TypeSafe directly and through Opper and got identical counts, so this is Jev's own accounting, not ours. Growing the text adds the same number of tokens on both models, but Jev adds a fixed charge of about 257 input tokens to every request, most likely its internal prompt. A one-line message with one question counts as 280 tokens on Jev and 23 on Kev.

So the gap in cost per call depends entirely on input length:

Request Jev tokens Kev tokens Jev cost vs Kev
One-line message, 1 question 280 23 12x
arXiv abstract, 1 question about 720 about 360 2x
Long message (about 40 sentences), 1 question 960 703 1.4x
Long message, 5 questions in one request 1,030 777 1.3x

The overhead is per request, not per question, so asking several questions about the same text in one request spreads it out. Our benchmark sent one question per request, which is the least favourable case for Jev.

Speed

Sent one at a time through Opper from Stockholm, Kev answered in about 180 to 200 ms and Jev in about 265 ms. With 8 requests in flight, Kev's median was about 220 ms and Jev's about 275 ms. For both models most of that time is the network trip rather than the model itself. Kev is scaled down when idle, so the first call after a quiet period is slower.

What we take from it

A week-old open reproduction, built without access to Jev's architecture or training data, gets surprisingly close on straightforward classification. Where it falls short is exactly where you would expect a small model trained on a few public datasets to fall short: calibrated confidence, subtle meaning and long documents. And comparing the two turned up something we would not have found otherwise: the per-request overhead in Jev's token count.

Both models are on Opper behind the same endpoint, so you can run your own questions against both and see which one fits.

Caveats

  • One run per suite, a few hundred items. Differences under about 5 points are within noise.
  • The fresh tasks are fairly easy for both models (95% and up), which compresses the differences. Harder fresh tasks would separate them more.
  • New data is not new tasks: Kev has trained on topic, intent and yes/no classification, even though it never saw these items. Jev's training data is not published.
  • On the older datasets, Kev's authors used emotion, offensive tweets, QNLI, PAWS and SciQ to choose between training runs, so Kev's results there are not fully independent.
  • Kev was trained on short inputs (states of up to 384 tokens) and misses details buried deep in long documents.
  • Latency was measured from one location, through Opper.

Links

Reproduce it

The benchmark code, the list of test items and our results are at github.com/opper-ai/jev-vs-kev. With an Opper API key, run python3 fetch_texts.py and then SUITE=fresh python3 jev_vs_kev.py.