官方更新 AWS ML Blog Fri, 11 Sep 2026 正文 ok 2249 words

超越每 Token 价格:为工作负载选择 Amazon Bedrock 上合适的 OpenAI 模型

英文标题:Beyond the price per token: Choosing the right OpenAI model on Amazon Bedrock for your workload

企业在比较生成式 AI 模型时,不能只看每百万 Token 的价格,更应衡量每个正确答案、每条 Agent 轨迹和每份合格专业交付物的实际成本。本文基于开源基准测试工具,对 Amazon Bedrock 上的 gpt-5.6-luna、gpt-5.6-terra、gpt-5.6-sol,以及 OpenAI API 上的 gpt-5.4-mini、gpt-5.4-nano 进行了比较。测试显示,在所测样本中,luna 在正确答案成本、Agent 成功轨迹成本和 GDPval 合格交付物成本上均表现突出;terra 和 sol 则在更高质量门槛的任务中体现出能力优势。文章同时强调,Agent 的轮次效率、上下文重复传输、重试策略和输出长度上限,都会显著影响生产环境的总成本与质量。
打开原文相关 6.3 · 机会 4
media1.3 MB media101.1 KB media124.1 KB media345.0 KB media141.2 KB media346.5 KB media75.5 KB media99.8 KB media101.4 KB media25.7 KB media32.7 KB media22.7 KB media22.8 KB media204.1 KB

中文

构建生成式 AI 应用的组织通常以同一种方式比较模型:每百万 Token 的美元价格。这是每个定价页面上的数字,因此也成为每张电子表格中的数字。但生产工作负载购买的不是 Token,而是结果:一张已解决的支持工单、一份完成的研究简报、一个正确的财务摘要。在定价页面与最终结果之间,存在标价所忽略的多个乘数:模型答对的频率、达到正确结果所需的 Token 数量,以及对于 Agent 工作负载而言所需的轮次,因为每一轮都会重新发送不断增长的对话内容。

本文分享一套开源基准测试工具的结果。该工具衡量了 Amazon Bedrock 上 OpenAI 模型(gpt-5.6-luna、gpt-5.6-terra 和 gpt-5.6-sol)以及 OpenAI API 上两个广泛使用的高成本效率模型(gpt-5.4-mini 和 gpt-5.4-nano)的这些乘数。我们选择后两者作为许多团队起步时采用的成本优化基线,而不是将其视为同代模型进行一对一比较,因为我们最常听到的问题是:“我们现在运行的是 mini 或 nano。Amazon Bedrock 上更新的模型值得迁移吗?”我们重点关注三个问题:

  1. 一个正确答案的成本是多少? 而不是一个 Token 的成本。
  2. Agent 轨迹的成本是多少? 轮次可能主导最终账单。
  3. 它能否产出专业人士愿意接受的成果? 这通过真实职业交付物衡量,而非答题测试。

这里的核心基准测试具备可复现性:测试工具 openai-on-aws/benchmarks-openai 通过同一条代码路径(OpenAI Responses API)对两个应用进行测试。我们鼓励你在作出任何决策前,使用自己的任务运行它。

方案概述

该测试工具通过同一个 Responses API 客户端评估全部五个模型,在保持评估逻辑不变的前提下,仅切换后端和模型 ID。结果仍会反映模型、提供商基础设施及模型特定配置之间的差异。尤其是,Amazon Bedrock 上的模型关闭了推理功能,而 OpenAI API 基线模型则采用默认配置。这是对实际部署配置的比较,而不是对模型内在能力的受控估算。

我们衡量三类指标:在仍能区分前沿模型的基准上,衡量单次调用的准确率和成本(AIME 竞赛数学、GPQA Diamond 研究生级科学问答和 MMLU-Pro);在实时 Web 研究任务上衡量多轮 Agent 轨迹;以及衡量按评分量规评判的专业交付物。评分结合确定性检查和大语言模型(LLM)评审器(gpt-5.5,并非被评估模型之一),并使用固定提示词;这些提示词的哈希值会记录在每个结果文件中。每次运行均会写入带时间戳的结果 JSON 文件,本文中的每个数字和图表都在构建时由这些文件生成。

结果解读方式:样本量范围为 48–198 个项目。除非同时提供不确定性估计,否则应将较小差距视为方向性信号;在选择模型之前,请在自己的工作负载上复现评估。

衡量一个正确答案的成本

对于每项基准,我们将模型在正确和错误尝试上的总支出除以其正确答案数量。这估算了样本中观测到的每个正确答案成本。下图展示了全部五个模型在 AIME 上的准确率及每个正确答案成本。相同模式也出现在 GPQA Diamond 和 MMLU-Pro 上(完整表格见代码库)。

展示五个模型 AIME 准确率与每个正确答案成本的图表,其中 luna 成本最低,sol 准确率最高

图 1:五个模型的 AIME 准确率和每个正确答案成本

有两点尤为突出:

  • 本样本中可以看到能力层级。 Sol 解出了 75% 的 AIME 问题,mini 为 37%;在 GPQA Diamond 上,sol 以 68% 对 43% 领先;在 MMLU-Pro 上,sol 为 82%,mini 为 59%。如果各次尝试相互独立且准确率保持不变,37% 的准确率意味着平均每次成功大约需要 2.7 次尝试。实际重试彼此相关,因此应衡量实际重试策略,而非假设这一估计可以直接迁移。
  • 在价格变动之前,Token 效率已经决定了账单;随后价格也发生了变化。 按最初的标价(约为 mini 的 1.5 倍),luna 在这一配置下每个正确 AIME 答案的成本已经低 25%,因为在关闭推理功能时,它使用的计费 Token 少于默认配置下的 mini。根据 2026 年 7 月 30 日 Amazon Bedrock 对 GPT-5.6 Luna 和 Terra 的降价公告(luna 降价 80%,terra 降价 20%),我们记录的每个正确 AIME 答案成本为:luna 0.0021 美元,mini 0.0139 美元。在这些样本中,luna 的观测每个正确答案成本最低,其中也包括与 nano 的比较;尽管按照结果文件记录的价格假设,nano 的名义 Token 价格略低。**定价说明:**结果文件目前对 luna 使用每 100 万输入/输出 Token 0.22/1.32 美元的价格,对 terra 使用 2.20/13.20 美元的价格。发布前,请根据实时定价页面确认适用的 Amazon Bedrock 推理层级和区域。7 月 30 日的公告描述了第一方价格一致性,而定价页面的更新可能存在滞后。对客户而言,更有价值的模型选型问题不只是“哪个模型每 Token 最便宜?”,而是“在我的业务可接受的质量水平下,哪个模型每个结果最便宜?”

衡量 Agent 轨迹成本

单次调用定价遗漏了我们 Agent 工作负载的一项决定性特征。该测试工具使用由客户端管理的历史记录,并设置 store: false,因此每一轮都会重新发送系统提示词、此前的工具结果和对话上下文。每轮上下文大致线性增长,因此累计计费输入量可能随轮次近似二次增长。每一轮还会增加一次延迟往返。相比在八轮内完成,在五轮内完成的模型所节省的成本可能不止轮次减少 37% 所暗示的程度。

为在真实工作负载上衡量这一点,我们使用真实的 web_search 和 fetch_page 工具,通过实时 Agent 循环运行了 DeepSearchQA 的一个包含 50 个问题的分层样本(多步骤 Web 研究问题)。答案先经过确定性预处理,再由固定的 gpt-5.5 自动评分器打分。当 F1 ≥ 0.7 时,一个问题视为通过。下图展示轮次如何驱动输入 Token 量(左侧),以及每个通过答案最终的成本(右侧)。

双面板图表,展示轮次如何驱动每题输入 Token 量,以及各模型每个通过 DeepSearchQA 答案的成本

图 2:轮次驱动的输入 Token 量与每个通过 DeepSearchQA 答案的成本

左侧面板展示的是机制,右侧面板展示的是结果。Mini 在所有模型中每题轮次最多(7.6 轮,主要是重复搜索循环),而每增加一轮,都会重新发送包含累积搜索结果的上下文。最终,其每题输入 Token 量达到 terra 的 2.3 倍(11.4 万 Token 对 5 万 Token)。在本样本中,terra 较高的 Token 价格被更少的轮次和更高的质量所抵消:其每个通过答案成本为 0.31 美元,而 mini 为 0.40 美元;平均 F1 为 0.50,而 mini 为 0.39。记录显示,三种 GPT-5.6 配置的平均 F1 均高于两种基线模型;luna 的轮次少于 mini,每个通过答案成本为 0.05 美元,而 mini 为 0.40 美元。Nano 的名义 Token 价格更低,但其通过率为 18%,观测到的每个通过答案成本为 0.07 美元,高于 luna 的 0.05 美元。由于仅有 50 个问题,应将接近的差距视为方向性信号。

对客户而言,关键结论是:轮次效率是一个定价变量,但它在定价页面上不可见。如果你的 Agent 会串联工具调用(研究、多跳检索、迭代检索),应将轨迹成本与调用成本一同进行基准测试。

使用 GDPval 评估专业交付物

基准测试对答案进行评分。客户实际交付的许多内容是合规简报、财务计划和护理方案等文档;对于这类工作,“正确”由评分量规界定,而不是字符串匹配。因此,我们运行了 GDPval 的一个 48 任务切片:这些是真实职业交付物,由平均拥有 14 年经验的专业人士创建,每项均依据其人工编写的评分量规进行评估。当交付物达到加权评分点的 70% 或以上时视为通过。下图展示通过率和每份通过交付物的成本。

展示五个模型 GDPval 通过率与每份通过交付物成本的图表

图 3:GDPval 通过率和每份通过交付物的成本

在关闭推理功能的情况下,全部三种 gpt-5.6 配置的观测评分量规得分均高于 mini 和 nano。观测到的最大差异出现在法律、护理和财务建议任务中,这些任务的评分量规要求特定的注意事项、结构和完整性。各类别的样本量较小,因此应将这一模式视为探索性结果。Luna 在 48 份交付物中有 31 份得分高于 mini、9 份低于 mini、8 份持平。它通过了 27 份交付物,mini 通过 20 份。

该工作负载是没有轮次影响的单次调用,因此在重新定价前,mini 和 nano 的每份通过交付物成本更低,而 gpt-5.6 仅带来通过率提升。降价后,luna 在该样本中具有最低观测成本:每份通过交付物 0.010 美元,而 mini 为 0.030 美元、nano 为 0.012 美元;其通过率为 56%,mini 为 42%,nano 为 35%。Terra 和 sol 的每次通过成本仍较高,但记录到更高的评分量规表现。该溢价是否合理,取决于目标工作流中审核和返工的成本。

**GDPval 注意事项:**输出被限制为 8,192 Token,导致 6 份 luna、9 份 terra、7 份 sol、0 份 mini 和 1 份 nano 交付物被截断。这些是在测试上限下的真实结果。提高上限可能改善质量,但也可能增加成本。应同时测试这两个因素。

综合考量:决策框架

如果你当前运行的是 gpt-5.4-mini 或 nano,迁移问题取决于工作负载形态:

你的工作负载类似于…… 建议首先尝试 原因
高吞吐、低复杂度任务,失败成本低 Amazon Bedrock 上的 gpt-5.6-luna 在这些样本中,luna 记录到的每个成功结果成本最低。按照记录的价格假设,nano 的名义 Token 价格仍略低
交互式应用,准确性重要,且存在延迟服务级别目标(SLO) 首先对 Amazon Bedrock 上的 gpt-5.6-luna 进行基准测试 每次成功的观测成本表现强劲。延迟结果是特定时间点、特定区域的快照,应根据你的 SLO 重新测量
会串联工具调用的 Agent(研究、多跳任务) 对 Amazon Bedrock 上的 gpt-5.6-luna 和 -terra 进行基准测试 两者记录到的 DeepSearchQA F1 均高于 mini,且轮次更少。在这个 50 题样本中,luna 每个通过答案的成本约低 8 倍
有质量门槛的文档生产 Amazon Bedrock 上的 gpt-5.6-luna GDPval 上,成本效率模型中通过率最高(27/48,对比 mini 的 20/48),且每份通过交付物成本最低(0.010 美元,对比 0.030 美元)
在真正困难的工作中,准确率是硬性门槛 对 Amazon Bedrock 上的 gpt-5.6-sol 进行基准测试 其观测能力和价格层级不同(AIME 75%,GDPval 31/48)。当更低成本的配置无法达到质量门槛时,应考虑该模型

我们还使用同一流式测试工具,测量了两个应用中相同模型的延迟。在 2026 年 7 月的运行中(us-west-2、单区域、特定时间点),Amazon Bedrock 上 luna 的中位首 Token 时间在 12 个匹配配置中平均低 21%,terra 平均低 5%。共享服务会随负载变化,因此应将这些数据视为快照并重新测量。在输出长度 ≥500 Token 时,Amazon Bedrock 上 luna 的吞吐量平均高 43%(terra 为 +4%)。观测到的最坏 TTFT-to-median 比率在 Amazon Bedrock 上为 2.1–2.5×,而在 OpenAI API 上为 4.6–6.6×。这些最大值表明存在尾部波动,但并非 p99 延迟的估计。Sol 的表现不同:它是深度推理模型,首 Token 时间天然较长且波动较大,并且其 Amazon Bedrock 测试在 us-east-1 运行。三个模型的每配置详细信息均记录在代码库的性能部分,而 performance/run_all.sh 可在你的账户中复现该比较。

在你的工作负载上运行

这些结果反映了我们的样本量和配置选择:每个测试套件包含 50–198 个问题(DeepSearchQA 50 个、AIME 60 个、MMLU-Pro 140 个、GPQA Diamond 198 个)、48 份交付物,并且 Amazon Bedrock 上的模型关闭了推理功能,这是一个刻意设定的成本下限。启用推理会同时提高质量和支出。你的任务并不是我们的任务。具有持久价值的是方法论:

git clone https://github.com/openai-on-aws/benchmarks-openai
pip install -r requirements.txt
export OPENAI_API_KEY=sk-... # your own key; not stored
export AWS_REGION=us-west-2 # plus your standard AWS credentials
python quality/quick_evals.py # accuracy + cost per correct answer
python quality/deepsearchqa/run_deepsearchqa.py # live-web agent trajectories
python quality/gdpval_eval.py # professional deliverables, rubric-judged
performance/run_all.sh # latency, both applications

将 50–100 个你自己的任务替换进来,wi

English / 原文

Organizations building generative AI applications usually compare models the same way: dollars per million tokens. It’s the number on every pricing page, so it becomes the number in every spreadsheet. But production workloads don’t buy tokens. They buy outcomes: a resolved support ticket, a completed research brief, a correct financial summary. Between the pricing page and the outcome sit multipliers the sticker price ignores: how often the model is right, how many tokens it needs to get there, and, for agentic workloads, how many turns it takes, because every turn re-sends the growing conversation.

In this post, we share results from an open-source benchmarking harness that measures those multipliers across OpenAI models on Amazon Bedrock (gpt-5.6-luna, gpt-5.6-terra, and gpt-5.6-sol) and two widely used cost-efficient models on the OpenAI API (gpt-5.4-mini and gpt-5.4-nano). We chose the latter two as the cost-optimized baselines many teams start from, not as like-for-like generational peers, because “we run mini or nano today. Is a newer model on Amazon Bedrock worth it?” is the question we hear most. We focus on three questions:

  1. What does a correct answer cost? Not what a token costs.
  2. What do agent trajectories cost? Turn count can dominate the bill.
  3. Can it produce work a professional would accept? This is measured on real occupational deliverables, not quiz questions.

The core benchmarks here are reproducible: the harness, openai-on-aws/benchmarks-openai, runs one identical code path (the OpenAI Responses API) against both applications, and we encourage you to run it on your own tasks before making any decision.

Solution overview

The harness evaluates all five models through the same Responses API client, switching the backend and model ID while holding the evaluation logic constant. The results still reflect differences in the models, provider infrastructure, and model-specific configuration. In particular, the models on Amazon Bedrock ran with reasoning disabled, while the OpenAI API baselines ran at their defaults. This is a comparison of practical deployment configurations, not a controlled estimate of intrinsic model capability.

We measure three things: single-call accuracy and cost on benchmarks that still separate frontier models (AIME competition mathematics, GPQA Diamond graduate-level science, and MMLU-Pro), multi-turn agent trajectories on live web-research tasks, and rubric-graded professional deliverables. Grading combines deterministic checks with a large language model (LLM) judge (gpt-5.5, not one of the evaluated models) using frozen prompts whose hashes are recorded in every result file. Each run writes a timestamped result JSON, and every number and chart in this post is generated from those files at build time.

How to read the results: sample sizes range from 48–198 items. Treat small gaps as directional unless accompanied by uncertainty estimates, and reproduce the evaluation on your own workload before selecting a model.

Measuring the cost of a correct answer

For each benchmark we divide a model’s total spend, across right and wrong attempts alike, by its number of correct answers. This estimates the observed cost of a correct answer in our sample. The following graph shows accuracy and cost per correct answer on AIME for all five models. The pattern repeats on GPQA Diamond and MMLU-Pro (full tables in the repo).

Chart of AIME accuracy and cost per correct answer for the five models, with luna at the lowest cost and sol at the highest accuracy

Figure 1: AIME accuracy and cost per correct answer across the five models

Two things stand out:

  • Capability tiers are visible in this sample. Sol solves 75 percent of AIME problems to mini’s 37 percent, and leads GPQA Diamond (68 percent versus 43 percent) and MMLU-Pro (82 percent versus 59 percent). If attempts were independent and accuracy remained constant, 37 percent accuracy would imply roughly 2.7 attempts per success on average. Real retries are correlated, so measure your actual retry strategy rather than assuming that estimate transfers directly.
  • Token efficiency decided the invoice even before prices moved, and then prices moved. At its original list price (roughly 1.5× mini’s), luna was already 25 percent cheaper per correct AIME answer in this configuration, because with reasoning disabled it used fewer billed tokens than mini at its defaults. Following the July 30, 2026 price reduction for GPT-5.6 Luna and Terra on Amazon Bedrock (luna −80 percent, terra −20 percent), our recorded cost per correct AIME answer is $0.0021 for luna versus $0.0139 for mini. Across these samples, luna has the lowest observed cost per correct answer, including against nano, even though nano’s nominal token price is slightly lower under the price assumptions recorded in the result files.Pricing note: the result files currently use $0.22/$1.32 per 1M input/output tokens for luna and $2.20/$13.20 for terra. Confirm the applicable Amazon Bedrock inference tier and region against the live pricing page before publication. The July 30 announcement describes first-party price parity, while pricing-page updates may lag.For customers, the more useful model-selection question is not only “which is cheapest per token?” but “which is cheapest per outcome , at a quality level my business accepts?”

Measuring agent trajectory cost

Single-call pricing misses a defining feature of our agentic workload. The harness uses client-managed history with store: false, so every turn re-sends the system prompt, prior tool results, and conversation context. Per-turn context grows roughly linearly. Cumulative billed input can therefore grow approximately quadratically with turn count. Each turn also adds a round trip of latency. A model that finishes in five turns instead of eight can save more than the 37 percent reduction in turns alone suggests.

To measure this on a real workload, we ran a 50-question stratified sample of DeepSearchQA (multi-step web-research questions) through a live agent loop with real web_search and fetch_page tools. Answers were scored using a deterministic pre-pass followed by a frozen gpt-5.5 autorater. A question passes at F1 ≥ 0.7. The following graph shows how turn count drives input-token volume (left) and what each passing answer ends up costing (right).

Two-panel chart showing turn count driving input-token volume per question and the cost per passing DeepSearchQA answer by model

Figure 2: Turn count driving input-token volume and cost per passing DeepSearchQA answer

The left panel is the mechanism. The right panel is the consequence. Mini took the most turns per question of any model (7.6, mostly re-search loops), and every extra turn re-sent a context containing accumulated search results. By the end, that reached 2.3× terra’s input-token volume (114k versus 50k tokens per question). In this sample, terra’s higher token price was offset by fewer turns and higher quality: it cost $0.31 per passing answer versus mini’s $0.40, with mean F1 of 0.50 versus 0.39. The three GPT-5.6 configurations recorded higher mean F1 than both baselines, while luna combined fewer turns than mini with $0.05 per passing answer versus mini’s $0.40. Nano’s nominal token price was lower, but at an 18 percent pass rate its observed cost per passing answer was $0.07 versus luna’s $0.05. With 50 questions, treat close gaps as directional.

The customer takeaway: turn efficiency is a pricing variable, and it’s invisible on the pricing page. If your agents chain tool calls (research, multi-hop lookups, iterative retrieval), benchmark trajectory cost alongside call cost.

Evaluating professional deliverables with GDPval

Benchmarks grade answers. Much of what customers ship is documents such as compliance briefs, financial plans, and care protocols, where “correct” is a rubric, not a string match. So we ran a 48-task slice of GDPval: real occupational deliverables created by professionals averaging 14 years of experience, each graded against its human-authored rubric. A deliverable passes at ≥70% of weighted rubric points. The following graph shows pass rates and cost per passing deliverable.

Chart of GDPval pass rates and cost per passing deliverable across the five models

Figure 3: GDPval pass rates and cost per passing deliverable

All three gpt-5.6 configurations achieved higher observed rubric scores than mini and nano with reasoning disabled. The largest observed differences appeared in law, nursing, and financial-advice tasks, where rubrics demand specific caveats, structure, and completeness. The per-category samples are small, so treat this pattern as exploratory. Luna scored higher than mini on 31 of 48 deliverables, lower on 9, and tied on 8. It passed 27 deliverables to mini’s 20.

This workload is a single call with no turn effects, so before the repricing, mini and nano were cheaper per passing deliverable and gpt-5.6 bought pass rate alone. After repricing, luna has the lowest observed cost in this sample: $0.010 per passing deliverable versus mini’s $0.030 and nano’s $0.012, at a 56 percent pass rate to mini’s 42 percent and nano’s 35 percent. Terra and sol still carry a premium per pass while recording higher rubric performance. Whether that premium is justified depends on the cost of review and rework in the target workflow.

GDPval caveat: outputs were capped at 8,192 tokens, truncating 6 luna, 9 terra, 7 sol, 0 mini, and 1 nano deliverables. These are real outcomes under the tested cap. A higher cap may improve quality, but it might also increase cost. Test both together.

Putting it together: a decision framework

If you run gpt-5.4-mini or nano today, the migration question comes down to workload shape:

Your workload looks like… Start with Why
High-volume, low-complexity tasks. Failures are cheap gpt-5.6-luna on Amazon Bedrock Luna recorded the lowest cost per successful outcome across these samples. Nano retains a slightly lower nominal token price under the recorded price assumptions
Interactive apps. Accuracy matters. Latency service-level objectives (SLOs) Benchmark gpt-5.6-luna on Amazon Bedrock first Strong observed cost per success. Latency results are a point-in-time regional snapshot. Re-measure them against your SLOs
Agents that chain tool calls (research, multi-hop) Benchmark gpt-5.6-luna and -terra on Amazon Bedrock Both recorded higher DeepSearchQA F1 with fewer turns than mini. Luna cost about 8× less per passing answer in this 50-question sample
Quality-gated document production gpt-5.6-luna on Amazon Bedrock Highest pass rate among the cost-efficient models on GDPval (27/48 versus mini’s 20/48) at the lowest cost per passing deliverable ($0.010 versus $0.030)
Accuracy is a hard gate on genuinely difficult work Benchmark gpt-5.6-sol on Amazon Bedrock A different observed capability and price tier (AIME 75%, GDPval 31/48). Consider it when lower-cost configurations do not clear your quality gate

We also measured same-model latency on both applications with the same streaming harness. In our July 2026 runs (us-west-2, single-region, and point-in-time), median time-to-first-token on Amazon Bedrock averaged 21 percent lower for luna and 5 percent lower for terra across the 12 matched configurations. Shared services vary with load, so treat these as a snapshot and re-measure. Luna’s throughput averaged 43 percent higher on Amazon Bedrock at ≥500-token outputs (terra +4 percent). The observed worst-case TTFT-to-median ratios were 2.1–2.5× on Amazon Bedrock versus 4.6–6.6× on the OpenAI API. These maxima indicate tail variability but are not estimates of p99 latency. Sol behaves differently: it’s a deep-reasoning model with inherently long and variable time-to-first-token, and its Amazon Bedrock runs used us-east-1. Per-configuration detail for all three models is documented in the repository’s performance section, and performance/run_all.sh reproduces the comparison on your own account.

Run it on your workload

These results reflect our sample sizes and configuration choices: 50–198 questions per suite (DeepSearchQA 50, AIME 60, MMLU-Pro 140, GPQA Diamond 198), 48 deliverables, and reasoning disabled for the models on Amazon Bedrock, a deliberate cost floor. Enabling it raises both quality and spend. Your tasks are not our tasks. The methodology is the durable part:

git clone https://github.com/openai-on-aws/benchmarks-openai
pip install -r requirements.txt
export OPENAI_API_KEY=sk-... # your own key; not stored
export AWS_REGION=us-west-2 # plus your standard AWS credentials
python quality/quick_evals.py # accuracy + cost per correct answer
python quality/deepsearchqa/run_deepsearchqa.py # live-web agent trajectories
python quality/gdpval_eval.py # professional deliverables, rubric-judged
performance/run_all.sh # latency, both applications

Swap in 50–100 of your own tasks with known-good outputs and the same scripts produce cost-per-success numbers in your domain, on your account.

Conclusion

The per-token price is one input to a decision, not the decision. In the configurations we tested, normalizing cost by successful outcomes changed the model ranking after accuracy, token efficiency, and trajectory length were included. Pricing pages also move, as the July 2026 GPT-5.6 reductions showed. Define the outcome and quality threshold your application needs, measure the full cost of successes and failures, and re-run the evaluation when prices, models, or workload patterns change.

To get started with OpenAI models on Amazon Bedrock, see the Amazon Bedrock documentation. To reproduce these benchmarks on your own tasks, clone openai-on-aws/benchmarks-openai and follow the steps in the previous section.


About the authors

Nick McCarthy

Nick is a Senior Generative AI Specialist Solutions Architect on the Amazon Bedrock team, focused on OpenAI APIs and Codex. He has worked with AWS clients across a wide range of industries — including healthcare, finance, sports, telecommunications, and energy — helping them accelerate business outcomes with AI and machine learning.

Sudeesh Sasidharan

Sudeesh is a Member of Go-to-Market Staff at OpenAI, focused on OpenAI APIs and Codex. His work includes collaborating with AWS on Amazon Bedrock to help organizations adopt and deploy OpenAI’s frontier models.

Sharadha Kandasubramanian

Sharadha is a Bedrock GenAI Specialist at AWS, focused on OpenAI models on Amazon Bedrock. Her work sits at the intersection of go-to-market strategy, field enablement, and customer adoption — shaping how AWS brings OpenAI’s frontier models to market and helping organizations move from evaluation to production. Her focus is on translating product capabilities into actionable GTM motions that accelerate real-world generative AI deployments at scale.

Saurabh Trikande

Saurabh is a Senior Product Manager for Amazon Bedrock and Amazon SageMaker Inference. He is passionate about working with customers and partners, motivated by the goal of democratizing AI. He focuses on core challenges related to deploying complex AI applications, inference with multi-tenant models, cost optimizations, and making the deployment of generative AI models more accessible. In his spare time, Saurabh enjoys hiking, learning about innovative technologies, following TechCrunch, and spending time with his family.