借助自我改进 Agent 加速端到端推理
英文标题：🔶 Speeding up end-to-end inference with self-improving agents
Asari AI 表示，其“协同发明者”自我改进 Agent 已针对运行在 NVIDIA B200 与 vLLM v0.23 上的 DeepSeek v4 Pro、GLM 5.2 优化完整推理栈，在多个最大并发级别下将吞吐量和交互性最高提升 16%。这些优化覆盖 kernel、调度器、负载均衡器及配置设置；截至 vLLM v0.26，其中大多数改动尚无上游等价实现。该系统通过端到端与微基准测试结合、噪声控制及统计检验来确认性能收益，并以 token 级概率分布匹配验证模型行为未被破坏。每轮实验产生的经验会被 Agent 复用，例如其已将避免分布式死锁的经验从 DeepSeek v4 Pro 迁移至 GLM 5.2，避免了 44 分钟的无谓等待。
664 B
28.0 KB
中文
借助自我改进 Agent 加速端到端推理
关于我们的协同发明者 Agent 如何通过严格的自我改进流程提升完整 AI 推理栈的早期成果。
2026 年 7 月 28 日
作者：Asari AI
引言
我们构建协同发明者：能够设计并优化高性能基础设施的自我改进 Agent。将其应用于 AI 推理后，这些 Agent 为两款最大的开源 LLM 优化了整个推理栈：在 NVIDIA B200 上运行、使用 vLLM v0.23 的 DeepSeek v4 Pro 和 GLM 5.2。在多个最大并发级别下，它们在通过分布级正确性检查保持模型行为不变的同时，将吞吐量和交互性最高提升 16%。Agent 针对每个并发级别大约需要一天完成优化。
截至 vLLM v0.26，我们的 Agent 所做的大多数改动都尚无上游等价实现。这些改动覆盖了完整推理栈，包括 kernel、调度器、负载均衡器和配置设置。
单个快速 kernel 并不能构成快速系统（阿姆达尔定律），也不一定能构成正确的系统。因此，我们同时优化整个栈，而非单独优化各个组件。
只有在保留正确模型行为的前提下，速度提升才有意义，而这一点的验证可能非常微妙。定制化改动可能悄然改变模型输出，但在 GSM8K 或 GPQA 等标准基准上仍能获得相同准确率。我们采用严格的分布匹配检查，而非功能测试或基准分数。
我们的 Agent 会随时间学习。每次实验都会产生可复用的洞见，Agent 利用这些洞见改进其跨模型和工作负载的发明流程。例如，我们的 Agent 学会了避免分布式死锁；这一经验最初来自 DeepSeek v4 Pro 的优化，随后应用到 GLM 5.2，避免了 44 分钟不必要的等待时间。
为什么推理系统优化如此困难？
Agent 必须考虑数百万行包含可调参数和算法实现的代码，以及无数服务器配置设置，因此搜索空间极其庞大。在恰当位置进行微小改动即可带来显著收益，但困难在于发现这些位置，以及确定优先级。
现代推理系统依赖许多相互作用的因素：
- **软件：**操作系统、kernel（如 PTX、CuTeDSL、CUDA C 和 Triton）、高级脚本层（如 PyTorch）、Python 胶水代码、运行时、调度、缓存、负载均衡、互连、客户端—服务器配置。
- **硬件：**GPU、CPU、内存、磁盘、节点内和节点间连接。
- **运行时动态：**输入分布、缓存行为、内存压力、限频、网络通信。
由于这些系统高度动态，优化需要持续评估和性能剖析；静态代码分析并不足够。但端到端评估缓慢且计算成本高：单次评估可能耗时数小时，涵盖编译、服务器启动、预热和请求处理。
随着 Agent 持续根据评估结果迭代，我们的评估系统使用端到端基准与微基准的分层分解，以保持较短的反馈闭环。
发明循环
我们的协同发明者通过并行运行多个 Agent，执行严格的“发现与构建 → 评估 → 验证 → 自我改进”循环。它们探索并实施优化策略，评估每次加速是否真实，验证其是否保留模型输出，并提炼所学知识用于下一轮运行。
发现与构建
我们的 Agent 通过自我改进机制，利用动态性能剖析、静态分析和既有经验来提出优化策略。它们通过权衡潜在加速收益与对数值准确性的影响来确定优化目标的优先级。这缩小了搜索空间，并有助于平衡探索与利用。
评估
我们的 Agent 从一个复现真实生产服务环境的沙箱中获得评估反馈。在那里，它们测试定制化的 vLLM 服务栈，将请求调度至自身优化的 kernel 和代码，以替代原有实现。
对于在复杂工作负载上运行、参数规模达数千亿至数万亿的模型，测量速度缓慢且天然存在噪声。批处理行为变化以及 GPU 功耗限频等硬件效应，都会为性能评估带来方差。
针对有噪声评估进行优化的 Agent，可能会报告并不真实的加速结果。我们识别并缓解测量噪声来源，以确保 Agent 在优化过程中获得干净的反馈。随后，我们应用统计分析和反作弊检查，确保测得的收益能够在生产环境中成立。
验证
只有在充分保留模型行为的情况下，加速才有价值。微小的输出误差可能在长序列中不断累积，从而降低质量。
我们采用基于模型输出分布匹配的严格正确性检查。这些检查涉及对优化模型与原始模型之间 token 级概率分布进行统计一致性测试。我们在来自 AllenAI Common Crawl C4 数据集的多样化多语言提示语料上进行评估。
验证频率同样重要：过于频繁会浪费计算资源，过于稀疏则会让 Agent 偏离轨道。经过验证的评估方法使快速且可信的迭代成为可能。这与我们的 Safe Rust 迁移工作背后的严谨理念一致。
自我改进
我们的 Agent 从实验中提炼洞见，并跨模型复用，持续更新自身的推理方式、策略和知识。这些洞见超越了基础事实检索：Agent 利用经验改变自身工作流和发现过程。
通过知识迁移实现自我改进
点击灯泡，查看我们各次运行中不同的自我改进示例
Agent 运行 1｜Agent 运行 2｜Agent 运行 3
- 反模式 · GLM 5.2 → GLM 5.2
- Kernel 探索 · GLM 5.2 → GLM 5.2
- 工作流路由 · DeepSeek v4 Pro → GLM 5.2
- 评估工具链接线 · DeepSeek v4 Pro → GLM 5.2
优化示例
展望未来
我们正在构建能够处理实际系统设计与优化工作的 Agent。这要求它们能够在较长时间跨度内可靠工作，并具备最高程度的严谨性。
我们的协同发明流程会持续累积，其驱动力来自以下相互强化的要素：
- 准确观察处于循环中的完整系统，使 Agent 能够推理系统级改进方案。
- 正确性使每一项性能收益都能够部署，并成为有意义洞见的来源。
- 自主性让 Agent 能够搜索人类无法独自探索的庞大空间。
- 学习将这些洞见传递至后续任务，因此每次运行都会为 Agent 带来更多可借鉴的经验。
可扩展验证仍是一个开放问题，推进这一能力是让自主优化在前沿规模上具备实用性的必要条件。在这一早期阶段，我们执行了严格的数值容差要求，这为将方法泛化到更广泛的正确性定义奠定了坚实基础。
我们的 Agent 能够在更通用的近似容差条件下进行优化。我们正积极将协同发明者应用于不同用例和部署模式。
如果您希望我们的 Agent 优化您的高性能系统，请通过 hello@asari.ai 联系我们。
如果您热衷于构建能够交付经验证系统代码的 Agent，欢迎加入我们！
English / 原文
Speeding up end-to-end inference with self-improving agents
Early results on how our co-inventor agents improve the full AI inference stack using a rigorous self-improving process.
July 28, 2026
By Asari AI
Introduction
We build co-inventors: self-improving agents that design and optimize high-performance infrastructure. Applied to AI inference, they optimized the entire inference stack for two of the largest open-source LLMs: DeepSeek v4 Pro and GLM 5.2 running on NVIDIA B200s with vLLM v0.23. They improved both throughput and interactivity by up to 16% across multiple max-concurrency levels, while preserving model behavior through distribution-level correctness checks. Each concurrency level took the agents about a day to optimize.
As of vLLM v0.26, most of the changes made by our agents have no upstream equivalent. The changes span the entire inference stack, including kernels, schedulers, load-balancers, and configuration settings.
A single fast kernel doesn't make a fast system (Amdahl's law), and it doesn't necessarily make a correct one. That's why we optimize the whole stack at once rather than individual components.
Speed only counts if correct model behavior is preserved, and that can be subtle to check. Customizations can quietly alter model outputs, yet still give the same accuracy on standard benchmarks like GSM8K or GPQA. We use stringent distribution-matching checks rather than functional tests or benchmark scores.
Our agents learn over time. Every experiment produces reusable insights that the agents use to improve their invention process across models and workloads. For example, our agents learned to avoid distributed deadlocks that would have cost 44 minutes in unnecessary wait time, generalizing a lesson learned from optimizing DeepSeek v4 Pro applied to GLM 5.2.
What makes optimizing inference systems hard?
Agents must consider millions of lines of code containing tunable parameters and algorithm implementations, plus countless server configuration settings, making the search space enormous. Small changes in the right places can produce large gains, but the hard part is finding them and knowing what to prioritize.
Modern inference systems depend on many interacting factors:
- Software: OS, kernels (e.g., PTX, CuTeDSL, CUDA C, and Triton), high-level scripting (e.g., PyTorch), Python glue code, runtime, scheduling, caching, load balancing, interconnect, client-server configurations.
- Hardware: GPU, CPU, memory, disk, intra- and inter-node connectivity.
- Runtime dynamics: input distributions, caching behavior, memory pressure, throttling, network communication.
Because these systems are highly dynamic, optimizing them requires continuous evaluation and profiling; static code analysis is not enough. But end-to-end evaluations are slow and computationally expensive: a single one can take hours, spanning compilation, server launch, warm-up, and request processing.
As the agents continually iterate on evaluation results, our evaluation system uses a hierarchical decomposition of end-to-end and micro benchmarks to keep feedback loops short.
The invention loop
Our co-inventors run a rigorous discover + build → evaluate → verify → self-improve loop using many agents in parallel. They explore and implement optimization strategies, evaluate whether each speedup is real, verify that it preserves the model's outputs, and distill what they learn for the next run.
Discover and build
Our agents propose optimization strategies using dynamic profiling, static analysis, and learned experiences via a self-improvement mechanism. They prioritize optimization targets by weighing potential speedups against the impact on numerical accuracy. This narrows the search space and helps balance exploration versus exploitation.
Evaluate
Our agents get feedback from evaluations in a sandbox that replicates the real production serving environment. There, they test customized vLLM serving stacks, dispatching to their own optimized kernels and code in place of the stock implementations.
For models with hundreds of billions to trillions of parameters running on complex workloads, measurement is slow and inherently noisy. Shifts in batching behavior and hardware effects like GPU power throttling introduce variance in performance evaluations.
An agent optimizing against noisy evaluations will report speedups that are not real. We identify and mitigate sources of measurement noise to ensure that the agent gets clean feedback while optimizing. We then apply statistical analysis and anti-cheating checks so that a measured gain holds up in production.
Verify
A speedup is only useful if it sufficiently preserves the model's behavior. Small output errors can compound over long sequences, degrading quality.
We use a stringent correctness check based on model output distribution matching. These checks involve statistical consistency tests on token-level probability distributions between the optimized model and the stock model. We evaluate this across a diverse multilingual corpus of prompts from the AllenAI Common Crawl C4 dataset.
Verification cadence matters as well: too frequent wastes compute, and too rare lets agents drift off track. A validated evaluation method makes fast, trustworthy iteration possible. This is the same thorough philosophy behind our Safe Rust migration work.
Self-improve
Our agents distill insights from their experiments and reuse them across models, updating their own reasoning, tactics, and knowledge over time. These insights go beyond basic fact-retrieval: the agents use their experience to change their own workflow and discovery process.
Self Improvement via Knowledge Transfer
Click a lightbulb to see different examples of self-improvement throughout our runs
Agent run 1Agent run 2Agent run 3
- Antipatterns· GLM 5.2 → GLM 5.2
- Kernel Exploration· GLM 5.2 → GLM 5.2
- Workflow Routing· DeepSeek v4 Pro → GLM 5.2
- Evaluation Harness Wiring· DeepSeek v4 Pro → GLM 5.2
Optimization examples
Looking ahead
We are building agents that work on practical system design and optimization. This requires them to work reliably over long time horizons, with the highest levels of rigor.
Our co-invention process compounds, driven by these reinforcing elements:
- Accurate observations about the full system in the loop enable agents to reason about system-level improvements.
- Correctness makes each performance gain shippable and a source of meaningful insights.
- Autonomy lets the agents search a space too large for humans to explore alone.
- Learning carries these insights forward, so each run gives the agents more experience to draw from.
Scalable verification remains an open problem, and advancing it is necessary to make autonomous optimization practical at frontier scale. We enforced a tight numerical tolerance in this early phase, which gives us a solid foundation to generalize to broader notions of correctness.
Our agents can optimize under more general approximation tolerances. We are actively applying our co-inventors across different use cases and deployment patterns.
If you want our agents to optimize your high-performance systems, contact us at hello@asari.ai.
If you're passionate about building agents that ship verified systems code, join us!