摘要：对于本地大语言模型 Agent 而言，活跃上下文是一项稀缺资源：内存容量、预填充（prefill）延迟、缓存增长以及服务目标，共同限制了每次模型调用所能承受的输入 token 数量。我们提出了 BudgetBench，这是一个活跃预算（active-budget）协议与参考测试框架（reference harness），在对比不同内存策略（memory strategy）时，将单次调用的输入 token 预算作为自变量。在固定模型、任务、采样器和解码配置的前提下，该框架在 2K、4K、8K、16K 和 32K token 的预算范围内进行扫描测试，并记录质量、预算利用率、延迟，以及作为核心一等输出指标的预算违规率（budget-violation rates）。该工作的核心贡献在于这套可复用的测量体系：包含可插拔的 MemoryStrategy 契约、显式预算执行、确定性或带版本控制的评分器（graders）、Prompt 审计元数据以及可复现工件，开源发布于 此 GitHub 链接。
我们通过先导性试验研究（pilot studies）而非最终排行榜来论证该协议的有效性。在包含本地 qwen2.5:1.5b 先导测试（在 SWE-bench Verified 和 LongBench v2 上各 89 项）、具有精确分词（exact tokenization）的托管 50 项 Qwen3 30B-A3B LongBench 复现，以及由官方 GPT-4o 评估器打分的 500 项 LongMemEval Oracle 研究中，该测试框架揭示了预算合规失败、非单调质量曲线，以及单一预算评估所隐藏的运行特征点。带预算策略与全上下文（full-context）策略的优劣对比尚未定论：本地测试切片的结果接近零差异（near-null），而托管复现的点估计则倾向于全上下文。我们透明地报告了相关结果，包括早期先导测试的分词器近似算法低估了部分已服务模型的 Prompt，因此其违规记录行属于分词器近似诊断，而非支持结论的断言性结果；所有耗时均作为运行诊断数据。这项可复用的贡献在于规模化扩展固定预算内存策略评估所必需的协议、测试套件以及失败报告规范。

Abstract:For local large language model agents, active context is a scarce resource: memory capacity, prefill latency, cache growth, and service objectives all constrain how many input tokens each call can afford. We present BudgetBench, an active-budget protocol and reference harness that treats the per-call input-token budget as the independent variable when comparing memory strategies. Holding the model, task, sampler, and decoding fixed, it sweeps budgets over 2K, 4K, 8K, 16K, and 32K tokens and records quality, budget utilization, latency, and, as a first-class outcome, budget-violation rates. The core contribution is this reusable measurement surface: a swappable MemoryStrategy contract, explicit budget enforcement, deterministic or versioned graders, prompt-audit metadata, and reproducibility artifacts, released at this https URL.
We substantiate the protocol with pilot studies rather than final rankings. Across a local qwen2.5:1.5b pilot (89 items each on SWE-bench Verified and LongBench v2), a hosted 50-item Qwen3 30B-A3B LongBench replication with exact tokenization, and a 500-item LongMemEval oracle study scored by the official GPT-4o evaluator, the harness exposes budget-compliance failures, non-monotonic quality curves, and operating points that single-budget evaluation hides. The budgeted-versus-full-context direction remains unresolved: the local slice is near-null and the hosted replication favors full context in point estimate. We report results transparently, including that the early pilot's tokenizer approximation undercounts some served-model prompts, so its violation rows are tokenizer-approximation diagnostics, not claim-bearing results; all timings are operational diagnostics. The reusable contribution is the protocol, harness, and failure-reporting discipline needed to scale fixed-budget memory-strategy evaluation.