面向 Apple Silicon 的并行约束解码
英文标题：🤗 harshatheg/Qwen-2.5-1B-RLCD
该项目基于 MLX 为 Apple Silicon 提供高吞吐量的结构化信息抽取、决策路由与分类推理引擎。其“并行约束解码”可同时评估多字段 JSON Schema，而非按 token 自回归逐个生成；在 M4 Max 上，相比标准自回归解码实现了 5.6 倍至 7.0 倍的延迟降低。项目宣称输出可保证 100% 符合 Schema 语法，并为每个字段提供经过校准的置信度分数。对开发者而言，这种方式尤其适用于枚举、布尔值等候选集合有限的企业工作流，并支持 SDK、命令行基准测试和可视化 Web 界面。
打开原文相关 1.6 · 机会 2
4.5 KB
10.0 KB
12.3 KB
408.1 KB
中文
面向 Apple Silicon 的并行约束解码
在线演示：可在 Hugging Face Spaces 中试用实时的并排对比：drinkmoonshine/parallel-constrained-decoding。
这是一个基于 MLX、面向 Apple Silicon 的高吞吐量推理引擎，用于结构化信息抽取、决策路由和类别分类。
并行约束解码会同时评估多字段 JSON Schema，而不是按顺序生成 token。在 Apple Silicon M4 Max 上，相比标准自回归解码，它可实现 5.6 倍至 7.0 倍的延迟降低，同时提供 100% 的 Schema 有效性和经过校准的字段级置信度分数。
性能基准测试（Apple Silicon M4 Max）
在 macOS Sequoia 上使用 mlx-community/Qwen2.5-1.5B-Instruct-4bit 进行评估：
| 场景 | 字段数 | 自回归基线 | 并行约束解码 | 延迟加速比 | 语法有效性 |
|---|---|---|---|---|---|
| 金融科技欺诈路由 | 4 个字段 | 420 ms（120 tok/s） | 75 ms | 5.6 倍 | 100% 保证 |
| 代码安全审计 | 4 个字段 | 380 ms（125 tok/s） | 68 ms | 5.6 倍 | 100% 保证 |
| 高基数关税分类 | 1 个字段（255 个选项） | 500 ms（118 tok/s） | 89 ms | 5.6 倍 | 100% 保证 |
| 企业支持分流 | 28 个字段 | 1,900 ms（130 tok/s） | 270 ms | 7.0 倍 | 100% 保证 |
为什么采用并行约束解码？
自回归结构化生成的问题
标准 LLM 结构化生成（例如 JSON 模式或语法引导采样）依赖逐 token 的自回归解码：
[Context Prompt] -> "{" -> "\n" -> " " -> "risk" -> ":" -> " " -> "HIGH" -> ...
(Requires 150 to 500 sequential forward passes)
每个 token 都需要一次独立的 GPU/NPU 前向推理，以及顺序执行的内存带宽往返操作。随着 Schema 规模增长，延迟会随输出 token 长度线性增长：
Tautoregressive=∑k=1Ktstep(k)T_{\text{autoregressive}} = \sum_{k=1}^{K} t_{\text{step}}(k)Tautoregressive=k=1∑Ktstep(k)
此外，自回归解码容易出现语法退化、字段遗漏和幻觉键（hallucinated keys）问题。
解决方案：通过 KV-Cache 广播进行并行评估
在结构化抽取和分类任务中，字段值属于有界候选集合（布尔值或类别枚举）。并行约束解码利用了这一特性：
+---> [Field 1: "risk_level"] -------> Logit Slicing -> Top Choice
|
[Context Prefix Prefill] -+---> [Field 2: "requires_review"] ---> Logit Slicing -> Top Choice
(Single KV-Cache State) |
+---> [Field M: "action_tier"] ------> Logit Slicing -> Top Choice
(All fields evaluated simultaneously)
- 单次广播预填充：上下文文档和语义 Schema 描述仅预填充一次，写入 MLX 键值（KV）缓存。
- KV-Cache 广播：将 KV-cache 并行广播到全部 $M$ 个 Schema 字段。
- 子词表 Logit 切片：对于每个字段，仅评估属于有效 Schema 选项的候选 token ID，其余词表会被掩码。
- 经校准的 Softmax 概率：在候选切片上计算精确的归一化概率：$$P(c_i) = \frac{\exp(z_i / T)}{\sum_{j=1}^{C} \exp(z_j / T)}$$
- Token 树消歧：当候选选项共享多 token 前缀根节点时，引擎使用切片后的缓存状态执行续写步骤，且无需重新分配内存。
- 程序化组装：直接使用已验证的值构造输出 JSON，从而保证 100% 的有效语法，避免 JSON 解析错误。
安装
前置要求
- Apple Silicon Mac（M1、M2、M3、M4 系列）
- macOS 14.0 或更高版本
- Python 3.10+
配置
克隆仓库并安装依赖：
git clone https://github.com/your-org/parallel-constrained-decoding.git
cd parallel-constrained-decoding
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
Developer SDK 快速上手
1. 定义 Schema
使用 StructuredSchema 定义 Schema。每个字段指定 type（enum 或 boolean）、用于引导模型推理的 description，以及 choices（适用于 enum 类型，最多支持 255 个选项）：
from core.schema import StructuredSchema, FieldDefinition
schema_dict = {
"priority": {
"type": "enum",
"choices": ["P0_CRITICAL", "P1_HIGH", "P2_NORMAL", "P3_LOW"],
"description": "Urgency tier based on customer business impact"
},
"requires_escalation": {
"type": "boolean",
"description": "Whether an on-call engineer must be notified immediately"
},
"department": {
"type": "enum",
"choices": ["BILLING", "INFRASTRUCTURE", "SECURITY", "PRODUCT_SUPPORT"],
"description": "Target handling department"
}
}
schema = StructuredSchema(schema_dict)
也可以通过 FieldDefinition 显式构造字段：
fields = {
"tariff_classification": FieldDefinition(
name="tariff_classification",
field_type="enum",
description="Harmonized System 6-digit tariff category code",
choices=["0101.21", "0101.29", "8471.30", "8517.12", "8542.31", ...]
)
}
2. 运行并行生成
对上下文字符串执行并行约束推理：
from core.engine import run_parallel_generation
context = """
Incident Report: Production database db-primary-01 CPU at 100%.
Payment gateway failing for 40% of checkout requests.
Tier 1 Enterprise customer affected: Acme Global.
"""
result = run_parallel_generation(context, schema)
print(f"Latency: {result['elapsed_ms']} ms")
print(f"Prefill Time: {result['prefill_ms']} ms")
print(f"Passes: {result['sequential_forward_passes']}")
print("\nExtracted JSON:")
print(result["parsed_json"])
3. 响应结构
输出字典同时提供结构化 JSON 和详细的字段遥测数据：
{
"mode": "parallel_constrained_calibrated",
"elapsed_ms": 74.5,
"prefill_ms": 52.1,
"suffix_eval_ms": 18.2,
"sequential_forward_passes": 1,
"is_valid_json": True,
"schema_match": True,
"parsed_json": {
"priority": { "value": "P0_CRITICAL", "prob": 0.9924 },
"requires_escalation": { "value": "true", "prob": 0.9981 },
"department": { "value": "INFRASTRUCTURE", "prob": 0.9815 }
},
"field_telemetry": {
"priority": {
"value": "P0_CRITICAL",
"confidence": 0.9924,
"cardinality": 4,
"top_choices": [
{ "choice": "P0_CRITICAL", "probability": 0.9924 },
{ "choice": "P1_HIGH", "probability": 0.0068 },
{ "choice": "P2_NORMAL", "probability": 0.0006 },
{ "choice": "P3_LOW", "probability": 0.0002 }
]
}
}
}
4. 流式自回归基线
如需与标准自回归生成进行比较：
from core.engine import stream_naive_generation
for event in stream_naive_generation(context, schema):
if event["type"] == "token":
print(event["token"], end="", flush=True)
elif event["type"] == "done":
print(f"\nCompleted in {event['result']['elapsed_ms']} ms")
交互式 Web 可视化工具
该仓库包含一个 Web 界面，用于并排比较延迟和准确性。
启动 Web 服务器：
bash run.sh
或者直接使用 uvicorn 运行：
python3 -m uvicorn server.app:app --host 0.0.0.0 --port 8000
在浏览器中打开 http://localhost:8000。
功能
- 并排对比：并行约束解码与自回归流式生成的对比。
- 实时毫秒计时器：实时显示已耗费的延迟时间。
- 同步滚动：两个面板中的对应键可对齐显示。
- 交互式行高亮：将鼠标悬停在任一面板中的字段上，可高亮另一面板内的相应键。
- 幻觉检测：高亮朴素自回归输出中遗漏或产生幻觉的键。
命令行基准测试运行器
针对预配置的企业预设运行基准测试套件：
python3 -m core.benchmark
输出示例：
======================================================================
Parallel Constrained vs. Autoregressive Generation Benchmark
======================================================================
--> Running preset: Fintech Fraud Detection (4 fields)...
Autoregressive Baseline : 421.3 ms | 148 tokens (122.4 tok/s) | Passes: 148
Parallel Constrained : 74.8 ms | 0 tokens (O(1)) | Passes: 1
>> SPEEDUP: 5.6x faster (Step reduction: 148.0x)
>> Schema match: Naive=True | Parallel=True (100% guaranteed)
----------------------------------------------------------------------
--> Running preset: Support Triage Matrix (28 fields)...
Autoregressive Baseline : 1894.2 ms | 312 tokens (131.2 tok/s) | Passes: 312
Parallel Constrained : 268.4 ms | 0 tokens (O(1)) | Passes: 1
>> SPEEDUP: 7.1x faster (Step reduction: 312.0x)
>> Schema match: Naive=True | Parallel=True (100% guaranteed)
----------------------------------------------------------------------
--> Running preset: High-Cardinality Tariff (1 field, 255 choices)...
Autoregressive Baseline : 498.7 ms | 42 tokens (116.5 tok/s) | Passes: 42
Parallel Constrained : 88.6 ms | 0 tokens (O(1)) | Passes: 1
>> SPEEDUP: 5.6x faster (Step reduction: 42.0x)
>> Schema match: Naive=True | Parallel=True (100% guaranteed)
----------------------------------------------------------------------
仓库结构
.
├── core/
│ ├── __init__.py # SDK package exports
│ ├── engine.py # Parallel constrained decoding & autoregressive engines
│ ├── schema.py # Schema definitions, metadata compiler & logit mapping
│ ├── prompt_builder.py # Prompt templates for prefill catalog and naive baseline
│ └── benchmark.py # Command-line benchmark runner
├── presets/
│ ├── fintech_fraud.json # Fraud detection scenario (4 fields)
│ ├── code_security.json # Vulnerability audit scenario (4 fields)
│ ├── support_triage.json # Enterprise ticket triage (28 fields)
│ └── high_cardinality_255.json # 255-choice tariff classifier
├── server/
│ ├── app.py # FastAPI endpoints (/api/run-parallel, /api/stream-naive)
│ └── main.py # Server launcher
├── web/
│ ├── index.html # Side-by-side comparison UI
│ ├── app.js # Frontend streaming & synchronized scrolling
│ └── style.css # UI styling
├── MODEL_CARD.md # Hugging Face model card documentation
├── requirements.txt # Python package requirements
├── run.sh # Startup script
└── README.md # Project documentation
支持的模型
该引擎目前配置为使用 mlx-community/Qwen2.5-1.5B-Instruct-4bit。
通过在 core/engine.py 中设置 MODEL_ID，可加载 mlx-lm 支持的任意 decoder LLM。
许可证
Apache 2.0
English / 原文
Parallel Constrained Decoding for Apple Silicon
Live Demo : Try the side-by-side comparison live on Hugging Face Spaces: drinkmoonshine/parallel-constrained-decoding.
A high-throughput inference engine for structured information extraction, decision routing, and categorical classification on Apple Silicon using MLX.
Parallel Constrained Decoding evaluates multi-field JSON schemas simultaneously rather than generating tokens sequentially. On an Apple Silicon M4 Max, it delivers 5.6x to 7.0x latency reductions compared to standard autoregressive decoding with 100% schema validity and calibrated field-level confidence scores.
Performance Benchmarks (Apple Silicon M4 Max)
Evaluated with mlx-community/Qwen2.5-1.5B-Instruct-4bit on macOS Sequoia:
| Scenario | Fields | Autoregressive Baseline | Parallel Constrained | Latency Speedup | Syntax Validity |
|---|---|---|---|---|---|
| Fintech Fraud Routing | 4 fields | 420 ms (120 tok/s) | 75 ms | 5.6x | 100% guaranteed |
| Code Security Audit | 4 fields | 380 ms (125 tok/s) | 68 ms | 5.6x | 100% guaranteed |
| High-Cardinality Tariff | 1 field (255 choices) | 500 ms (118 tok/s) | 89 ms | 5.6x | 100% guaranteed |
| Enterprise Support Triage | 28 fields | 1,900 ms (130 tok/s) | 270 ms | 7.0x | 100% guaranteed |
Why Parallel Constrained Decoding?
The Problem with Autoregressive Structured Generation
Standard LLM structured generation (such as JSON mode or grammar-guided sampling) relies on token-by-token autoregressive decoding:
[Context Prompt] -> "{" -> "\n" -> " " -> "risk" -> ":" -> " " -> "HIGH" -> ...
(Requires 150 to 500 sequential forward passes)
Each token requires a distinct GPU/NPU forward pass and sequential memory bandwidth roundtrips. As schema size grows, latency scales linearly with output token length:
Tautoregressive=∑k=1Ktstep(k)T_{\text{autoregressive}} = \sum_{k=1}^{K} t_{\text{step}}(k)Tautoregressive=k=1∑Ktstep(k)
Additionally, autoregressive decoding is susceptible to syntax degradation, field omission, and hallucinated keys.
The Solution: Parallel Evaluation via KV-Cache Broadcasting
In structured extraction and classification, field values belong to bounded candidate sets (booleans or categorical enums). Parallel Constrained Decoding exploits this property:
+---> [Field 1: "risk_level"] -------> Logit Slicing -> Top Choice
|
[Context Prefix Prefill] -+---> [Field 2: "requires_review"] ---> Logit Slicing -> Top Choice
(Single KV-Cache State) |
+---> [Field M: "action_tier"] ------> Logit Slicing -> Top Choice
(All fields evaluated simultaneously)
- Single Broadcast Prefill : The context document and semantic schema descriptions are prefilled once into an MLX Key-Value (KV) cache.
- KV-Cache Broadcasting : The KV-cache is broadcast across all $M$ schema fields in parallel.
- Sub-Vocabulary Logit Slicing : For each field, only candidate token IDs belonging to valid schema choices are evaluated. The remaining vocabulary is masked.
- Calibrated Softmax Probabilities : Exact normalized probabilities are calculated over the candidate slice: $$P(c_i) = \frac{\exp(z_i / T)}{\sum_{j=1}^{C} \exp(z_j / T)}$$
- Token Tree Disambiguation : When candidate choices share multi-token prefix roots, the engine executes continuation steps using sliced cache states with zero memory reallocation.
- Programmatic Assembly : Output JSON is constructed directly from verified values, guaranteeing 100% valid syntax without JSON parsing errors.
Installation
Prerequisites
- Apple Silicon Mac (M1, M2, M3, M4 series)
- macOS 14.0 or later
- Python 3.10+
Setup
Clone the repository and install dependencies:
git clone https://github.com/your-org/parallel-constrained-decoding.git
cd parallel-constrained-decoding
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
Developer SDK Quickstart
1. Defining Schemas
Schemas are defined using StructuredSchema. Each field specifies a type (enum or boolean), a description to guide model reasoning, and choices (for enum types, supporting up to 255 choices):
from core.schema import StructuredSchema, FieldDefinition
schema_dict = {
"priority": {
"type": "enum",
"choices": ["P0_CRITICAL", "P1_HIGH", "P2_NORMAL", "P3_LOW"],
"description": "Urgency tier based on customer business impact"
},
"requires_escalation": {
"type": "boolean",
"description": "Whether an on-call engineer must be notified immediately"
},
"department": {
"type": "enum",
"choices": ["BILLING", "INFRASTRUCTURE", "SECURITY", "PRODUCT_SUPPORT"],
"description": "Target handling department"
}
}
schema = StructuredSchema(schema_dict)
You can also construct fields explicitly using FieldDefinition:
fields = {
"tariff_classification": FieldDefinition(
name="tariff_classification",
field_type="enum",
description="Harmonized System 6-digit tariff category code",
choices=["0101.21", "0101.29", "8471.30", "8517.12", "8542.31", ...]
)
}
2. Running Parallel Generation
Execute parallel constrained inference on your context string:
from core.engine import run_parallel_generation
context = """
Incident Report: Production database db-primary-01 CPU at 100%.
Payment gateway failing for 40% of checkout requests.
Tier 1 Enterprise customer affected: Acme Global.
"""
result = run_parallel_generation(context, schema)
print(f"Latency: {result['elapsed_ms']} ms")
print(f"Prefill Time: {result['prefill_ms']} ms")
print(f"Passes: {result['sequential_forward_passes']}")
print("\nExtracted JSON:")
print(result["parsed_json"])
3. Response Structure
The output dictionary provides both the structured JSON and detailed field telemetry:
{
"mode": "parallel_constrained_calibrated",
"elapsed_ms": 74.5,
"prefill_ms": 52.1,
"suffix_eval_ms": 18.2,
"sequential_forward_passes": 1,
"is_valid_json": True,
"schema_match": True,
"parsed_json": {
"priority": { "value": "P0_CRITICAL", "prob": 0.9924 },
"requires_escalation": { "value": "true", "prob": 0.9981 },
"department": { "value": "INFRASTRUCTURE", "prob": 0.9815 }
},
"field_telemetry": {
"priority": {
"value": "P0_CRITICAL",
"confidence": 0.9924,
"cardinality": 4,
"top_choices": [
{ "choice": "P0_CRITICAL", "probability": 0.9924 },
{ "choice": "P1_HIGH", "probability": 0.0068 },
{ "choice": "P2_NORMAL", "probability": 0.0006 },
{ "choice": "P3_LOW", "probability": 0.0002 }
]
}
}
}
4. Streaming Autoregressive Baseline
To compare against standard autoregressive generation:
from core.engine import stream_naive_generation
for event in stream_naive_generation(context, schema):
if event["type"] == "token":
print(event["token"], end="", flush=True)
elif event["type"] == "done":
print(f"\nCompleted in {event['result']['elapsed_ms']} ms")
Interactive Web Visualizer
The repository includes a web interface for side-by-side latency and accuracy comparison.
To launch the web server:
bash run.sh
Or run directly with uvicorn:
python3 -m uvicorn server.app:app --host 0.0.0.0 --port 8000
Open http://localhost:8000 in your browser.
Features
- Side-by-Side Comparison : Parallel Constrained Decoding vs. Autoregressive Streaming.
- Live Millisecond Timers : Real-time elapsed latency counters.
- Synchronized Scrolling : Matching keys align across both panes.
- Interactive Row Highlighting : Hover over any field in either panel to highlight the corresponding key in the other.
- Hallucination Detection : Highlights omitted or hallucinated keys in naive autoregressive output.
Command-Line Benchmark Runner
Run the benchmark suite across pre-configured enterprise presets:
python3 -m core.benchmark
Output example:
======================================================================
Parallel Constrained vs. Autoregressive Generation Benchmark
======================================================================
--> Running preset: Fintech Fraud Detection (4 fields)...
Autoregressive Baseline : 421.3 ms | 148 tokens (122.4 tok/s) | Passes: 148
Parallel Constrained : 74.8 ms | 0 tokens (O(1)) | Passes: 1
>> SPEEDUP: 5.6x faster (Step reduction: 148.0x)
>> Schema match: Naive=True | Parallel=True (100% guaranteed)
----------------------------------------------------------------------
--> Running preset: Support Triage Matrix (28 fields)...
Autoregressive Baseline : 1894.2 ms | 312 tokens (131.2 tok/s) | Passes: 312
Parallel Constrained : 268.4 ms | 0 tokens (O(1)) | Passes: 1
>> SPEEDUP: 7.1x faster (Step reduction: 312.0x)
>> Schema match: Naive=True | Parallel=True (100% guaranteed)
----------------------------------------------------------------------
--> Running preset: High-Cardinality Tariff (1 field, 255 choices)...
Autoregressive Baseline : 498.7 ms | 42 tokens (116.5 tok/s) | Passes: 42
Parallel Constrained : 88.6 ms | 0 tokens (O(1)) | Passes: 1
>> SPEEDUP: 5.6x faster (Step reduction: 42.0x)
>> Schema match: Naive=True | Parallel=True (100% guaranteed)
----------------------------------------------------------------------
Repository Structure
.
├── core/
│ ├── __init__.py # SDK package exports
│ ├── engine.py # Parallel constrained decoding & autoregressive engines
│ ├── schema.py # Schema definitions, metadata compiler & logit mapping
│ ├── prompt_builder.py # Prompt templates for prefill catalog and naive baseline
│ └── benchmark.py # Command-line benchmark runner
├── presets/
│ ├── fintech_fraud.json # Fraud detection scenario (4 fields)
│ ├── code_security.json # Vulnerability audit scenario (4 fields)
│ ├── support_triage.json # Enterprise ticket triage (28 fields)
│ └── high_cardinality_255.json # 255-choice tariff classifier
├── server/
│ ├── app.py # FastAPI endpoints (/api/run-parallel, /api/stream-naive)
│ └── main.py # Server launcher
├── web/
│ ├── index.html # Side-by-side comparison UI
│ ├── app.js # Frontend streaming & synchronized scrolling
│ └── style.css # UI styling
├── MODEL_CARD.md # Hugging Face model card documentation
├── requirements.txt # Python package requirements
├── run.sh # Startup script
└── README.md # Project documentation
Supported Models
The engine is currently configured for mlx-community/Qwen2.5-1.5B-Instruct-4bit.
Any decoder LLM supported by mlx-lm can be loaded by setting MODEL_ID in core/engine.py.
License
Apache 2.0