Diffusion Controller 如何统一并简化 AI 图像生成控制

Nano Banana、Stable Diffusion 和 Flux 等文本生成图像 AI 模型的快速发展，已从根本上改变了创意设计：任何人都可以通过文字描述合成照片级真实、高保真的图像。然而，要引导这些庞大的模型精准满足用户意图、下游目标或严格的视觉约束，仍是一场微妙且难以预测的平衡。例如，设想你要求模型生成“一只戴着太阳镜的蜥蜴”。模型可能会生成一只逼真的蜥蜴，却没有戴太阳镜；反之，如果强制模型加入太阳镜，又可能扭曲蜥蜴的脸部，破坏图像质量。

现有的图像生成引导或微调方法彼此之间高度割裂。一方面，开发者使用推理时技术（例如无分类器扩散引导）来调整文本提示词的影响力，并在生成过程中实时引导图像。另一方面，他们依赖高成本的微调，利用 LoRA 等参数高效适配器、奖励加权回归或策略梯度，改变模型行为。

由于这些工具在历史上一直被视为彼此独立、互不相关的补救方案，该领域始终缺少一种统一且有原则的数学语言，来整合、分析和优化我们控制生成模型的方式。这种碎片化的方法往往迫使工程师在平衡用户偏好对齐与图像质量时依赖经验猜测。

为解决这一权衡问题，我们提出了 Diffusion Controller 框架。Diffusion Controller 不再将图像生成视为一系列僵化、孤立的步骤，而是将完整的去噪过程重新定义为一个平滑、连续的控制问题。我们的结果显示，Diffusion Controller 的轻量级附加网络在匹配人类偏好方面优于行业标准方法。此外，其完全解锁版本（即能够以“白盒”方式、不受限制地访问并修改模型内部权重的微调模型）相较基线模型取得了 90% 的胜率。

The rapid advancement of text-to-image AI models, such as Nano Banana, Stable Diffusion and Flux, has fundamentally transformed creative design, allowing anyone to synthesize photorealistic, high-fidelity images from textual descriptions. However, steering these massive models to meet precise user intent, downstream goals, or strict visual constraints remains a delicate and unpredictable balancing act. For example, imagine prompting a model for "a lizard wearing sunglasses". The model might generate a realistic lizard that's not wearing sunglasses. Alternatively, forcing the model to include the sunglasses might distort the lizard's face, ruining the image quality.

Existing methodologies that guide or fine-tune image generation are very disconnected. On the one hand, developers use inference-time techniques (e.g., classifier-free diffusion guidance) to adjust the text prompt’s influence and guide the image generation process on the fly. On the other hand, they rely on heavy fine-tuning using parameter-efficient adapters like LoRA, reward-weighted regressions, or policy gradients to alter a model's behavior.

Because these tools have historically been treated as distinct and unrelated fixes, the field has lacked a single, principled mathematical language to unify, analyze, and optimize how we control generative models. This fragmented approach often forces engineers to rely on guesswork when balancing user preference alignment against image quality.

To solve this balancing act, we present the Diffusion Controller framework. Instead of treating image generation as a rigid sequence of isolated steps, Diffusion Controller reframes the entire denoising process as a smooth, continuous control problem. Our results show that Diffusion Controller’s lightweight add-on network outperformed the industry standard for matching human preferences. Moreover, its fully unlocked version (i.e., the fine-tuned model with "white-box" or unrestricted access to alter internal model weights) achieved a 90% win rate over the baseline model.