可信 Agentic AI:自主 LLM 系统的失效模式、缓解策略与生命周期框架
英文标题:Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems
这篇论文系统梳理了基于大语言模型的 Agentic AI 在多步骤规划、工具调用、记忆持久化和多智能体协作中面临的安全与运营风险。文章将间接提示注入、后门触发、目标泛化失误、记忆污染和跨会话数据泄露等问题归纳为统一分类体系,并评估了指令层级、上下文隔离、spotlighting、基于过程的监督、受限工具使用和隐私保护记忆等缓解手段。作者提出了包含规范、设计、训练、评估、部署和监控六个阶段的可信 Agent 开发生命周期(TADL),为每个阶段定义信任活动、预期证据与基于风险的决策关卡。该框架尚未经过实证验证,但可为开发者构建和评估更安全、更具问责性的自主 LLM Agent 提供结构化参考。
612 B
380 B
9.1 KB
32.7 KB
22.8 KB
24.2 KB
34.7 KB
中文
[提交于 2026 年 9 月 19 日]
标题:可信 Agentic AI:自主 LLM 系统的失效模式、缓解策略与生命周期框架
查看由 Fayeq Jeelani Syed 及另外 3 位作者撰写的论文《可信 Agentic AI:自主 LLM 系统的失效模式、缓解策略与生命周期框架》PDF。
摘要:基于大语言模型构建的 Agentic AI 系统能够进行多步骤规划、使用外部工具、在记忆中保留信息,并与其他 Agent 协作。这些能力使其比静态语言模型更有用,但也引入了新的安全与运营风险。来自网站、电子邮件、文档和数据库的不可信内容,可能与系统指令一同进入相同上下文;持久记忆可能在不同会话之间携带已被破坏的信息;而对外部工具的访问,则可能将错误的模型回答转化为具有实际后果的现实行动。本文从五个相互关联的维度审视 Agentic AI 的可信性:安全性与鲁棒性、对齐与人工监督、透明度与可审计性、隐私与数据治理,以及监管合规。文章将包括间接提示注入、后门触发、目标泛化失误、记忆污染和跨会话数据泄露在内的关键失效模式,组织为统一的分类体系。文章还考察了主要缓解方法,例如指令层级、上下文隔离、spotlighting、基于过程的监督、受限工具使用和隐私保护记忆,并区分了具有实证证据支持的技术与主要仍停留在概念层面的技术。在此分析基础上,我们提出可信 Agent 开发生命周期(Trustworthy Agent Development Lifecycle,TADL),这是一个涵盖规范、设计、训练、评估、部署和监控六个阶段的框架。对于每个阶段,TADL 都明确了相关的可信活动、预期证据以及基于风险的决策关卡。尽管 TADL 尚未经过实证验证,但它为开发和评估更安全、更具问责性的 Agentic 系统提供了结构化基础。文章最后指出了当前基准测试中的缺口,并概述了未来研究的优先事项。
提交历史
来自:Fayeq Jeelani Syed [
]
[v1]
2026 年 9 月 19 日,星期六,02:47:37 UTC(59 KB)
English / 原文
[Submitted on 19 Sep 2026]
Title:Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems
View a PDF of the paper titled Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems, by Fayeq Jeelani Syed and 3 other authors
Abstract:Agentic AI systems built on large language models can plan over multiple steps, use external tools, retain information in memory, and coordinate with other agents. These capabilities make them more useful than static language models, but they also introduce new security and operational risks. Untrusted content from websites, emails, documents, and databases can enter the same context as system instructions; persistent memory can carry compromised information across sessions; and access to external tools can turn an incorrect model response into a consequential real-world action. This article reviews the trustworthiness of agentic AI across five interconnected dimensions: safety and robustness, alignment and human oversight, transparency and auditability, privacy and data governance, and regulatory compliance. It organizes key failure modes, including indirect prompt injection, backdoor triggers, goal misgeneralization, memory contamination, and cross-session data leakage, into a unified taxonomy. It also examines major mitigation approaches, such as instruction hierarchies, context isolation, spotlighting, process-based supervision, constrained tool use, and privacy-preserving memory, while distinguishing techniques supported by empirical evidence from those that remain largely conceptual. Building on this analysis, we introduce the Trustworthy Agent Development Lifecycle (TADL), a six-phase framework covering specification, design, training, evaluation, deployment, and monitoring. For each phase, TADL identifies relevant trust activities, expected evidence, and risk-based decision gates. Although TADL has not yet been empirically validated, it provides a structured foundation for developing and evaluating more secure and accountable agentic systems. The article concludes by identifying gaps in current benchmarks and outlining priorities for future research.
Submission history
From: Fayeq Jeelani Syed [
]
[v1]
Sat, 19 Sep 2026 02:47:37 UTC (59 KB)