Orca Security 借助 Amazon Bedrock 实现自然语言安全查询

安全分析师需要快速获得有关云基础设施的答案，却不应被要求深入掌握特定领域的查询语言。Orca Security 的工具覆盖超过 900 种云模型（即 Sonar 跟踪的资源类型，例如 Amazon Elastic Compute Cloud（Amazon EC2）实例或 Amazon Simple Storage Service（Amazon S3）存储桶）。查询这些数据过去需要准确的模型名称、嵌套关系导航，以及从 18 种运算符中进行选择（例如用于筛选结果的 equals 或 contains 等比较函数）。

为解决这一问题，Orca Security 与 AWS 合作构建了 Discovery AI——一款基于 Amazon Bedrock 的自然语言查询 Agent。Orca 为云、代码和 AI 提供具备上下文感知能力的安全防护；Sonar 则是其领域专用查询系统，用户可借此在整个数据图谱中进行搜索、筛选和探索。

本文将介绍 Discovery AI 的架构和成果。该系统可将自然语言问题转换为 Sonar 查询。与初始基线相比，Orca 将查询成本降低了 90%，输入 Token 数量降低了 93%，延迟降低了 75%。

弥合自然语言与结构化查询之间的鸿沟

Orca Security 的 Sonar 查询系统支持 900 多种云基础设施和应用模型，这些模型之间存在复杂关系。要查询这些数据，用户需要构造 JSON 对象，指定准确的模型名称，从 18 种运算符中选择合适的操作，并在不同云服务商之间导航父子层级关系。要熟练完成这些操作，需要深入了解 Sonar 的 Schema。

为构建这一能力，Orca 团队着手开发一个自然语言界面，使其能够从纯文本生成有效的 Sonar 查询。

其最初的概念验证方案是在单个 Prompt 中加载完整的 Sonar Schema，并使用当时可用的最大模型。这包括超过 900 个模型定义、所有关系类型，以及大量示例查询。输出质量较好——设计合作伙伴和内部用户均给出了积极反馈——但在生产规模下，该方案暴露出三项重大缺点。

其 Token 消耗效率较低，每个工作负载约需 200 万输入 Token，导致成本攀升。延迟也很高，每个工作负载的端到端响应时间约为 60 秒。对于复杂的多模型查询，准确性有限，因为嵌套关系和多条件查询通常需要额外优化。

Orca 团队验证了这一概念确实可行。接下来的挑战是让它在生产环境中具备足够的成本效益、速度和准确性。为此，Orca 团队设计并部署了一套基于 Amazon Bedrock 的解决方案，可在 15 秒响应时间内处理每天数千次查询。

解决方案概览

Orca 将 Discovery AI 设计为一套构建在 Amazon Bedrock 上的混合解决方案，结合了检索增强生成（Retrieval Augmented Generation，RAG）与具备 Schema 感知能力的查询构建。该系统通过多阶段流水线，将自然语言问题转换为有效的 Discovery 筛选器；每个阶段均匹配最合适的 Amazon Bedrock 模型。

Orca 选择 Amazon Bedrock，是因为它通过单一 API 提供多模型访问能力，使团队可为流水线的不同阶段分配不同的基础模型，而无须管理彼此独立的集成。Amazon Bedrock 还可通过 AWS Identity and Access Management（IAM）、虚拟私有云（VPC）端点和 AWS CloudTrail 继承 AWS 治理控制能力，从而满足 Orca 的安全要求。这种模型无关的架构意味着，团队可以在新模型版本可用时进行替换，而无须重新设计流水线架构。

下图展示了这一九阶段流水线架构。

图 1：多阶段 Discovery AI 流水线架构，展示如何使用 Amazon Bedrock 上的基础模型，将自然语言输入转换为经过验证的 Discovery 筛选器查询。

Discovery AI 流水线通过以下九个步骤，将自然语言请求转换为经过验证的 Discovery 筛选器。

1. 请求入口与上下文设置： 流水线首先将用户的自然语言请求与相关上下文结合，包括用户所属组织、已连接的云服务商和请求元数据。这些上下文有助于流水线准确理解请求，并生成符合客户环境的 Discovery 筛选器。
2. 快速确定性匹配器： Quick Matcher 会识别 IP 地址、常见漏洞与暴露（Common Vulnerabilities and Exposures，CVE）、ARN、文件哈希、Amazon Machine Image ID、Oracle Cloud Identifier、EC2 ID、URL 和告警 ID 等结构化输入。它使用正则表达式和基于规则的匹配来检测高置信度模式。当发现匹配项时，流水线可通过快速路径返回预定义的 Discovery 筛选器，无须使用大语言模型（LLM）推理。当未找到匹配项时，请求将继续进入下一步。
3. LLM Guardrail 筛查： Guardrail 组件检查请求是否安全、相关、具备技术性且受支持。它会在请求进入流水线其余部分前，阻止 Prompt 注入尝试、偏离主题的请求和不受支持的输入。未通过 Guardrail 检查的请求将被拒绝，并返回相应错误信息。获批请求将继续进行示例检索。
4. 基于 Embedding 的语义候选召回： 流水线使用 Amazon Titan Text Embeddings 搜索先前已验证的相关查询示例。它依据语义相似度从向量存储中检索最接近的 20 个示例。这些候选示例可展示类似自然语言请求此前是如何被转换为 Discovery 筛选器的。
5. 神经相关性重排序： 系统使用 Amazon Bedrock 上的 Cohere Rerank 3.5 对检索到的候选结果进行重排序，识别与当前请求最相关的示例，并选出信号最强的前五个示例。当找到精确匹配或高置信度匹配时，流水线可以直接复用已知有效的筛选器。否则，重排序后的示例会被传递给生成阶段。
6. LLM 驱动的相关模型选择： Model Selection 会将 900 多个 Sonar 模型缩小至与请求相关的一小部分模型。它在选择适当模型时，会考虑用户请求、已连接的服务商和检索到的示例。系统会根据适当的 Schema 层级，将选定模型提供给 Generator，Schema 范围从紧凑的微型 Schema 到更详细的小型 Schema 不等。这减少了不必要的上下文，并提高了生成效率。
7. 工具增强的 Agentic 筛选器生成： Generator 使用工具增强型 Agent 构建最终的 Discovery 筛选器。它会基于选定模型的 Schema、检索到的示例、用户上下文和可用工具进行推理。该 Agent 可在工作过程中迭代优化筛选器，直至形成对用户请求的有效表达。
8. 静态验证与 API 支持的验证： 系统会检查生成的筛选器在语法和语义上是否正确。流水线先执行静态验证，再通过 Orca 的 Sonar API 验证筛选器。若验证失败，验证错误会返回给 Generator，以便 Agent 修正筛选器并再次尝试。如果达到最大重试次数后仍未生成有效筛选器，流水线将返回无效筛选器响应。Generator 被限制在重排序后的示例和选定 Schema 范围内。静态验证会将每个模型、字段和运算符解析回相应的 Schema 定义。每一项验证错误都会连同出错元素一并返回给 Agent，因此重试是基于事实重新锚定，而不是重新猜测。
9. 后处理与响应： 在通过 Quick Matcher、复用已知有效示例或 Agentic 生成得到有效筛选器后，流水线将准备最终响应。

团队根据各阶段的需求，在不同组件之间分配了 Amazon Bedrock 上的不同基础模型。Anthropic Claude Haiku 4.5 为 Model Selector 和 Generator 提供支持，这些组件需要较强的推理能力以准确构建查询。Amazon Nova 2 Lite 用于 Guardrail 和 Validator，这些环节优先考虑低成本下的快速推理。Amazon Titan 为 Example Retrieval 组件中的语义搜索提供 Embedding。Cohere Rerank 3.5 对检索到的候选结果进行重排序，以最大化相关性。有关各 AWS Region 的模型可用性，请参阅按模型划分的区域可用性。

在示例检索中最大化相关性与多样性

示例检索是整个流水线中对准确性影响最大的单一因素。强示例能够比单独的 Schema 更高效地教会 Generator 如何将问题转换为 Sonar 查询，因此 Orca 团队在 Amazon Bedrock 上对多种检索策略进行了基准测试。每种策略在相关性、多样性或速度方面都有不同取舍，直接比较后揭示了哪种组合效果最佳。

基于 Amazon Titan Embedding 的纯 top-K 语义搜索速度较快，但往往会返回近乎重复的示例，导致 Generator 缺乏多样性。最大边际相关性（Maximal Marginal Relevance，MMR）提升了多样性，但当排名第一的候选结果恰好就是正确答案时，它会牺牲相关性。检索更大的候选集合后再进行重排序，能够引入更广泛的候选池，并利用重排序器选出最相关的子集，这为最终的获胜方案指明了方向。

最终胜出的方案是：使用 Amazon Titan Embedding 进行 top-20 最近邻检索，随后通过 Cohere Rerank 3.5 将结果重排序至前五名。该方法为 Generator 提供了一组规模较小、兼具多样性和高度相关性的示例，而无须在检索阶段就在覆盖率和精确度之间进行取舍。

将超过 900 个模型加载到上下文中

一个核心约束塑造了这一架构：无法在每个 Prompt 中都包含 900 多个 Sonar 模型 Schema。即使使用长上下文模型，这样做也会同时损害成本和准确性。

Model Selector 解决了这一问题。它运行在 Anthropic Claude Haiku 4.5 上，并使用每个模型紧凑但语义丰富的表示形式，包括名称、用途、关键字段和关键关系。根据用户问题，它只返回 Generator 实际所需的模型。随后，Generator 仅接收这些选定模型的完整 Schema 细节，以及连接它们的关系。

评估方法论

为能够有把握地进行迭代，Orca 团队在项目早期便构建了持续评估框架。该框架结合了精选数据集、自动化评分层和基于切片的测量方法，使团队能够在每次变更上线前对其充满信心。这些要素共同使评估成为例行步骤，而非事后补救。

精选评估数据集覆盖生产环境中出现的问题类型、模型族和复杂度层级，因此能够反映真实使用情况。LLM-as-a-judge 评分层会从正确性、完整性，以及与参考查询的语义等价性等维度为生成的查询评分，为团队提供一致的质量信号。测量方法会再按不同切片拆解指标变化，使团队能够准确定位回归发生的位置，以及需要修正的内容。

对 Prompt、模型、检索参数或示例所做的每一项变更，在发布到生产环境前都会经过该框架。

用于持续改进的领域专家参与式标注系统

随着客户环境演变和新问题模式不断出现，要维持准确性，不能只依赖一次强有力的初始部署。Orca 团队为产品经理、支持工程师和安全研究人员构建了内部标注工具。借助该工具，他们可以标记查询、修正输出，并通过一次操作将修正结果推送到示例库。此外，系统会自动收集与低置信度匹配示例相关的生产查询，并将其路由给相应领域专家进行每日标注。下一次流水线运行时，系统会通过 Cohere Rerank 3.5 检索新示例，该类问题的准确性将立即提升，无须部署。

“标注 UI 是我们为长期质量所做的最重要投资。它将 Discovery AI 从一个静态部署转变为一个持续演进的系统，我们的领域专家可以在日常工作流程中不断改进它。”

—— Orca Security AI 总监 Or Gabay

结合端到端流水线可观测性，以及覆盖产品、工程和客户服务团队的共享仪表板，这一反馈循环正是 Discovery AI 能够在首次发布后持续改进的关键。

成果

将 Discovery AI 部署到生产环境后，新方案在成本、延迟和准确性方面均优于初始基线。输入 Token 使用量下降了 93%，从每个可比工作负载约 200 万 Token 降至 14 万 Token。总体成本下降了 90%，主要得益于更低的 Token 消耗，以及在 Amazon Bedrock 上按阶段分配模型。

延迟同样得到改善，端到端响应时间下降了 75%，从约 60 秒降至 15 秒。针对复杂多模型查询的准确性也有所提升，团队通过评估框架对此进行持续测量。

自 2026 年 3 月以来，Discovery AI 已在生产环境中每天处理数千次查询，内部主题专家与客户反馈均验证了其质量。

结论

在

Security analysts need fast answers about their cloud infrastructure without deep familiarity with domain-specific query languages. Orca Security’s tool covers over 900 cloud models (the resource types Sonar tracks, such as an Amazon Elastic Compute Cloud (Amazon EC2) instance or an Amazon Simple Storage Service (Amazon S3) bucket). Querying that data required precise model names, nested relationship navigation, and selection from 18 operators (comparison functions like equals or contains used to filter results).

To solve this, Orca Security partnered with AWS to build Discovery AI, a natural language query agent on Amazon Bedrock. Orca provides context-aware security for cloud, code, and AI, and Sonar is its domain-specific query system where users search, filter, and explore across the entire data graph.

In this post, we share the architecture and results of Discovery AI, which converts a natural language question into a Sonar query. As a result, Orca reduced query costs by 90 percent, input tokens by 93 percent, and latency by 75 percent compared to the initial baseline.

Bridging natural language and structured queries

Orca Security’s Sonar query system supports 900+ cloud infrastructure and application models with complex relationships between them. To query this data, users construct JSON objects that specify exact model names, select from 18 operators, and navigate parent-child hierarchies across cloud providers. Doing this well requires deep familiarity with the Sonar schema.

To build this capability, the Orca team set out to build a natural language interface that can generate valid Sonar queries from plain text.

Their initial proof of concept loaded the full Sonar schema into a single prompt on the largest available model. This included over 900 model definitions, every relationship type, and a generous set of example queries. The output quality was strong (design partners and internal users responded positively) but three significant drawbacks emerged at production scale.

Token consumption was inefficient, at roughly 2 million input tokens per workload, which drove up cost. Latency was high, with end-to-end response times of around 60 seconds per workload. Accuracy on complex, multi-model queries was limited because nested relationships and multi-condition queries often required additional refinement.

The Orca team validated that the concept worked. The next challenge was making it cost-effective, fast, and accurate enough for production. To address this, the Orca team designed and deployed a solution that handles thousands of queries per day at 15-second response times using Amazon Bedrock.

Solution overview

Orca designed Discovery AI as a hybrid solution on Amazon Bedrock that combines Retrieval Augmented Generation (RAG) with schema-aware query construction. The system converts natural language questions into valid Discovery filters through a multi-stage pipeline, with each stage matched to the most appropriate Amazon Bedrock model.

Orca chose Amazon Bedrock because it provides multi-model access through a single API, letting the team allocate different foundation models to each pipeline stage without managing separate integrations. Amazon Bedrock also inherits AWS governance controls through AWS Identity and Access Management (IAM), virtual private cloud (VPC) endpoints, and AWS CloudTrail, meeting Orca’s security requirements. The model-agnostic architecture meant the team could swap in newer model versions as they became available without rearchitecting the pipeline.

The following diagram illustrates the nine-stage pipeline architecture.

Figure 1: Multi-stage Discovery AI pipeline architecture showing the flow from natural language input to validated Discovery filter queries using foundation models on Amazon Bedrock

The Discovery AI pipeline converts a natural language request into a validated Discovery filter through nine steps.

1. Request Entry and Context Setup: The pipeline starts by combining the user’s natural language request with relevant context, including the user’s organization, connected cloud providers, and request metadata. This context helps the pipeline interpret the request accurately and generate a Discovery filter that matches the customer’s environment.
2. Quick Deterministic Matcher: The Quick Matcher identifies structured inputs such as IP addresses, Common Vulnerabilities and Exposures (CVEs), ARNs, file hashes, Amazon Machine Image IDs, Oracle Cloud Identifiers, EC2 IDs, URLs, and alert IDs. It uses regex and rule-based matching to detect high-confidence patterns.When a match is found, the pipeline can return a predefined Discovery filter through a fast path, without using large language model (LLM) inference. When no match is found, the request continues to the next step.
3. LLM Guardrail Screening: The Guardrail component checks whether the request is safe, relevant, technical, and supported. It blocks prompt injection attempts, off-topic requests, and unsupported inputs before they reach the rest of the pipeline. Requests that fail the guardrail checks are rejected with an appropriate error. Approved requests continue to example retrieval.
4. Embedding-Based Semantic Candidate Recall: The pipeline searches for relevant previously validated query examples using Amazon Titan Text Embeddings. It retrieves the 20 nearest examples from the vector store based on semantic similarity. These candidates provide the pipeline with examples of how similar natural language requests were previously translated into Discovery filters.
5. Neural Relevance Reranking: The retrieved candidates are reranked using Cohere Rerank 3.5 on Amazon Bedrock. It identifies the examples that are most relevant to the current request and selects the five highest-signal examples. When an exact or high-confidence match is found, the pipeline can reuse the known-good filter directly. Otherwise, the reranked examples are passed to the generative stages.
6. LLM-Driven Relevant Model Selection: Model Selection narrows more than 900 Sonar models to the small set relevant to the request. It considers the user’s request, connected providers, and retrieved examples when selecting the appropriate models. The selected models are provided to the Generator using the appropriate schema level, ranging from compact micro schemas to more detailed mini schemas. This reduces unnecessary context and makes generation more efficient.
7. Tool-Augmented Agentic Filter Generation: The Generator uses a tool-augmented agent to construct the final Discovery filter. It reasons over the selected model schemas, retrieved examples, user context, and available tools. The agent can iteratively refine the filter as it works toward a valid representation of the user’s request.
8. Static and API-Backed Validation: The generated filter is checked for syntactic and semantic correctness. The pipeline first applies static validation and then verifies the filter against Orca’s Sonar API. When validation fails, the validation error is returned to the Generator so the agent can correct the filter and try again. If the maximum number of retries is reached without producing a valid filter, the pipeline returns an invalid-filter response. The Generator is constrained to the reranked examples and selected schemas. Static validation resolves every model, field, and operator back to those schema definitions. Each validation error goes back to the agent with the offending element identified, so retries re-ground rather than re-guess.
9. Post-Processing and Response: After a valid filter is produced, whether through the Quick Matcher, reuse of a known-good example, or agentic generation, the pipeline prepares the final response.The team allocated different foundation models on Amazon Bedrock across components based on each stage’s requirements. Anthropic Claude Haiku 4.5 powers the Model Selector and Generator. These components require strong reasoning for accurate query construction. Amazon Nova 2 Lite handles the Guardrail and Validator, where fast inference at lower cost is prioritized. Amazon Titan provides embeddings for semantic search in the Example Retrieval component. Cohere Rerank 3.5 reranks retrieved candidates to maximize relevance. For model availability by AWS Region, refer to Regional availability by models.

Maximizing relevance and diversity in example retrieval

Example retrieval emerged as the single biggest accuracy lever in the pipeline. Strong examples teach the Generator how to translate questions into Sonar queries more efficiently than schema alone, so the Orca team benchmarked several retrieval strategies on Amazon Bedrock. Each strategy traded off relevance, diversity, or speed in a different way, and comparing them directly showed which combination worked best.

Pure top-K semantic search on Amazon Titan embeddings performed quickly, but it tended to return near-duplicate examples that starved the Generator of diversity. Maximal Marginal Relevance (MMR) improved diversity, but it traded off relevance when the top candidate was actually the right answer. Retrieving a larger candidate set and then reranking it pulled in a broader pool and used a reranker to select the most relevant subset, which pointed toward the winning approach.

The winner was top-20 nearest-neighbor retrieval using Amazon Titan embeddings. Orca then reranked these results with Cohere Rerank 3.5 down to the top five. This approach gives the Generator a small, diverse, highly relevant set of examples without forcing a trade-off between coverage and precision at retrieval time.

Loading over 900 models into context

A core constraint shaped the architecture. It wasn’t feasible to include 900+ Sonar model schemas into every prompt. Even with long-context models, doing so degrades both cost and accuracy.

The Model Selector resolves this. Running on Anthropic Claude Haiku 4.5, it works against compact, semantically rich representations of each model (name, purpose, key fields, and key relationships). Given the user’s question, it returns only the models the Generator actually needs. The Generator then receives full schema detail only for those selected models plus the relationships that connect them.

Evaluation methodology

To iterate confidently, the Orca team built a continuous evaluation harness early in the project. The harness combines a curated dataset, an automated scoring layer, and a slice-based measurement approach so the team can trust every change before it ships. Together these elements turn evaluation into a routine step rather than an afterthought.

The curated evaluation dataset spans the question types, model families, and complexity tiers seen in production, so it reflects real usage. An LLM-as-judge scoring layer grades generated queries on correctness, completeness, and semantic equivalence to a reference query, which gives the team a consistent quality signal. The measurement methodology then breaks deltas down by slice, allowing the team to identify exactly where regressions occur and what to correct.

Every change to a prompt, a model, a retrieval parameter, or an example flows through the harness before it ships to production.

Domain-expert-in-the-loop labeling system for continuous improvement

Sustaining accuracy as customer environments evolve and new question patterns emerge required more than a strong initial deployment. The Orca team built an internal labeling tool for product managers, support engineers, and security researchers. With the tool, they can flag queries, correct outputs, and push corrections into the example bank in one action. Furthermore, the system automatically collects production queries with low-confidence matching examples and routes them to the appropriate domain expert for daily labeling. The next pipeline run retrieves the new example through Cohere Rerank 3.5, and accuracy on that class of question improves the same moment, with no deployment.

“The labeling UI is the single most important investment we made for long-term quality. It turns Discovery AI from a static deployment into a living system, with our domain experts continuously improving it inside their normal workflow.”

— Or Gabay, Director of AI, Orca Security

Combined with end-to-end pipeline observability and shared dashboards spanning product, engineering, and customer-facing teams, this feedback loop is what keeps Discovery AI improving after initial launch.

Results

After deploying Discovery AI to production, the new solution outperformed the initial baseline across cost, latency, and accuracy. Input token usage fell by 93 percent, from approximately 2 million tokens down to 140,000 per comparable workload. Overall cost fell by 90 percent, driven by that lower token consumption together with per-stage model allocation across Amazon Bedrock.

Latency also improved, with end-to-end response times dropping by 75 percent, from approximately 60 seconds to 15 seconds. Accuracy on complex, multi-model queries also improved, measured continuously through the team’s evaluation harness.

Discovery AI now handles thousands of queries per day in production since March 2026, with internal subject matter experts and customer feedback validating quality.

Conclusion

In this post, we shared how Orca Security built Discovery AI on Amazon Bedrock. The solution reduced query costs by 90 percent, input token usage by 93 percent, and latency by 75 percent compared to the initial baseline. The solution combines several key technologies. These include RAG-based example retrieval with Cohere Rerank 3.5, per-stage model allocation across Anthropic Claude Haiku 4.5 and Amazon Nova 2 Lite, and semantic search powered by Amazon Titan. Additionally, a continuous human-in-the-loop labeling system provides ongoing improvement. This combination allowed Orca to turn a working prototype into a production-grade solution that is both cheaper to run and more accurate.

To get started building similar solutions, explore Amazon Bedrock for foundation model (FM) access, Cohere Rerank on Amazon Bedrock for retrieval optimization, and Amazon Titan for embeddings. To learn more about Orca Security’s cloud security tool, visit orca.security.

To learn more about how Orca Security applies AI across its framework, see how Orca drives security outcomes with AI and how Orca secures AI adoption, including Amazon Bedrock.

For more on the techniques used in this post, read related posts on the AWS Machine Learning Blog.

---

About the authors

Deniz Yilmaz

Deniz is a Machine Learning Engineer at the AWS Generative AI Innovation Center. She partners with enterprise customers across EMEA to design and deploy production-grade generative AI solutions, from proof of concept through to scaled deployment. Her work focuses on agentic architectures, retrieval-augmented generation, and evaluation methodology for building reliable AI systems in production.

Natalie Tarasova

Natalie is a Machine Learning Engineer in the AWS Custom Model Optimisation team. She focuses on LLM fine-tuning, continued pretraining, and post-training. She is the author of SWE-InfraBench, a benchmark for LLMs on infrastructure-as-code.

Julia Wagner

Julia is a Senior AI Strategist at the AWS Generative AI Innovation Center. She partners with large enterprises and innovative startups across multiple industry verticals to define and implement generative and agentic AI strategies and products. With over a decade of product and technology leadership experience, she helps organizations automate complex workflows, accelerate R&D, and build the organizational capabilities needed for scaled AI adoption.

Barak Tarab

Barak is the AI Team Lead at Orca Security, driving the development of advanced artificial intelligence capabilities. His professional focus centers on building scalable AI solutions and robust infrastructure.

Saurav Hiremath

Saurav, Software Engineer at Orca Security. Lead and built API Security suite at Orca and now transitioned to building Agentic AI systems for enterprise customers. Occasionally spotted contributing to OSS (prev. Facebook’s Jest, React Hermes)