Amazon Bedrock 中 Automated Reasoning 策略的自动化优化

在 Amazon Bedrock 中优化 Automated Reasoning 策略，过去一直是一个由诊断、手动编辑、重新测试和重复操作组成的人工循环。今天，我们宣布推出自动策略优化功能，将这一循环中的诊断与修复工作自动化。优化引擎会诊断失败的测试，并提出形式逻辑修复方案。每一项变更在生效前都需要由您批准。

Amazon Bedrock Guardrails 中的 Automated Reasoning 检查使用形式化验证来证明答案的正确性。正如 GA 发布公告所述，对于从自然语言到形式逻辑的无歧义转换，其验证准确率最高可达 99%。开始使用时，您需要根据源文档构建 Automated Reasoning 策略，并通过测试用例对其进行验证。客户告诉我们，这种迭代调优过程是策略开发中最大的摩擦点。

本文将介绍两种新的优化模式：用于解决规则问题的 Iterative Refinement，以及用于解决语言问题的 Ambiguous Variable Refinement。对于每种模式，我们都会展示完整的 API 工作流（启动、轮询、获取结果），以及可重复使用的控制台工作流，用于将失败的策略转化为通过测试的策略。

Automated Reasoning 检查究竟是什么

Automated Reasoning 检查将自然语言转换为形式逻辑，然后应用自动推理技术生成结论：VALID、INVALID、SATISFIABLE、IMPOSSIBLE 或 TRANSLATION_AMBIGUOUS。有关策略工作方式的完整介绍，请参阅我们的 GA 发布公告文章。

本文的关键概念是 两步验证管道。首先，translate 步骤使用策略中的变量描述，将自然语言输入/输出映射为变量赋值。其次，validate 步骤将形式规则应用于这些赋值。测试失败时，根本原因存在于这两个步骤之一，而每种优化模式分别针对不同的步骤。图 1 展示了该管道的端到端流程。

图 1：Automated Reasoning 检查如何在运行时验证响应。Automated Reasoning 检查使用策略的变量描述将自然语言转换为变量，然后根据策略的形式规则验证这些变量并返回结论。正是由于该两步管道，优化功能才分为两种模式。

测试策略。 您可以通过附加测试来验证策略：每个测试由输入/输出文本以及预期结果组成。您可以单独运行测试，也可以批量运行。失败结果会准确指出策略与您意图不一致的位置。

为什么策略需要优化：两种失败模式

回顾两步管道：先是 translate（自然语言到变量赋值），然后是 validate（形式逻辑到结论）。测试失败意味着其中一个步骤产生了与您预期不同的结果。Automated Reasoning 检查会呈现两种明确的失败信号，且每种信号都可清晰地映射到相应步骤。

失败模式 1：规则问题（逻辑有误）

在规则问题导致的失败中，转换过程是正确的：正确的变量获得了正确的值，但验证结果与您的预期不一致。问题存在于规则中：某项规则过于宽松、过于严格，或完全缺失。具体而言，您可能预期得到 INVALID，但却得到 SATISFIABLE，因为缺失或过于宽松的规则让错误答案通过了验证。或者，您可能预期得到 SATISFIABLE，但却得到 INVALID，因为过于严格的规则阻止了正确答案。

心智模型： 系统完全理解了问题，但应用了错误的逻辑。您需要修复的是 规则。

失败模式 2：翻译存在歧义（语言有误）

当测试返回 TRANSLATION_AMBIGUOUS 时，验证引擎会根据所采用的解释产生不同结果。在某些情况下，翻译模型对于如何将自然语言输入映射到策略变量存在分歧，每种相互竞争的解释都会导致不同的验证结果。该结论会显示两个或更多选项，每个选项都包含自己的翻译和结论，并通过 differenceScenarios 展示这些解释在实际应用中发生分歧的位置。常见根本原因包括重叠的变量定义（例如“任职期限”与“服务年限”）、模糊的描述，以及不一致的值格式（例如“5%”被表示为 5 或 0.05）。

失败模式与优化模式的对应关系

下表概括了每种优化模式所处理的失败类型：

失败模式	根本原因	优化模式	作用
规则问题	逻辑有误	Iterative Refinement	提出规则或变量的新增、编辑或删除建议
翻译存在歧义	语言存在歧义	Ambiguous Variable Refinement	提出更清晰的变量描述，将多种解释收敛为一种

当系统无法确定唯一翻译时，应使用 Ambiguous Variable Refinement。接下来的两节将依次介绍每种模式：其作用、适用场景、审查机制，以及如何以编程方式启动。我们先介绍 Iterative Refinement，因为规则问题导致的失败更为常见。

Iterative Refinement：修复规则

当测试因逻辑错误而失败时（即转换结果清晰，但验证结果与您的预期不一致），问题出在规则中。Iterative Refinement（ITERATIVELY_REFINE_POLICY）可自动执行诊断和修复循环，因此您无需手动追踪每条规则、推测修正方式并手工编辑形式逻辑。

假设某个策略包含 10 至 30 条规则。此前，修复问题需要领域专家经过多轮人工诊断和手动编辑 SMT-LIB 形式逻辑。现在，这项工作被压缩为一次审查和批准步骤，且无需手写形式逻辑。

工作原理

Iterative Refinement 接收三项输入。第一项是您现有的策略定义，包括当前的规则、变量和类型。第二项是源文档，其中包含描述规则 应当 如何运作的权威自然语言文本。第三项为可选输入：以自然语言提供的反馈，其中包含您希望进行的变更的明确说明。

例如，反馈字段可以包含：“根据修订文档第 3 节的规定，将育儿假的任职期限要求从 12 个月更新为 6 个月。”

基于这些输入，优化引擎会分析当前规则与源文档及您的反馈之间的偏差。随后，它会提出一组候选变更方案，例如新增规则、编辑规则和添加变量，使策略保持一致。

收敛循环

顾名思义，Iterative Refinement 会进行迭代。在幕后，引擎会生成候选变更，模拟该变更对已保存测试的影响，检查此前失败的测试现在是否能通过；如果不能，则继续调整。对于单个请求，这可能涉及多个内部循环，尤其是当一项规则修复会连带影响其他规则时。不过，迭代过程发生在 内部：您不会看到每次中间尝试，也无需对其进行引导。您最终获得的是收敛后的结果：一份建议差异，准确显示哪些规则发生了变化、哪些变量发生了变化，以及该变更如何影响测试套件中的每项测试。

审查关卡

收敛后，将出现 Review policy changes（审查策略变更）界面。

随后，您可以选择 Accept changes（接受变更）或 Discard changes（丢弃变更）。接受将把变更写入您的 DRAFT 策略。丢弃则会使所有内容保持原样。

前提条件与适用场景

Iterative Refinement 要求至少有一项测试附加到策略中。没有失败测试信号，就没有可驱动优化的依据。当转换结果正确（变量正确、值正确），但验证结果不符合预期时，应使用此模式。当结论为 TRANSLATION_AMBIGUOUS 时，请不要使用它。那是语言问题，更适合通过 Ambiguous Variable Refinement 解决。

通过 API 使用

优化以异步构建工作流的形式运行。使用 AWS SDK for Python（Boto3）时，该流程包含四个步骤：导出当前策略定义、启动工作流、轮询完成状态，以及获取建议的变更。将 buildWorkflowType 设置为 ITERATIVELY_REFINE_POLICY。

iterativeRefinementContent 块接受一到五份源文档（必填），以及最多 4,000 个字符的可选反馈：

import boto3

bedrock = boto3.client("bedrock")

# Export the current policy definition (required input to the workflow).
policy_definition = bedrock.export_automated_reasoning_policy_version(
    policyArn=policy_arn,
)["policyDefinition"]

with open("hr-leave-policy.pdf", "rb") as f:
    source_document = f.read()

response = bedrock.start_automated_reasoning_policy_build_workflow(
    policyArn=policy_arn,
    buildWorkflowType="ITERATIVELY_REFINE_POLICY",
    sourceContent={
        "policyDefinition": policy_definition,
        "workflowContent": {
            "iterativeRefinementContent": {
                "documents": [
                    {
                        "document": source_document,
                        "documentName": "hr-leave-policy.pdf",
                        "documentContentType": "pdf",
                    }
                ],
                "feedback": "Update the tenure requirement from 12 to 6 months per section 3.",
            }
        },
    },
)

build_workflow_id = response["buildWorkflowId"]

该调用会立即返回一个 buildWorkflowId，而不是建议的变更。工作流会从 SCHEDULED 转为 BUILDING，直至达到 COMPLETED、FAILED 或 CANCELLED 状态。收敛通常需要一到数分钟，具体取决于策略规模。轮询 get_automated_reasoning_policy_build_workflow，直到状态达到终止状态。然后，通过 get_automated_reasoning_policy_build_workflow_result_assets 获取收敛后的建议；请求 POLICY_DEFINITION 资产可查看更新后的规则，获取 BUILD_LOG 则可查看操作日志：

import time

while True:
    workflow = bedrock.get_automated_reasoning_policy_build_workflow(
        policyArn=policy_arn,
        buildWorkflowId=build_workflow_id,
    )
    status = workflow["status"]
    if status in ("COMPLETED", "FAILED", "CANCELLED"):
        break
    time.sleep(10)

if status == "COMPLETED":
    assets = bedrock.get_automated_reasoning_policy_build_workflow_result_assets(
        policyArn=policy_arn,
        buildWorkflowId=build_workflow_id,
        assetType="POLICY_DEFINITION",
    )
    proposed_definition = assets["buildWorkflowAssets"]["policyDefinition"]

返回的策略定义是建议的 DRAFT，即完整的新定义。要提交它，请使用此定义调用 update_automated_reasoning_policy。要查看具体变更，请将其与启动工作流前导出的策略定义进行差异比较。控制台中的 Review policy changes 界面封装了相同的启动、轮询和获取流程，并在 Accept changes 与 Discard changes 按钮背后呈现差异内容。

Ambiguous Variable Refinement：修复语言

Iterative Refinement 可处理规则问题，但并非所有失败测试都是规则问题。当转换本身不稳定时，任何规则编辑都无法解决问题。您需要修复策略用于描述变量的语言。这正是 Ambiguous Variable Refinement 的作用。它遵循相同的异步启动、轮询和获取模式，并进入相同的审查与接受界面。区别在于建议内容：其重点是变量描述和变量合并，并会在需要时更新规则和类型以保持策略一致性。

当测试产生 TRANSLATION_AMBIGUOUS 结果时（请参阅本文前面的失败模式 2），相互竞争的翻译会导致不同验证结果。歧义也可能来自待验证内容本身的表述方式。本节聚焦于策略变量中的歧义。

工作原理

由策略变量问题引起的翻译歧义，通常源于少数几类根本原因。变量重叠发生在两个变量描述同一概念时。例如，tenureMonths（“员工以月为单位的工作时长”）和 monthsOfService（“员工的服务月数”）都表示雇佣持续时间。因此，翻译模型会对应该使用哪一个产生分歧。不完整的描述出现在变量描述过于模糊、无法指导翻译时。不一致的值格式会造成歧义，因为系统无法判断“5%”应被转换为 interestRate = 5，还是 interestRate = 0.05。将逻辑写入变量名也会造成混淆。像 timelyReportingNotFeasible 这样的名称已经包含否定，因此要表达肯定情形时，需要对否定再进行否定。翻译模型往往会漏掉其中一个否定。

这些只是最常见的模式。由于检测机制是通过在翻译过程中实际使用策略变量来发现问题，而非根据固定的已知问题列表进行检查，因此，任何导致翻译结果出现分歧的变量级问题都可能被发现。

当您运行 Ambiguous Variable Refinement 时，它会准确定位导致分歧的变量描述或重叠定义，然后提出优化后的

Refining an Automated Reasoning policy in Amazon Bedrock has been a manual cycle of diagnose, hand-edit, retest, and repeat. Today, we are announcing automatic policy refinement, which automates the diagnose-and-fix work in that cycle. The refinement engine diagnoses failing tests and proposes formal-logic fixes. You approve every change before it takes effect.

Automated Reasoning checks in Amazon Bedrock Guardrails use formal verification to prove answer correctness. On unambiguous translations from natural language to formal logic, they deliver up to 99% verification accuracy, as reported in the GA announcement. To get started, you build an Automated Reasoning policy from a source document and validate it with test cases. Customers told us that this iterative tuning creates the biggest friction point in policy development.

In this post, we walk through two new refinement modes: Iterative Refinement for rule issues, and Ambiguous Variable Refinement for language issues. For each mode, we show a complete API workflow (start, poll, retrieve) and a repeatable console workflow for turning failing policies into passing ones.

What Automated Reasoning checks actually are

Automated Reasoning checks translate natural language into formal logic, then apply automated reasoning techniques to produce a finding: VALID, INVALID, SATISFIABLE, IMPOSSIBLE, or TRANSLATION_AMBIGUOUS. For a full introduction to how policies work, refer to our GA announcement post.

For this post, the key concept is the two-step validation pipeline. First, the translate step maps natural-language input/output to variable assignments using the variable descriptions in your policy. Second, the validate step applies your formal rules to those assignments. When a test fails, the root cause lives in one of those two steps, and each refinement mode targets a different one. Figure 1 traces that pipeline end to end.

Figure 1: How Automated Reasoning checks validate a response at runtime. Automated Reasoning checks translate natural language into variables using the policy’s variable descriptions, then validate those variables against the policy’s formal rules to return a finding. This two-step pipeline is why refinement has two modes.

Testing your policy. You validate a policy by attaching tests: each test is input/output text plus the result you expect. Run tests individually or as a batch. Failures tell you exactly where the policy diverges from your intent.

Why policies need refinement: Two failure modes

Recall the two-step pipeline: translate (natural language to variable assignments) and then validate (formal logic to finding). A failed test means one of these steps produced something you didn’t expect. Automated Reasoning checks surface two distinct failure signals that map cleanly to each step.

Failure mode 1: Rule issues (the logic is wrong)

In a rule-issue failure, the translation works correctly: the right variables have the right values, but the validation result doesn’t match your expectation. The problem lives in your rules: a rule is too permissive, too restrictive, or missing entirely. Concretely, you expected INVALID but got SATISFIABLE because a missing or too-permissive rule lets a bad answer through. Or you expected SATISFIABLE but got INVALID because an overly strict rule blocks a correct answer.

Mental model: The system understood the question perfectly but applied the wrong logic. You need to fix the rules.

Failure mode 2: Translation ambiguous (the language is wrong)

When a test returns TRANSLATION_AMBIGUOUS, the validation engine runs and produces different outcomes depending on which interpretation it follows. In some cases, the translation models disagreed on how to map the natural-language input to your policy’s variables, and each competing interpretation led to a different validation result. The finding surfaces two or more options, each with its own translation and conclusion, plus differenceScenarios showing where the interpretations diverge in practice. Common root causes include overlapping variable definitions (“tenure” compared to “years of service”), vague descriptions, and inconsistent value formats (5 compared to 0.05 for “5%”).

Matching mode to failure

This table summarizes which refinement mode addresses which failure type:

Failure mode	Root cause	Refinement mode	What it does
Rule issues	Logic is wrong	Iterative Refinement	Proposes rule or variable additions, edits, or deletions
Translation ambiguous	Language is ambiguous	Ambiguous Variable Refinement	Proposes clearer variable descriptions that collapse multiple interpretations into one

Use Ambiguous Variable Refinement when the system cannot determine a single translation. The next two sections walk through each mode in turn: what it does, when to use it, how the review gate works, and how to launch it programmatically. We start with Iterative Refinement because rule-issue failures are the more common case.

Iterative Refinement: Fixing the rules

When tests fail because the logic is wrong (the translation is clean but the validation result doesn’t match your expectation), the problem lives in your rules. Iterative Refinement (ITERATIVELY_REFINE_POLICY) automates the diagnose-and-fix cycle so you don’t need to manually trace each rule, hypothesize a correction, and hand-edit formal logic.

Consider a policy with 10–30 rules. Previously, a fix would take a subject matter expert multiple rounds of manual diagnosis and hand-editing of SMT-LIB formal logic. That work now compresses to a single review-and-approve step, with no formal logic written by hand.

How it works

Iterative Refinement takes three inputs. The first is your existing policy definition (the current rules, variables, and types). The second is a source document containing the authoritative natural-language text that describes how things should work. The third input is optional: natural language feedback with explicit instructions describing the change you want.

For example, the feedback field might contain: “Update the tenure requirement for parental leave from 12 months to 6 months, as specified in section 3 of the revised document.”

Given these inputs, the refinement engine analyzes how the current rules diverge from the source document and your feedback. It proposes a set of candidate changes (new rules, edited rules, added variables) that bring the policy in line.

The convergence loop

Iterative Refinement, as the name suggests, iterates. Behind the scenes, the engine generates a candidate change, simulates its effect on your saved tests, checks whether the previously failing tests now pass, and adjusts if they don’t. This can involve several internal cycles for a single request, especially when a fix in one rule ripples into others. The iteration happens internally , though: you don’t observe each intermediate attempt, and you don’t need to shepherd it. What you receive is the converged result: a proposed diff that shows exactly which rules changed, which variables changed, and how the change affects every test in your suite.

The review gate

After convergence, the Review policy changes screen appears.

You then select Accept changes or Discard changes. Accepting writes the changes to your DRAFT policy. Discarding leaves everything exactly as it was.

Prerequisites and when to use

Iterative Refinement requires at least one test attached to your policy. Without a failing test signal, there’s nothing to drive the refinement. Use this mode when the translation is correct (right variables, right values) but the validation result is unexpected. Do not use it when the finding is TRANSLATION_AMBIGUOUS. That’s a language problem better addressed by Ambiguous Variable Refinement.

Through the API

Refinement runs as an asynchronous build workflow. Using the AWS SDK for Python (Boto3), the flow has four steps: export the current policy definition, start the workflow, poll for completion, and retrieve the proposed changes. Set buildWorkflowType to ITERATIVELY_REFINE_POLICY.

The iterativeRefinementContent block accepts one to five source documents (required) and up to 4,000 characters of optional feedback:

import boto3

bedrock = boto3.client("bedrock")

# Export the current policy definition (required input to the workflow).
policy_definition = bedrock.export_automated_reasoning_policy_version(
    policyArn=policy_arn,
)["policyDefinition"]

with open("hr-leave-policy.pdf", "rb") as f:
    source_document = f.read()

response = bedrock.start_automated_reasoning_policy_build_workflow(
    policyArn=policy_arn,
    buildWorkflowType="ITERATIVELY_REFINE_POLICY",
    sourceContent={
        "policyDefinition": policy_definition,
        "workflowContent": {
            "iterativeRefinementContent": {
                "documents": [
                    {
                        "document": source_document,
                        "documentName": "hr-leave-policy.pdf",
                        "documentContentType": "pdf",
                    }
                ],
                "feedback": "Update the tenure requirement from 12 to 6 months per section 3.",
            }
        },
    },
)

build_workflow_id = response["buildWorkflowId"]

The call returns immediately with a buildWorkflowId, not the proposed changes. The workflow moves from SCHEDULED to BUILDING until it reaches COMPLETED, FAILED, or CANCELLED. Convergence typically takes one to a few minutes, depending on policy size. Poll get_automated_reasoning_policy_build_workflow until the status reaches a terminal state. Then retrieve the converged proposal with get_automated_reasoning_policy_build_workflow_result_assets, requesting the POLICY_DEFINITION asset to see the updated rules (and BUILD_LOG for the action log):

import time

while True:
    workflow = bedrock.get_automated_reasoning_policy_build_workflow(
        policyArn=policy_arn,
        buildWorkflowId=build_workflow_id,
    )
    status = workflow["status"]
    if status in ("COMPLETED", "FAILED", "CANCELLED"):
        break
    time.sleep(10)

if status == "COMPLETED":
    assets = bedrock.get_automated_reasoning_policy_build_workflow_result_assets(
        policyArn=policy_arn,
        buildWorkflowId=build_workflow_id,
        assetType="POLICY_DEFINITION",
    )
    proposed_definition = assets["buildWorkflowAssets"]["policyDefinition"]

The returned policy definition is the proposed DRAFT, the full new definition. To commit it, call update_automated_reasoning_policy with this definition. To see what changed, diff it against the policy definition you exported before starting the workflow. The console Review policy changes screen wraps this same start-poll-retrieve sequence, rendering the diff behind the Accept changes and Discard changes buttons.

Ambiguous Variable Refinement: Fixing the language

Iterative Refinement handles rule issues, but not every failing test is a rule issue. When the translation itself is unstable, no amount of rule-editing will help. You need to fix the language the policy uses to describe its variables. That is what Ambiguous Variable Refinement does. It follows the same asynchronous start-poll-retrieve pattern and lands on the same review-and-accept screen. The difference is in the proposals. They center on variable descriptions and merges, with rule and type updates applied as needed to keep the policy consistent.

When tests produce TRANSLATION_AMBIGUOUS results (refer to failure mode 2 earlier in this post), competing translations lead to different validation outcomes. Ambiguity can also come from how the validated content itself is phrased. This section focuses on ambiguity in the policy variables.

How it works

Translation ambiguity, because of policy variable issues, typically stems from a handful of root causes. Overlapping variables occur when two variables describe the same concept. For example, tenureMonths (“How long the employee has worked in months”) and monthsOfService (“The employee’s months of service”) both capture employment duration. As a result, translation models disagree on which one to use. Incomplete descriptions arise when a variable’s description is too vague to guide translation. Inconsistent value formatting creates ambiguity when the system can’t determine if “5%” should become interestRate = 5 or interestRate = 0.05. Logic baked into variable names creates confusion. A name like timelyReportingNotFeasible already contains a negation, so expressing the positive case requires negating a negative. Translation models often drop one of the two.

These are only the most common patterns. Because detection works by exercising the policy’s variables in translation rather than checking against a fixed list of known problems, variable-level issues that causes translations to disagree can surface.

When you run Ambiguous Variable Refinement , it pinpoints which variable descriptions or overlapping definitions cause the disagreement, then proposes refined descriptions that collapse multiple interpretations into one precise definition.

These refined descriptions incorporate unit conversion rules, synonyms, alternative phrasings, and explicit format guidance. This before/after example illustrates a typical proposal:

Before	After (proposed)
tenureMonths : “How long the employee has worked in months.”	tenureMonths : “The number of complete months the employee has been continuously employed. When users mention years of service, convert to months (for example, 2 years = 24 months). This variable captures references to employment duration, length of service, time at the company, or seniority.”

If overlapping variables are detected, a merge may also be proposed: one variable is deleted and rules that reference it are updated to use the surviving variable.

The review gate

Just like Iterative Refinement , you review the proposed changes before anything is applied.

It also displays Test results, where tests that previously returned TRANSLATION_AMBIGUOUS now produce a definitive VALID, INVALID, or SATISFIABLE result. You select Accept changes or Discard changes. No change touches your DRAFT policy until you approve.

When to use it

Use Ambiguous Variable Refinement when tests produce TRANSLATION_AMBIGUOUS results, or when you inspect a VALID/INVALID finding and discover that the translation assigned values to the wrong variables. Do not use it when the translation is correct but the validation result is unexpected. That’s a rule problem for Iterative Refinement.

Through the API

Ambiguous Variable Refinement uses the same asynchronous start-poll-retrieve pattern. Set buildWorkflowType to RESOLVE_POLICY_AMBIGUITIES. This mode analyzes your policy’s variables directly and needs neither a source document nor attached tests, so workflowContent can be omitted. The current policy definition is still required in sourceContent. Export it first with export_automated_reasoning_policy_version as shown for Iterative Refinement earlier:

response = bedrock.start_automated_reasoning_policy_build_workflow(
    policyArn=policy_arn,
    buildWorkflowType="RESOLVE_POLICY_AMBIGUITIES",
    sourceContent={
        "policyDefinition": policy_definition,
    },
)

build_workflow_id = response["buildWorkflowId"]

As with Iterative Refinement , the response is a buildWorkflowId. Poll get_automated_reasoning_policy_build_workflow until the status reaches a terminal state. Then call get_automated_reasoning_policy_build_workflow_result_assets with assetType="POLICY_DEFINITION" to retrieve the proposed variable descriptions and merges:

while True:
    workflow = bedrock.get_automated_reasoning_policy_build_workflow(
        policyArn=policy_arn,
        buildWorkflowId=build_workflow_id,
    )
    status = workflow["status"]
    if status in ("COMPLETED", "FAILED", "CANCELLED"):
        break
    time.sleep(10)

if status == "COMPLETED":
    assets = bedrock.get_automated_reasoning_policy_build_workflow_result_assets(
        policyArn=policy_arn,
        buildWorkflowId=build_workflow_id,
        assetType="POLICY_DEFINITION",
    )
    proposed_definition = assets["buildWorkflowAssets"]["policyDefinition"]

The changes remain a proposal until you accept them.

You approve every change: The human-in-the-loop gate

Both refinement modes share one non-negotiable property: no change takes effect until you say so. The refinement engine has suggestion authority. It can analyze, diagnose, and propose. You have commit authority. You decide what reaches your DRAFT policy and, ultimately, production.

The five-step loop

Regardless of which refinement mode you use, the workflow follows the same five-step pattern. First, test: run your saved tests against the current policy. Second, check: identify which tests don’t match their expected result. Third, propose: the system generates candidate fixes for rules or variable descriptions. Fourth, review: you inspect the proposed diff and its impact on the tests. Fifth, apply: you accept the changes into DRAFT, or reject and nothing changes.

After you accept, re-test to confirm the fix resolved the issue without breaking other tests. This creates a ratchet: each cycle either moves you closer to a passing policy or gives you new diagnostic information.

What the review screen shows you

The review screen lets you understand the ramifications of a change, not just the change itself. It answers two questions at once: what did the engine propose, and how does that proposal affect every test you care about?

The proposal comes in three sections. Changes to rules lists the rules added, edited, or deleted, with the formal-logic expression for each. Changes to variables shows updated variable descriptions and added or deleted variables, with the previous wording alongside the proposed wording so you can compare the two directly. Changes to custom variable types covers changes to the policy’s enumerated types.

Alongside those changes, the screen shows a Test results section that lists the saved test with its previous and new outcome. Each row gives the expected finding and whether the test passed before and after. Choose View findings on a row to see the finding itself. This view is the single most important indicator on the screen. If your failing tests now pass and your passing tests still pass, you can accept with confidence.

Validate with the Fidelity Report

After applying changes, generate a Fidelity Report (GENERATE_FIDELITY_REPORT) to validate that your updated policy still faithfully represents your source document. The report provides three measurements. The coverage score (0.0–1.0) indicates how much of your source document is represented in the policy. The accuracy score (0.0–1.0) indicates how faithfully the rules match the intent of the original document. Per-rule grounding links each rule to the specific source-document statements that support it, with justifications.

Compare Fidelity Reports before and after refinement. If your accuracy score drops, the proposed fix may have drifted from your source material, which is a signal to reject or iterate further.

The principle

Automatic refinement accelerates the labor of diagnosing failures and generating candidate fixes. It does not accelerate the authority to change what your guardrail enforces. Every fix remains a suggestion until you decide to accept it.

Steering the engine: Source documents and custom feedback

You can steer both modes by providing context that guides the system toward the right fix faster.

Source documents

When you launch Iterative Refinement , you supply a source document representing the ground truth your policy should encode. The console offers three modes for supplying it: Recently used re-selects a previously uploaded doc, Upload takes a new PDF or text file, and Enter text accepts pasted content directly. The clearer and more focused the document, the more precise the proposals.

Custom feedback: Explicit if-then guidance

You can also provide natural language feedback that tells the system exactly what to fix. Effective feedback is specific and testable. Vague feedback like “Fix the tenure rule” gives the engine too much latitude. Compare it with a specific, testable alternative: “If an employee is full-time and has worked for more than 6 months (not 12), they should be eligible for parental leave.”

Your feedback acts as a constraint: the system generates changes that satisfy both the source document and your guidance. If the two conflict, the conflict is flagged for your review.

You can provide both a source document and feedback together. When your document is dense, feedback focuses the system on the specific section that matters.

End-to-end walkthroughs

This section provides two walkthroughs, one for each refinement mode. Together they give you a repeatable workflow for both failure types.

Prerequisites

To use automatic policy refinement with Automated Reasoning checks in Amazon Bedrock, make sure you have met the following prerequisites:

• An active AWS account.
• Confirmation of AWS Regions where Automated Reasoning checks is available, and access to Amazon Bedrock in one of them.
• IAM permissions to create, view, and refine Automated Reasoning policies and to work with Amazon Bedrock Guardrails.
• An Automated Reasoning policy in your account, created from a source document that captures the rules you want to enforce. For step-by-step instructions, refer to Create your Automated Reasoning policy. You need this policy to have at least one attached test — Iterative Refinement requires a failing test signal to drive its diagnosis. To add tests, refer to Test an Automated Reasoning policy.

Walkthrough A: Fixing a rule issue with Iterative Refinement

Your HR leave-eligibility policy has a failing test. The assistant reports that a part-time employee with 8 months of tenure qualifies for parental leave. The test expects INVALID, but the policy returns VALID.

Step 1: Validate the tests

1. In the Amazon Bedrock console, navigate to Automated Reasoning and open your policy.
2. Choose the Tests tab, and then choose Validate all tests. Figure 2 shows the results. Three of the four tests pass, and the parental-leave test fails with an unexpected VALID finding.

Figure 2: The Tests tab after validation, with three of four tests passing (75%)

Step 2: Inspect the failing finding

3. Open the failing test’s finding and inspect the translation, shown in Figure 3. The variables are correctly captured:
   1. monthsOfContinuousService = 8.
   2. employmentStatus = PART_TIME.

Figure 3: The failing test’s finding, with the translation showing correctly captured variables

The finding is VALID because the only supporting rule grants eligibility after 6 months of continuous service, and no rule excludes part-time employees. The translation is clean and the logic is wrong, so this is a rule issue.

Step 3: Refine the policy

1. Choose Refine policy , then choose Automatically refine policy.
2. Set the refinement type to Iterative Refinement.
3. Provide a source document using any of the three modes:
   1. Recently used.
   2. Upload (a new version).
   3. Enter text (paste the relevant paragraph).
4. Optionally, add custom feedback to focus the fix, for example: “ Add a rule that explicitly makes part-time employees ineligible for parental leave. ”
5. When your inputs are ready, choose Start.

Figure 4 shows the completed setup, with the refinement type selected, the source document attached, and the custom feedback entered.

Figure 4: Iterative Refinement setup with a source document and custom feedback

Step 4: Review and accept the changes

1. When convergence completes, the Review policy changes screen appears. Under Changes to rules , it lists two changes: one deleted rule and one added rule. The engine removed the rule that granted eligibility on continuous service alone:

   •    * if `monthsOfContinuousService` is at least 6, then `isEligibleForParentalLeave` is `true`
     

And added a rule that excludes part-time employees:

 *        * if `employmentStatus` is equal to `PART_TIME`, then `isEligibleForParentalLeave` is false

Under Test results , the part-time test moves from Failed to Passed , and the other three tests remain Passed. Figure 5 shows the screen before you accept.

Figure 5: The Review policy changes screen. Changes to rules lists the deleted tenure-only eligibility rule and the added part-time exclusion, and Test results shows the part-time test moving from Failed to Passed with no regressions.

Step 5: Confirm the fix

1. Return to the Tests tab and choose Validate all tests again.
2. Figure 6 shows the expected result. The previously failing test now passes, and the pass rate reaches 100%.

Figure 6: The Tests tab after refinement, with all four tests passing (100%)

If tests still fail or regressions appear

There is no guarantee that a single refinement cycle resolves all failures or avoids regressions. If the previously failing test still fails, or if other tests that previously passed are now failing:

1. Choose Discard changes on the review screen.
2. Add more specific custom feedback describing what the engine should fix differently. For example, “The rule should exclude part-time employees entirely, not only reduce their eligibility window.”
3. Narrow the source document to the exact section covering the failing rule.
4. Run Iterative Refinement again with these tighter inputs.

If the test still won’t pass after two or three rounds, fall back to manual editing. Use the failing finding’s translation and rule trace as a guide to hand-edit the specific rule in the policy editor.

Step 6: Report and deploy

1. Select Generate Fidelity Report and compare scores with the previous report.
2. When satisfied, create a numbered version to deploy.

Walkthrough B: Fixing a language issue with Ambiguous Variable Refinement

You have the same HR policy, but this time a test returns TRANSLATION_AMBIGUOUS. The finding shows two options:

• One interpreted “2 years of service” as tenureMonths = 24.
• The other as monthsOfService = 24.

The two options reveal overlapping variables. Both tenureMonths and monthsOfService describe employment duration, and the translation models can’t agree on which to use. This overlap is the root cause of the ambiguity.

Step 1: Refine the policy

1. Choose Refine policy , then Ambiguous Variable Refinement.
2. As Figure 7 shows, no source document is required for this mode, so the setup is a single selection.

Figure 7: Ambiguous Variable Refinement setup, with no source document required

3. Choose Start.

Step 2: Review and accept the changes

1. On the Review policy changes screen, review the proposed variable merge:
   1. monthsOfService is deleted.
   2. Rules are updated to reference tenureMonths.
   3. The description of tenureMonths expanded to:
      • “The number of complete months the employee has been continuously employed. When users mention years of service, convert to months (for example, 2 years = 24 months). This variable captures all references to employment duration, length of service, time at the company, or seniority.”
   4. Under Test results , the previously ambiguous test now produces a definitive VALID result.
2. Select Accept changes.

Step 3: Confirm the fix

1. Return to the Tests tab and choose Validate all tests again.
2. Figure 8 shows the confirmation. The test that previously returned TRANSLATION_AMBIGUOUS now produces a definitive VALID finding and passes.

Figure 8: Tests tab after the ambiguity refinement is applied

Step 4: Report and deploy

1. Choose Generate Fidelity Report and compare the scores with the previous report.
2. When you are satisfied with the results, create a numbered version to deploy.

Note

In both refinement modes, there is no guarantee that the test will pass or the proposed changes don’t introduce regressions. If the test is not passing or other passing tests are now failing, you can discard changes and try again.

Best practices and real-world use cases

The refinement engine works best when you give it a clear signal and a bounded change. The following practices show how, and the use cases show where they pay off.

Best practices

The following practices come from the two most common ways refinement goes sideways. Either you drive it with an ambiguous failure signal, or you let it change more of the policy than you intended.

• Read the translation before you pick a mode. Open the failing finding and look at the variable assignments first. If the right variables have the right values but the result is wrong, that’s a rule issue for Iterative Refinement. If the assignments are wrong, or the finding is TRANSLATION_AMBIGUOUS, that’s a language issue for Ambiguous Variable Refinement. Choosing the mode by symptom rather than by inspecting the translation is the most frequent cause of a refinement that “fixes” the wrong thing.
• Scope the inputs to bound the change. Refine against a source document scoped to a single part of your policy (for example, parental-leave eligibility). Pair it with specific if-then feedback when you need only one rule fixed. Focused inputs produce tighter, more reviewable proposals than pointing the engine at a 40-page handbook with vague guidance like “fix the tenure rule.”
• Trust Test results over the diff. The review screen shows both the changed rules and their effect on every saved test. The Test results panel is the decision-maker: accept when your failing tests flip to passing and your passing tests stay green. A clean-looking diff that regresses a previously passing test is not a fix.
• Compare Fidelity Reports before you version. Refinement optimizes to make tests pass. That optimization can pull a rule away from what your source document actually says. After accepting, compare the accuracy score to the pre-refinement run. A drop signals drift, and is a reason to reject and iterate rather than re-run blindly. Once tests pass and the report holds, create a numbered version to promote the change to production.

Real-world use cases

These patterns apply across regulated industries. In HR eligibility scenarios, the employee handbook updates annually, and Iterative Refinement ingests the new document and proposes rule changes so the HR team can review without touching formal logic. In financial services, overlapping variables such as debtToIncomeRatio and DTI cause translation ambiguity across customer phrasings, and Ambiguous Variable Refinement consolidates them to support deterministic validation. When clinical protocols are revised, healthcare compliance teams upload the new guideline. Iterative Refinement proposes rule updates, and the team validates the proposal before pushing to production. In manufacturing QA, tolerances tighten over time, and the team refines policies iteratively while maintaining an audit trail through versioned snapshots and Fidelity Reports.

Conclusion

Teams shipping generative AI into regulated domains have paid a hidden tax: policy maintenance.

Automatic policy refinement changes the labor equation without changing the authority equation.

The rules-vs-language dichotomy. When the logic is wrong (rules that are too permissive, too strict, or missing), Iterative Refinement proposes formal-logic fixes grounded in your source document and feedback. When the language is wrong (overlapping variables, vague descriptions, inconsistent formats), Ambiguous Variable Refinement proposes precise variable definitions that collapse competing interpretations into one.

The approval gate. In both cases, the system suggests and you decide. Every proposed change surfaces on a review screen that shows the diff and its impact on your saved tests. No change reaches your DRAFT policy, let alone production, without your explicit approval. Automatic in labor. Guided in authority.

Your next step

If you don’t yet have an Automated Reasoning policy with attached tests, start by creating a policy from a source document and adding tests that cover your key scenarios. Refer to the Prerequisites section for links. With a policy and at least one failing test in hand, pick one failing test today and inspect the finding. If the translation is correct but the result is wrong, run Iterative Refinement with a focused source document. If the result is TRANSLATION_AMBIGUOUS, run Ambiguous Variable Refinement.

Accept the proposal, re-test, and compare Fidelity Reports before and after. You will have a tighter policy, verified to up to 99% accuracy on unambiguous translations (see the GA announcement), without writing a single line of formal logic by hand.

Resources

About the 99% accuracy figure

The 99% figure cited in the introduction and conclusion refers to verification accuracy that measures the rate of correct findings (VALID, INVALID, or SATISFIABLE) given an unambiguous translation from natural language to formal logic. It was first published in the GA announcement, Minimize AI hallucinations and deliver up to 99% verification accuracy with Automated Reasoning checks: Now available (AWS News Blog, August 6, 2025).

---

About the authors

Nafi Diallo

Nafi is a Senior Applied Scientist at Amazon Web Services, specialized in AI safety, Automated Reasoning, and guardrails implementation for trustworthy AI solutions. She brings deep experience evaluating and improving the reliability of generative AI and agentic systems on Amazon Bedrock. Nafi also serves as the Regional Lead for North America for the Women in AI and ML (WAIML) organization at AWS, where she supports chapter growth and advances WAIML’s mission across the region.

Ferhat Erata

Ferhat is an Applied Scientist at AWS AI Labs, where he builds neuro-symbolic systems that combine large language models with formal reasoning to make generative and agentic AI verifiable. His work spans AI safety and security, autoformalization, and verified code generation, and he post-trains reasoning models with verifier feedback to align their outputs with formal specifications. Ferhat holds a PhD in Computer Science from Yale University and reviews for NeurIPS, ICML, AAAI, and ACL.

Fernando Galves

Fernando is a Senior Applied AI Architect at AWS, where he helps enterprises in financial services and other regulated industries take generative and agentic AI from experimentation to production. He specializes in AI security, governance, and evaluations, and his work with customers directly informs the security direction of AWS agentic AI services. With more than 25 years of experience across software engineering, cybersecurity, and machine learning, he previously built cloud-based biometric authentication and fraud prevention systems used by top-tier banks and retailers.