2026 年 10 月 9 日，我们宣布 TwelveLabs Pegasus 1.5 正式上线 Amazon Bedrock。Pegasus 1.5 是一款视频语言模型，能够将长视频转化为结构化、带时间戳的元数据。你只需描述工作流中关心的关键时刻，模型便会以 JSON 格式返回所有匹配的视频片段。

视频作为一种重要且复杂的非结构化数据形态，目前绝大部分仍处于不可检索、未被充分利用的状态。长期以来，从视频内容中提取结构化且可操作的洞察，通常需要耗费大量人力进行人工复审，或者依赖碎片化的工具链。

在早期的 Pegasus 版本中，你可以指定模型查看某一片段、提出问题并获取解答。现在，Pegasus 1.5 可以分析整段长视频，检索符合用户自定义标准的所有片段，并输出带有时间戳的结构化元数据。此外，新版本还引入了视频切片（Video Segmentation）、基于参考图像的多模态 Prompting（Multimodal Prompting）以及即时处理能力（On-the-fly Processing）。

借助 Amazon Bedrock 上的 Pegasus 1.5，你可以直接通过 Amazon Simple Storage Service (Amazon S3) URI 或 Base64 字符串对视频进行分析，无需预先构建索引流程，同时你的数据始终保留在 AWS 数据安全边界之内。

本文将介绍 Pegasus 1.5 的新特性、该模型在 Amazon Bedrock 上的封装方式以及如何发送首个请求，随后将深入探讨视频切片、结构化 JSON 输出和多模态 Prompting 的具体实践。

Pegasus 1.5 的新特性

与早期版本相比，Pegasus 1.5 带来了以下更新：

• 基于时间的元数据生成（Time-based metadata generation）：你可以定义工作流中关心的片段、事件或特定领域线索，例如剪辑分段、体育动作精彩瞬间、发言人切换或品牌露出。Pegasus 1.5 会将每个匹配的片段以 JSON 格式返回，包含起止时间戳并填充所请求的字段。
• 多模态 Prompting（Multimodal prompting）：片段定义可以将文本描述与视觉参考图像相结合。这使得精准定位特定产品、Logo 或人物，并分析其在画面中的行为动作成为可能。
• 即时处理（On-the-fly processing）：无需任何数据摄取管道或预先索引步骤。视频直接从 Amazon S3 发送，结果直接在响应中返回。
• 处理容量扩展（Extended processing capacity）：Pegasus 1.5 单次请求支持分析长达 2 小时的视频，较 Pegasus 1.2 的 1 小时大幅提升。

Pegasus 1.5 延续并强化了此前让 Pegasus 1.2 成为强大视频理解工具的核心能力，包括：

• 视频到文本生成（Video-to-text generation）：根据视频内容生成详尽的文本描述。
• 细粒度视觉理解（Granular visual comprehension）：分析视频帧内的物体、屏幕文本以及数字内容。
• 时间定位（Temporal grounding）：精确识别视频中特定事件发生的时间戳。
• 多模态理解（Multimodal understanding）：结合视觉、音频与文本信息进行全方位综合分析。
• 多语言支持（Multilingual support）：完整支持英语，部分支持阿拉伯语、中文、法语、德语、意大利语、日语、韩语、葡萄牙语、俄语、西班牙语、泰语及越南语。

Pegasus 1.5 在 Amazon Bedrock 上的工作机制

Pegasus 1.5 接收一段视频以及文本 Prompt 或一组片段定义作为输入，并返回文本或 JSON。你可以在 bedrock-runtime 端点上使用 InvokeModel API 调用它；如果希望以流式方式实时获取生成的响应，可以使用 InvokeModelWithResponseStream API。

配置项	取值
模型 ID	twelvelabs.pegasus-1-5-v1:0
AWS 区域	美国西部（俄勒冈），us-west-2
推理模式	按需（On-demand）
支持的 API	InvokeModel, InvokeModelWithResponseStream

每次请求通过 analysisMode 字段在以下两种分析模式中选择其一运行：

分析模式	analysisMode 取值	你发送的内容	你收到的返回
通用模式（General）	general（默认）	在 inputPrompt 中发送文本 Prompt	在 message 中返回文本答案，若提供了 Schema 则返回 JSON
时间元数据模式（Time-based metadata）	time_based_metadata	在 responseFormat 中定义片段规格	包含起止时间及所需字段的 segments 列表

你可以通过两种方式为模型指定视频输入。S3 存储路径（s3Location）接收对象的 uri 和 bucketOwner 账号 ID，适合长视频；Base64 字符串（base64String）将视频直接内嵌在请求体中，最大支持 25 MB，适合短片段。

若需按应用统计成本，可以基于 Pegasus 1.5 模型创建 Application Inference Profile，为其添加标签，并在调用时代替模型 ID 使用其 Amazon 资源名称（ARN）。

开始使用 Pegasus 1.5

以下部分将指导你快速上手 Pegasus 1.5。

前提条件

在开始之前，请确保具备以下环境与权限：

1. 一个具备 Amazon Bedrock 访问权限的有效 AWS 账户。
2. Python 3.10 或更高版本，并安装了 AWS SDK for Python (Boto3)。
3. 具有 bedrock:InvokeModel 和 bedrock:InvokeModelWithResponseStream 权限的 AWS Identity and Access Management (IAM) 策略。
4. 已在 Amazon Bedrock 控制台 中启用了 TwelveLabs Pegasus 1.5 的模型访问权限。
5. 视频已存储在 Amazon S3 存储桶中，或者准备了可通过内嵌方式发送的简短视频片段。

以下 IAM 策略授予了针对 Pegasus 1.5 的上述两项调用操作权限：

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream" ], "Resource": [ "arn:aws:bedrock:us-west-2::foundation-model/twelvelabs.pegasus-1-5-v1:0" ] } ] }

Boto3 会使用你的常用 AWS 凭证对请求进行签名。你也可以通过在 AWS_BEARER_TOKEN_BEDROCK 环境变量中设置 Amazon Bedrock API Key 进行鉴权。在生产环境中，推荐使用 aws-bedrock-token-generator 工具包基于 IAM 凭据生成短期 Token，因为该 Token 会自动过期。使用 Bearer-Token 鉴权还需要附加 bedrock:CallWithBearerToken 权限。

发送你的第一个请求

安装 Boto3：

下面的示例展示了如何在通用模式（General mode）下分析存储在 Amazon S3 中的视频。本示例使用了全局跨区域推理（Cross Region Inference, CRIS）端点。如需指定美国地理端点，只需将端点 URL 中的“global.”替换为“us.”。

python import boto3 import json from botocore.config import Config

AWS_REGION = "us-west-2" MODEL_ID = "global.twelvelabs.pegasus-1-5-v1:0"

bedrock_runtime = boto3.client( service_name="bedrock-runtime", region_name=AWS_REGION, config=Config( # 为视频切片等长耗时调用保留较长的读取超时时间 read_timeout=900, retries={"max_attempts": 0}, connect_timeout=10, ), )

分析存储在 Amazon S3 中的视频

payload = { "inputPrompt": "Summarize this video and identify the key topics discussed.", "mediaSource": { "s3Location": { "uri": "s3://amzn-s3-demo-pegasus-15/your-video.mp4", "bucketOwner": "your-account-id" } }, "temperature": 0.2, "maxOutputTokens": 4096, "includeMetering": True }

response = bedrock_runtime.invoke_model( modelId=MODEL_ID, body=json.dumps(payload) ) result = json.loads(response["body"].read()) print(result["message"])

接口返回的响应中，答案存储在 message 字段，Token 消耗量分别在 input_tokens 与 output_tokens 中，生成结束原因包含在 stopReason 中。如果 stopReason 为 length，说明回答在完成前达到了 maxOutputTokens 限制，因此在采纳输出结果前请务必检查该状态：

python if result["stopReason"] == "length": print("The answer was cut off. Raise maxOutputTokens and try again.") print("Tokens in:", result["input_tokens"], "out:", result["output_tokens"])

通用模式请求字段说明：

字段	是否必选	说明
inputPrompt	是	关于视频的问题或分析指令
mediaSource.s3Location.uri	两种视频源二选一	视频的 S3 URI
mediaSource.s3Location.bucketOwner	与 uri 配合使用	拥有该存储桶的 AWS 账户 ID
mediaSource.base64String	两种视频源二选一	Base64 编码的视频数据，最大 25 MB
temperature	否	输出随机性，范围 0 到 1，默认值为 0.2
maxOutputTokens	否	最大生成 Token 数，范围 512 到 98,304
responseFormat	否	回答必须遵循的 JSON Schema
analysisMode	否	general（默认）或 time_based_metadata

对于短视频片段，你可以通过内嵌方式直接发送 Base64 编码数据，而无需先上传至 S3：

python import base64

with open("your-video.mp4", "rb") as f: video_b64 = base64.b64encode(f.read()).decode("utf-8")

payload["mediaSource"] = {"base64String": video_b64} response = bedrock_runtime.invoke_model(modelId=MODEL_ID, body=json.dumps(payload)) print(json.loads(response["body"].read())["message"])

注意，上述示例使用了可选的 includeMetering 属性。设置 includeMetering: True 后，Pegasus 1.5 会在非流式响应中额外附带两个计费维度的使用量字段：outputTokenCount 以及 inputVideoSecond（所分析视频的实际秒数）。

流式接收响应

对于交互式应用，使用 InvokeModelWithResponseStream 可以在模型生成答案的同时分块返回结果。每个分块通过 message 携带新生成的一段文本：

python response = bedrock_runtime.invoke_model_with_response_stream( modelId=MODEL_ID, body=json.dumps(payload) ) for event in response["body"]: chunk = json.loads(event["chunk"]["bytes"]) print(chunk.get("message", ""), end="", flush=True) print()

返回结构化 JSON

在通用模式下，你可以利用 JSON Schema 约束输出，便于下游系统直接解析。只需将 responseFormat.type 设置为 json_schema，并在 jsonSchema 字段下指定结构模式。生成的答案将以 JSON 字符串形式在 message 中返回：

python payload = { "inputPrompt": "Analyze this video and return a structured summary.", "mediaSource": { "s3Location": { "uri": "s3://amzn-s3-demo-pegasus-15/your-video.mp4", "bucketOwner": "your-account-id" } }, "temperature": 0, "maxOutputTokens": 2048, "includeMetering": True, "responseFormat": { "type": "json_schema", "jsonSchema": { "type": "object", "properties": { "summary": {"type": "string"}, "key_scenes": {"type": "array", "items": {"type": "string"}}, "setting": {"type": "string"} }, "required": ["summary", "key_scenes", "setting"] } } } response = bedrock_runtime.invoke_model(modelId=MODEL_ID, body=json.dumps(payload)) summary = json.loads(json.loads(response["body"].read())["message"]) print(summary["summary"]) for scene in summary["key_scenes"]: print(" -", scene)

视频切片：将视频转化为结构化数据

基于时间的元数据生成是 Pegasus 1.5 较以往版本突破最大的能力。与传统的文本 Prompt 不同，你只需提交一个或多个片段定义，模型就会返回视频中匹配这些定义的所有片段。

调用时需将 analysisMode 设为 time_based_metadata，并将 responseFormat.type 设为 segment_definitions。片段定义取代了原有的 inputPrompt，因此可省略该字段。每个片段定义包含：

• id：定义的标识名称，会返回在每个匹配的片段结果中。
• description：匹配片段的具体特征描述。
• fields：需要为每个片段提取的字段列表，每个字段包含 name、type（支持 string、boolean、number、integer、array 或 timestamp）以及 description 说明。

以下示例展示了如何识别新闻播报中的各个编辑栏目分段，并返回对应的标题、摘要及核心议题：

python payload = { "analysisMode": "time_based_metadata", "mediaSource": { "s3Location": { "uri": "s3://amzn-s3-demo-pegasus-15/news-broadcast.mp4", "bucketOwner": "your-account-id" } }, "responseFormat": { "type": "segment_definitions", "segmentTimeFormat": "hh:mm:ss", "segmentDefinitions": [ { "id": "editorial_section", "description": "A distinct editorial section of the broadcast, " "such as a new story, topic, or segment of the program.", "fields": [ {"name": "segment_title", "type": "string", "description": "A descriptive title"}, {"name": "summary", "type": "string", "description": "A brief summary of the segment content"}, {"name": "key_topics", "type": "array", "items": {"type": "string"}, "description": "Main topics discussed"} ] } ] }, "temperature": 0, "maxOutputTokens": 32768, "includeMetering": True } response = bedrock_runtime.invoke_model( modelId=MODEL_ID, contentType="application/json", accept="application/json", body=json.dumps(payload) ) result = json.loads(response["body"].read()) for segment in result["segments"]: print(segment["start_sec"], "-", segment["end_sec"], segment["segment_title"]) print(" ", segment["summary"]) print(" topics:", ", ".join(segment["key_topics"]))

On Oct 9, 2026 we announced the availability of TwelveLabs’ Pegasus 1.5 on Amazon Bedrock. Pegasus 1.5 is a video language model that turns long-form video into structured, timestamped metadata. You describe the moments that matter to your workflow, and the model returns every matching segment as JSON.

Video represents a significant and complex category of unstructured data, yet most of it remains unsearchable and underutilized. Extracting structured, actionable insights from that content has historically required labor-intensive manual review or fragmented toolchains.

With earlier versions of Pegasus, you can point the model at a clip, ask a question, and get an answer. Now, Pegasus 1.5 analyzes the entire video, finds every segment that matches user-defined criteria, and returns structured, timestamped metadata. It also introduces video segmentation, multimodal prompting with reference images, and on-the-fly processing.

With Pegasus 1.5 on Amazon Bedrock, you can analyze videos directly from an Amazon Simple Storage Service (Amazon S3) URI or a base64 string with no pre-indexing required, and your data remains within the AWS data boundary.

In this post, we cover what’s new in Pegasus 1.5, how the model is packaged on Amazon Bedrock, and how to send your first request. We then walk through video segmentation, structured JSON output, and multimodal prompting.

What’s new in Pegasus 1.5

Pegasus 1.5 has the following changes compared to previous versions:

• Time-based metadata generation. You define the segments, events, or domain-specific cues that matter to your workflow, such as editorial segments, sports plays, speaker changes, or brand appearances. Pegasus 1.5 returns every matching segment as JSON, with start and end times and the requested fields populated.
• Multimodal prompting. A segment definition can combine a text description with a visual reference. This makes it possible to locate a specific product, logo, or person, along with what they are doing on screen.
• On-the-fly processing. No ingestion pipeline or pre-indexing step is required. Videos are sent directly from Amazon S3 and results are returned in the response.
• Extended processing capacity. Pegasus 1.5 analyzes videos up to two hours long in a single request, up from one hour in Pegasus 1.2.

Pegasus 1.5 retains and enhances the core capabilities that made Pegasus 1.2 a powerful tool for video understanding, including:

• Video-to-text generation : Creates detailed textual descriptions based on video content.
• Granular visual comprehension : Analyzes objects, on-screen text, and numerical content within video frames.
• Temporal grounding : Accurately identifies timestamps of specific events within videos.
• Multimodal understanding : Combines visual, audio, and textual information for comprehensive analysis.
• Multilingual support : Full support for English, with partial support for Arabic, Chinese, French, German, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Thai, and Vietnamese.

How Pegasus 1.5 works on Amazon Bedrock

Pegasus 1.5 takes a video and either a text prompt or a set of segment definitions, and returns text or JSON. You call it on the bedrock-runtime endpoint with the InvokeModel API, or with InvokeModelWithResponseStream when you want the response streamed as it’s generated.

Setting	Value
Model ID	twelvelabs.pegasus-1-5-v1:0
AWS Region	US West (Oregon), us-west-2
Inference	On-demand
APIs	InvokeModel, InvokeModelWithResponseStream

Each request runs in one of two analysis modes, chosen with the analysisMode field:

Analysis mode	analysisMode	You send	You get back
General	general (default)	A text prompt in inputPrompt	A text answer in message, or JSON when you supply a schema
Time-based metadata	time_based_metadata	Segment definitions in responseFormat	A segments list with start and end times and the fields you asked for

You can point the model at video in two ways. An S3 location (s3Location) takes the object’s uri and the bucketOwner account ID, and suits long-form video. A base64 string (base64String) embeds the video in the request body, up to 25 MB, and suits short clips.

To track cost by application, create an application inference profile from the Pegasus 1.5 model, tag it, and call it by its Amazon Resource Name (ARN) in place of the model ID.

Getting started with Pegasus 1.5

The following section outlines how to get started with Pegasus 1.5.

Prerequisites

Before you begin, make sure you have:

1. An active AWS account with Amazon Bedrock access.
2. Python 3.10 or later, with the AWS SDK for Python (Boto3) installed.
3. AWS Identity and Access Management (IAM) permissions for bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream.
4. Model access enabled for TwelveLabs Pegasus 1.5 in the Amazon Bedrock console.
5. Videos stored in an Amazon S3 bucket, or short clips you can send inline.

The following policy grants both invoke actions on Pegasus 1.5.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "bedrock:InvokeModel",
                "bedrock:InvokeModelWithResponseStream"
            ],
            "Resource": [
                "arn:aws:bedrock:us-west-2::foundation-model/twelvelabs.pegasus-1-5-v1:0"
            ]
        }
    ]
}

Boto3 signs requests with your usual AWS credentials. You can also authenticate with an Amazon Bedrock API key by setting it in the AWS_BEARER_TOKEN_BEDROCK environment variable. For production, prefer a short-term key generated from your IAM credentials with the aws-bedrock-token-generator package, because it expires automatically. Bearer-token authentication also requires the bedrock:CallWithBearerToken permission.

Send your first request

Install Boto3:

The following example analyzes a video stored in Amazon S3 in general mode. This example uses the global Cross Region Inference (CRIS) endpoint. To target the US geographic endpoint instead, replace “global.” with “us.” in the endpoint URL.

import boto3
import json
from botocore.config import Config

AWS_REGION = "us-west-2"
MODEL_ID = "global.twelvelabs.pegasus-1-5-v1:0"

bedrock_runtime = boto3.client(
    service_name="bedrock-runtime",
    region_name=AWS_REGION,
    config=Config(
        # keep a longer read_timeout for lengthier invocations such as video segmentation
        read_timeout=900,
        retries={"max_attempts": 0},
        connect_timeout=10,
    ),
)

# Analyze a video stored in Amazon S3
payload = {
    "inputPrompt": "Summarize this video and identify the key topics discussed.",
    "mediaSource": {
        "s3Location": {
            "uri": "s3://amzn-s3-demo-pegasus-15/your-video.mp4",
            "bucketOwner": "your-account-id"
        }
    },
    "temperature": 0.2,
    "maxOutputTokens": 4096,
    "includeMetering": True
}

response = bedrock_runtime.invoke_model(
    modelId=MODEL_ID,
    body=json.dumps(payload)
)
result = json.loads(response["body"].read())
print(result["message"])

The response carries the answer in message, the token counts in input_tokens and output_tokens, and the reason generation ended in stopReason. A stopReason of length means the answer reached maxOutputTokens before it finished, so check it before you rely on the output:

if result["stopReason"] == "length":
    print("The answer was cut off. Raise maxOutputTokens and try again.")
print("Tokens in:", result["input_tokens"], "out:", result["output_tokens"])

Request fields for general mode:

Field	Required	Description
inputPrompt	Yes	The question or instruction about the video
mediaSource.s3Location.uri	One video source is required	S3 URI of the video
mediaSource.s3Location.bucketOwner	With uri	AWS account ID that owns the bucket
mediaSource.base64String	One video source is required	Base64-encoded video, up to 25 MB
temperature	No	Randomness of the output, from 0 to 1. Default 0.2
maxOutputTokens	No	Maximum tokens to generate, from 512 to 98,304
responseFormat	No	A JSON Schema the answer must follow
analysisMode	No	general (default) or time_based_metadata

For a short clip, you can send the video inline instead of from S3:

import base64

with open("your-video.mp4", "rb") as f:
    video_b64 = base64.b64encode(f.read()).decode("utf-8")

payload["mediaSource"] = {"base64String": video_b64}
response = bedrock_runtime.invoke_model(modelId=MODEL_ID, body=json.dumps(payload))
print(json.loads(response["body"].read())["message"])

Note that the preceding example uses the optional includeMetering attribute. includeMetering: True makes Pegasus 1.5 add two billing-usage fields to the non-streaming response: outputTokenCount and inputVideoSecond (the analyzed video’s duration in seconds).

Stream the response

For interactive applications, InvokeModelWithResponseStream returns the answer in chunks as it’s generated. Each chunk carries the next piece of text in message:

response = bedrock_runtime.invoke_model_with_response_stream(
    modelId=MODEL_ID,
    body=json.dumps(payload)
)
for event in response["body"]:
    chunk = json.loads(event["chunk"]["bytes"])
    print(chunk.get("message", ""), end="", flush=True)
print()

Return structured JSON

In general mode, you can constrain the answer to a JSON Schema so downstream code can parse it directly. Set responseFormat.type to json_schema and put the schema under jsonSchema. The answer arrives in message as a JSON string:

payload = {
    "inputPrompt": "Analyze this video and return a structured summary.",
    "mediaSource": {
        "s3Location": {
            "uri": "s3://amzn-s3-demo-pegasus-15/your-video.mp4",
            "bucketOwner": "your-account-id"
        }
    },
    "temperature": 0,
    "maxOutputTokens": 2048,
    "includeMetering": True,
    "responseFormat": {
        "type": "json_schema",
        "jsonSchema": {
            "type": "object",
            "properties": {
                "summary": {"type": "string"},
                "key_scenes": {"type": "array", "items": {"type": "string"}},
                "setting": {"type": "string"}
            },
            "required": ["summary", "key_scenes", "setting"]
        }
    }
}
response = bedrock_runtime.invoke_model(modelId=MODEL_ID, body=json.dumps(payload))
summary = json.loads(json.loads(response["body"].read())["message"])
print(summary["summary"])
for scene in summary["key_scenes"]:
    print(" -", scene)

Video segmentation: Turning video into structured data

Time-based metadata generation is where Pegasus 1.5 differs most from earlier versions. Instead of a prompt, you send one or more segment definitions, and the model returns every segment of the video that matches each one.

Set analysisMode to time_based_metadata and responseFormat.type to segment_definitions. The definitions take the place of inputPrompt, so leave that field out. Each definition has:

• id: a name for the definition, returned on every segment it matches.
• description: what a matching segment looks like.
• fields: the values to extract for each segment, each with a name, a type (string, boolean, number, integer, array, or timestamp), and a description.

The following example identifies each editorial section in a news broadcast and returns a title, summary, and key topics for each one:

payload = {
    "analysisMode": "time_based_metadata",
    "mediaSource": {
        "s3Location": {
            "uri": "s3://amzn-s3-demo-pegasus-15/news-broadcast.mp4",
            "bucketOwner": "your-account-id"
        }
    },
    "responseFormat": {
        "type": "segment_definitions",
        "segmentTimeFormat": "hh:mm:ss",
        "segmentDefinitions": [
            {
                "id": "editorial_section",
                "description": "A distinct editorial section of the broadcast, "
                "such as a new story, topic, or segment of the program.",
                "fields": [
                    {"name": "segment_title", "type": "string",
                     "description": "A descriptive title"},
                    {"name": "summary", "type": "string",
                     "description": "A brief summary of the segment content"},
                    {"name": "key_topics", "type": "array", "items": {"type": "string"},
                     "description": "Main topics discussed"}
                ]
            }
        ]
    },
    "temperature": 0,
    "maxOutputTokens": 32768,
    "includeMetering": True
}
response = bedrock_runtime.invoke_model(
    modelId=MODEL_ID,
    contentType="application/json",
    accept="application/json",
    body=json.dumps(payload)
)
result = json.loads(response["body"].read())
for segment in result["segments"]:
    print(segment["start_sec"], "-", segment["end_sec"], segment["segment_title"])
    print(" ", segment["summary"])
    print(" topics:", ", ".join(segment["key_topics"]))

The previous sample code when tested against a sports clip may produce an output like the following sample.

{
    "segments": [
        {
            "start_sec": "00:04:24",
            "end_sec": "00:05:22",
            "segment_title": "Substitutions and Added Time",
            "summary": "The referee announces three minutes of added time, and both teams make substitutions in preparation for a potential penalty shootout.",
            "key_topics": ["Substitutions", "Added time", "Penalty shootout preparation", "Tactical changes"]
        },
        {
            "start_sec": "00:05:22",
            "end_sec": "00:08:57",
            "segment_title": "Last Gasp Attempts and Final Whistle",
            "summary": "The final minutes of the match see intense action from both sides, with several close calls before the referee blows the final whistle, ending the game in a 3-3 draw.",
            "key_topics": ["Final whistle", "Draw"]
        },
        {
            "start_sec": "00:08:57",
            "end_sec": "00:09:29",
            "segment_title": "Match Conclusion and Penalty Shootout Setup",
            "summary": "The match concludes with a 3-3 draw, leading to a penalty shootout. Players from both teams shake hands and prepare for the decisive moments.",
            "key_topics": ["Match conclusion", "Penalty shootout", "Player handshakes", "Final"]
        }
    ],
    "definition_errors": [],
    "input_tokens": 249408,
    "output_tokens": 802,
    "stopReason": "stop",
    "finish_reason": "stop"
}

Each segment carries start_sec, end_sec, and the fields you defined. segmentTimeFormat controls how the times are written: seconds returns numbers you can do arithmetic on, while hh:mm:ss and hh:mm:ss.fff return readable timestamps as strings. The response also includes a definition_errors list, which is empty when every definition was processed.

A single request can include multiple segment definitions, for example one for editorial sections and another for on-screen brand appearances. Pegasus 1.5 returns the matches for all of them in one segments list, and tags each segment with _definition_id so you can tell them apart. The following example adds a second definition to the previous request and groups the results:

from collections import defaultdict

payload["responseFormat"]["segmentDefinitions"].append({
    "id": "brand_appearance",
    "description": "A shot where a company logo or branded product "
    "is clearly visible on screen.",
    "fields": [
        {"name": "brand", "type": "string", "description": "The brand shown"},
        {"name": "is_prominent", "type": "boolean",
         "description": "Whether the brand is the main focus of the shot"}
    ]
})
response = bedrock_runtime.invoke_model(modelId=MODEL_ID, body=json.dumps(payload))
result = json.loads(response["body"].read())
by_definition = defaultdict(list)
for segment in result["segments"]:
    by_definition[segment["_definition_id"]].append(segment)
for definition_id, segments in by_definition.items():
    print(definition_id, len(segments), "segments")

Because the schema is defined once, the same definitions can be applied across an entire video library, producing consistent metadata for every asset.

Multimodal prompting: Adding visual context to your queries

Pegasus 1.5 introduces multimodal prompting, so you can include reference images alongside your text prompts. This is useful when you want to ask the model about specific objects, people, or scenes that are easier to show than describe.

For example, you can provide a product image and ask the model to identify every moment where that product appears in a video. Or you can provide a reference image of a specific logo and ask the model to detect brand appearances and their timestamps. This capability enhances the precision of video analysis, and opens up use cases such as visual brand monitoring, product placement analysis, and image-guided content search.

Reference images attach differently depending on the mode. In general mode, put them in promptV2.mediaSources, alongside the question in promptV2.inputText. In segmentation mode (analysisMode: time_based_metadata), attach them to the mediaSources of each segment definition inside responseFormat.segmentDefinitions. A definition can then match a visual reference as well as a text description. Each image needs a name, a media_type of image, and exactly one source: either an inline base64String or an s3Location with uri and bucketOwner. A single request can carry up to 30 reference images.

A general-mode request asks the model about an object shown in a reference image. The following sample payload and response provide an example in general mode.

payload = {
    "analysisMode": "general",
    "mediaSource": {"s3Location": {"uri": "s3://...", "bucketOwner": "..."}},
    "promptV2": {
        "inputText": "Does the object in the reference image appear in this video?",
        "mediaSources": [
            {"name": "ref1", "media_type": "image", "base64String": "..."}
        ]
    }
    …
}


{
    "input_tokens": 144,
    "output_tokens": 11,
    "message": "Yes, the green canoe appears in the video.",
    "stopReason": "stop"
}

In segmentation mode, the reference image lives inside the definition, so Pegasus returns only the segments where that product appears. The following sample payload and response provide an example in segmentation mode.

payload = {
    "analysisMode": "time_based_metadata",
    "mediaSource": {"s3Location": {"uri": "s3://...", "bucketOwner": "..."}},
    "responseFormat": {
        "type": "segment_definitions",
        "segmentTimeFormat": "hh:mm:ss",
        "segmentDefinitions": [
            {
                "id": "product_appearance",
                "description": "A shot showing the product in the reference image",
                "fields": [
                    {"name": "what", "type": "string", "description": "what is on screen"}
                ],
                "mediaSources": [
                    {"name": "ref1", "media_type": "image",
                     "s3Location": {"uri": "s3://...", "bucketOwner": "..."}}
                ]
            }
        ]
    }
}


{
    "segments": [
        {"start_sec": "00:00:00", "end_sec": "00:00:02",
         "what": "A family of four in a green canoe, paddling on a calm river surrounded by towering red rock formations."},
        {"start_sec": "00:00:09", "end_sec": "00:00:10",
         "what": "The family back in the canoe, paddling away from the shore, with the children smiling and enjoying the serene surroundings."}
    ],
    "definition_errors": [],
    "input_tokens": 4498,
    "output_tokens": 292,
    "stopReason": "stop",
    "finish_reason": "stop"
}

Monitoring usage

Pegasus 1.5 publishes Amazon CloudWatch metrics in the AWS/Bedrock namespace under its model ID, including Invocations, OutputTokenCount, InvocationLatency, and InvocationClientErrors. Use them to track volume and latency. Pair them with application inference profiles when you need usage broken down by application.

Use cases

Pegasus 1.5 is designed for organizations that manage large volumes of long-form video:

• News and broadcast archives : Teams can define editorial segments, topics, and speaker changes once, then catalog years of footage that was previously searchable only by transcript keywords.
• Sports broadcasters and leagues : Detect plays, scoring events, and other key moments across full-length game footage, speeding up highlight production.
• Streaming applications : Scene-level metadata supports moment-based recommendations, contextual ad placement, and segment-level brand safety review.

In each case, people remain part of the workflow. Their work shifts from finding and tagging every moment by hand to reviewing and refining the metadata generated by the model.

Input requirements

Pegasus 1.5 supports the following video specifications:

• Duration : Up to 2 hours.
• File size : Up to 2 GB from Amazon S3, or up to 25 MB as a base64 string.
• Resolution : 360×360 to 5184×2160.
• Aspect ratio : Between 1:1 and 2.4:1, or their inverses.
• Formats : FFmpeg-supported formats.

Now available

TwelveLabs Pegasus 1.5 is available today on Amazon Bedrock in the US West (Oregon) AWS Region. Check the full Region list for future updates, and visit the Amazon Bedrock pricing page for pricing information. Pegasus 1.5 is billed through AWS Marketplace, so its charges appear on your AWS bill and in AWS Cost Explorer under TwelveLabs rather than Amazon Bedrock.

To get started:

We’re excited to see what you build with TwelveLabs Pegasus 1.5 in Amazon Bedrock.

---

About the authors

Ron Tal

Ron is a Machine Learning Engineer at TwelveLabs, where he leads inference deployment of TwelveLabs models through third-party platforms, including Amazon Bedrock. He has spent more than a decade building machine learning platforms and production inference infrastructure, from founding ML platform teams to serving foundation models at scale.

Shannon Hong

Shannon is a Product Manager at TwelveLabs, where she leads the development of Pegasus, a video language model. She is Research Affiliate of the Oxford Martin School AI Governance Initiative and has conducted research on democratic governance of AI with Open Future Foundation and Creative Commons. She started her career as a Product Manager at Scale AI, where she led technical teams to build data labeling products for autonomous vehicles. Shannon holds an M.B.A from Oxford University, and a B.A. in Development Studies and Data Science at the University of California, Berkeley.

Nate Stecz

Nate is a Senior Partner Development Manager at AWS, where he helps AI ISVs build and execute go-to-market strategies. With 10 years at AWS, Nate focuses on helping AI model providers build, market, and sell their solutions – spanning Amazon Bedrock integrations, AWS Marketplace, and field enablement. In his spare time, Nate enjoys being outdoors. You can connect with or follow Nate on LinkedIn.

Narcisse Zekpa

Narcisse is a Principal Solutions Architect based in Boston. He helps customers in the Northeast U.S. accelerate their business transformation through innovative, and scalable solutions, on the AWS Cloud. He is passionate about enabling organizations to transform their business, using advanced analytics and AI. When Narcisse is not building, he enjoys spending time with his family, traveling, and road racing.

Suheel Farooq

Suheel is a Principal Solutions Architect at AWS, specializing in artificial intelligence, machine learning, and generative AI. He helps Foundation Model Provider customers design, build, modernize, and scale their AI/ML and generative AI workloads on AWS — from bringing foundation models to Amazon Bedrock to optimizing training and inference and building Agentic AI solutions. His experience spans the AWS AI/ML and generative AI portfolio, particularly Amazon Bedrock, Amazon Bedrock AgentCore, and Amazon SageMaker AI. In his free time, Suheel enjoys working out and hiking.

Badri Raghunathan Srikumar

Badri is a Software Development Engineer at Amazon SageMaker AI. He helps bring foundation models from leading model providers to customers by building scalable, secure and observable hosting infrastructure behind them on Amazon SageMaker. Beyond AWS, he is deeply passionate about robotics, especially autonomous vehicles. When not working, Badri likes to go on hikes, swimming and cafe-hopping.