📦 pathwaycom/serviette：面向 RAG、AI 管道与企业搜索的实时数据云模板
英文标题：📦 pathwaycom/serviette
Pathway 的 Live Data Framework 提供可直接部署的 AI Pipelines 模板，用于构建具备高准确率、可扩展性的 RAG 和 AI 企业搜索应用。模板可与本地文件系统、Google Drive、Sharepoint、S3、Kafka、PostgreSQL 及实时数据 API 持续同步，并支持数百万页文档的实时索引与检索。该项目内置向量检索、混合检索和全文检索能力，减少开发者整合向量数据库、缓存和 API 框架等独立组件的运维负担。开发者可在本地以 Docker 运行，并部署至 GCP、AWS、Azure、Render 或本地环境。
3.6 KB
5.1 KB
7.7 KB
1.4 KB
2.3 KB
927 B
4.3 MB
3.8 MB
6.8 KB
1.2 KB
743.9 KB
中文
Pathway Live Data Framework 的 AI Pipelines 可帮助你快速将 AI 应用投入生产，利用数据源中最最新的知识，实现可大规模运行的高准确率 RAG 和 AI 企业搜索。它提供可直接部署的 LLM（大语言模型）应用模板。你可以在自己的机器上测试这些模板，并将其部署至云端（GCP、AWS、Azure、Render 等）或本地环境。
这些应用可连接并同步（包括所有新增、删除和更新）位于你的文件系统、Google Drive、Sharepoint、S3、Kafka、PostgreSQL、实时数据 API 中的数据源。它们无需依赖需要额外配置的基础设施组件。应用内置数据索引能力，支持向量搜索、混合搜索和全文搜索，全部在内存中完成并带有缓存。
本仓库提供的应用模板可扩展至处理数百万页文档。其中一些针对简洁性进行了优化，另一些则针对卓越准确率进行了优化；请选择最适合你的模板。你可以开箱即用，也可以修改管道中的部分步骤——例如，如需添加新数据源，或将向量索引改为混合索引，只需一行改动。
| 应用（模板） | 描述 |
|---|---|
Question-Answering RAG App |
基础的端到端 RAG 应用。这是一条问答管道，使用你选择的 GPT 模型，为连接至实时数据源（文件、Google Drive、Sharepoint 等）的文档（PDF、DOCX 等）提供查询答案。你还可以试用一个演示 REST 端点。 |
Live Document Indexing (Vector Store / Retriever) |
用于 RAG 的实时文档索引管道，可作为向量存储服务运行。它会对来自已连接数据源（文件、Google Drive、Sharepoint 等）的文档（PDF、DOCX 等）执行实时索引。它可与任意前端配合使用，也可作为 Langchain 或 Llamaindex 应用的检索器后端集成。你还可以试用一个演示 REST 端点。 |
Multimodal RAG pipeline with GPT4o |
使用 GPT-4o 的多模态 RAG，在解析阶段对来自已连接数据源（文件、Google Drive、Sharepoint 等）的 PDF 及其他文档建立索引。它尤其适合从文件夹中的非结构化金融文档（包括图表和表格）提取信息，并会在文档发生变更或有新文档到达时更新结果。 |
Unstructured-to-SQL pipeline + SQL question-answering |
一个 RAG 示例：连接非结构化金融数据源（财务报告 PDF），将数据结构化为 SQL，并加载至 PostgreSQL 表。它还可通过 LLM 将用户针对这些金融文档的自然语言查询翻译为 SQL，并在 PostgreSQL 表上执行查询来回答问题。 |
Adaptive RAG App |
使用 Adaptive RAG 的 RAG 应用。Adaptive RAG 是 Pathway 开发的一项技术，可在保持准确率的同时，将 RAG 的 token 成本最多降低 4 倍。 |
Private RAG App with Mistral and Ollama |
使用 Pathway Live Data Framework、Mistral 和 Ollama 构建的完全私有化（本地）版本 question_answering_rag RAG 管道。 |
Slides AI Search App |
用于检索幻灯片的索引管道。它对 PowerPoint 和 PDF 执行多模态处理，并维护幻灯片的实时索引。 |
Video RAG with TwelveLabs |
面向视频的 RAG 管道。它使用 TwelveLabs Pegasus 将视频转换为丰富的文本描述，并使用 Marengo 多模态嵌入为视频建立索引，因此你可以针对实时连接数据源（文件、Google Drive 等）中的视频提问。 |
这些 AI Pipelines 如何工作？
这些应用可作为 Docker 容器运行，并暴露 HTTP API 供前端连接。为便于快速测试和演示，部分应用模板还包含可选的 Streamlit UI，并通过该 API 连接。
这些应用依赖 Pathway Live Data Framework 来同步数据源和处理 API 请求（Pathway 是一个独立的 Python 库，内置 Rust 引擎）。它们为后端、嵌入、检索和 LLM 技术栈提供了简洁统一的应用逻辑。你无需为生成式 AI 应用整合和维护独立模块：向量数据库（如 Pinecone/Weaviate/Qdrant）+ 缓存（如 Redis）+ API 框架（如 Fast API）。Pathway 默认采用的内置向量索引基于极速 usearch 库，而混合全文索引则使用 Tantivy 库。一切均可开箱即用。
本仓库中的每个应用模板都包含 README.md，其中提供了运行说明。
你还可以在 Pathway 网站找到更多可直接运行的代码模板。
通过多模态 RAG，轻松从 PDF、文档等内容中提取并整理表格和图表数据——实时完成：
（查看 Multimodal RAG pipeline with GPT4o，了解完整管道的实际运行方式。你也可以查看 Unstructured-to-SQL pipeline，这是一个同样适用于非多模态模型的最简示例。）
自动化实时知识挖掘与告警：
（查看 Alerting when answers change on Google Drive 应用示例。）
视频/嵌入内容：使用 Pathway Live Data Framework 构建 LLM 应用简介 - 作者：Jan Chorowski
视频/嵌入内容：用 11 分钟构建一个真实世界的 LLM 应用 - 作者：Pau Labarta Bajo
如需提出反馈或报告 Bug，请在我们的问题追踪器中提交 issue。
我们非常鼓励任何人参与本项目贡献，包括文档、功能、Bug 修复、代码清理、测试或代码审查。如果这是你第一次为 GitHub 项目贡献，请参考入门指南。
如果你希望进行需要更多工作的贡献，只需在 Pathway Discord 服务器（#get-help）中举手，并告知我们你的计划即可！
支持与维护方
English / 原文
The Pathway Live Data Framework's AI Pipelines allow you to quickly put in production AI applications that offer high-accuracy RAG and AI enterprise search at scale using the most up-to-date knowledge available in your data sources. It provides you ready-to-deploy LLM (Large Language Model) App Templates. You can test them on your own machine and deploy on-cloud (GCP, AWS, Azure, Render,...) or on-premises.
The apps connect and sync (all new data additions, deletions, updates) with data sources on your file system, Google Drive, Sharepoint, S3, Kafka, PostgreSQL, real-time data APIs. They come with no infrastructure dependencies that would need a separate setup. They include built-in data indexing enabling vector search, hybrid search, and full-text search - all done in-memory, with cache.
The application templates provided in this repo scale up to millions of pages of documents. Some of them are optimized for simplicity, some are optimized for amazing accuracy. Pick the one that suits you best. You can use it out of the box, or change some steps of the pipeline - for example, if you would like to add a new data source, or change a Vector Index into a Hybrid Index, it's just a one-line change.
| Application (template) | Description |
|---|---|
Question-Answering RAG App |
Basic end-to-end RAG app. A question-answering pipeline that uses the GPT model of choice to provide answers to queries to your documents (PDF, DOCX,...) on a live connected data source (files, Google Drive, Sharepoint,...). You can also try out a demo REST endpoint. |
Live Document Indexing (Vector Store / Retriever) |
A real-time document indexing pipeline for RAG that acts as a vector store service. It performs live indexing on your documents (PDF, DOCX,...) from a connected data source (files, Google Drive, Sharepoint,...). It can be used with any frontend, or integrated as a retriever backend for a Langchain or Llamaindex application. You can also try out a demo REST endpoint. |
Multimodal RAG pipeline with GPT4o |
Multimodal RAG using GPT-4o in the parsing stage to index PDFs and other documents from a connected data source files, Google Drive, Sharepoint,...). It is perfect for extracting information from unstructured financial documents in your folders (including charts and tables), updating results as documents change or new ones arrive. |
Unstructured-to-SQL pipeline + SQL question-answering |
A RAG example which connects to unstructured financial data sources (financial report PDFs), structures the data into SQL, and loads it into a PostgreSQL table. It also answers natural language user queries to these financial documents by translating them into SQL using an LLM and executing the query on the PostgreSQL table. |
Adaptive RAG App |
A RAG application using Adaptive RAG, a technique developed by Pathway to reduce token cost in RAG up to 4x while maintaining accuracy. |
Private RAG App with Mistral and Ollama |
A fully private (local) version of the question_answering_rag RAG pipeline using Pathway Live Data Framework, Mistral, and Ollama. |
Slides AI Search App |
An indexing pipeline for retrieving slides. It performs multi-modal of PowerPoint and PDF and maintains live index of your slides." |
Video RAG with TwelveLabs |
A RAG pipeline over video. It uses TwelveLabs Pegasus to turn videos into rich text descriptions and Marengo multimodal embeddings to index them, so you can ask questions about your videos on a live connected data source (files, Google Drive,...). |
How do these AI Pipelines work?
The apps can be run as Docker containers , and expose an HTTP API to connect the frontend. To allow quick testing and demos, some app templates also include an optional Streamlit UI which connects to this API.
The apps rely on the Pathway Live Data Framework for data source synchronization and for serving API requests (Pathway is a standalone Python library with a Rust engine built into it). They bring you a simple and unified application logic for back-end, embedding, retrieval, LLM tech stack. There is no need to integrate and maintain separate modules for your Gen AI app: Vector Database (e.g. Pinecone/Weaviate/Qdrant) + Cache (e.g. Redis) + API Framework (e.g. Fast API). Pathway's default choice of built-in vector index is based on the lightning-fast usearch library, and hybrid full-text indexes make use of Tantivy library. Everything works out of the box.
Each of the App templates in this repo contains a README.md with instructions on how to run it.
You can also find more ready-to-run code templates on the Pathway website.
Effortlessly extract and organize table and chart data from PDFs, docs, and more with multimodal RAG - in real-time:
(Check out Multimodal RAG pipeline with GPT4o to see the whole pipeline in the works. You may also check out the Unstructured-to-SQL pipeline for a minimal example that works with non-multimodal models as well.)
Automated real-time knowledge mining and alerting:
(Check out the Alerting when answers change on Google Drive app example.)
▶️ An introduction to building LLM apps with Pathway Live Data Framework - by Jan Chorowski
▶️ Let's build a real-world LLM app in 11 minutes - by Pau Labarta Bajo
To provide feedback or report a bug, please raise an issue on our issue tracker.
Anyone who wishes to contribute to this project, whether documentation, features, bug fixes, code cleanup, testing, or code reviews, is very much encouraged to do so. If this is your first contribution to a GitHub project, here is a Get Started Guide.
If you'd like to make a contribution that needs some more work, just raise your hand on the Pathway Discord server (#get-help) and let us know what you are planning!
Supported and maintained by