第一阶段 RAG 系统：arXiv 论文策展助手

以学习者为中心的生产级 RAG 系统学习之旅

通过动手实现，从零开始构建现代 AI 系统。

掌握最受市场需要的 AI 工程技能：RAG（检索增强生成，Retrieval-Augmented Generation）。

这是一个以学习者为中心的项目。你将构建一个完整的研究助手系统：它可以自动获取学术论文、理解论文内容，并利用先进 RAG 技术回答你的研究问题。

arXiv 论文策展助手将带你使用行业最佳实践构建生产级 RAG 系统。与直接跳到向量搜索的教程不同，我们遵循专业路径：先掌握关键词搜索基础，再通过向量增强形成混合检索。

**🎯 专业化差异：**我们以成功企业的方式构建 RAG 系统——先建立扎实的搜索基础，再用 AI 加强；而不是采用忽视搜索基本功的 AI 优先方法。

完成本课程后，你将拥有自己的 AI 研究助手，以及为任何领域构建生产级 RAG 系统所需的深厚技术能力。

• **第 1 周：**使用 Docker、FastAPI、PostgreSQL、OpenSearch 和 Airflow 搭建完整基础设施
• **第 2 周：**自动化数据管线，从 arXiv 获取并解析学术论文
• **第 3 周：**具备过滤与相关性评分能力的生产级 BM25 关键词搜索
• **第 4 周：**智能分块 + 混合搜索，将关键词与语义理解相结合
• **第 5 周：**完整 RAG 管线，包含本地 LLM、流式响应和 Gradio 界面
• **第 6 周：**使用 Langfuse 追踪进行生产监控，并借助 Redis 缓存优化性能
• **第 7 周：**使用 LangGraph 的 Agentic RAG，以及支持移动访问的 Telegram Bot

---

🏗️ 系统架构演进

第 7 周：Agentic RAG 与 Telegram Bot 集成

完整的第 7 周架构，展示 Telegram Bot 如何与 Agentic RAG 系统集成。

LangGraph Agentic RAG 工作流

详细的 LangGraph 工作流，展示决策节点、文档评分和自适应检索。

第 7 周代码讲解 + 博文：使用 LangGraph 和 Telegram 构建 Agentic RAG

第 7 周的关键创新：

• **智能决策：**Agent 评估并调整检索策略
• **文档评分：**通过语义评估自动判断相关性
• **查询改写：**当检索结果不足时，自适应地优化查询
• **护栏机制：**通过领域外检测防止幻觉
• **移动端访问：**Telegram Bot 支持在任意设备上进行对话式 AI 交互
• **透明性：**完整追踪推理步骤，便于调试与建立信任

---

• Docker Desktop（含 Docker Compose）

• Python 3.12+

• UV 包管理器（安装指南）

• 8GB 以上内存和20GB 以上可用磁盘空间

  1. 克隆并设置

  git clone cd arxiv-paper-curator

  2. 配置环境（重要！）

  cp .env.example .env

  .env 文件包含 OpenSearch、

  arXiv API 以及服务连接所需的全部配置。默认配置可直接使用。

  你需要添加 Jina embeddings 免费 API 密钥和 Langfuse 密钥（请查看博客）。

  3. 安装依赖

  uv sync

  4. 启动所有服务

  docker compose up --build -d

  5. 验证所有服务是否正常运行

  curl http://localhost:8000/api/v1/health

📥 克隆某一周的特定版本：

# 克隆特定周次的代码
git clone --branch <WEEK_TAG> https://github.com/jamwithai/arxiv-paper-curator
cd arxiv-paper-curator
uv sync
docker compose down -v
docker compose up --build -d

# 将 <WEEK_TAG> 替换为：week1.0、week2.0 等。

注意：请在 airflow/simple_auth_manager_passwords.json.generated 中查看 Airflow 用户名和密码。

---

📚 第 1 周：基础设施基础 ✅

**从这里开始！**掌握为现代 RAG 系统提供支撑的基础设施。

• 使用 Docker Compose 完成基础设施设置
• 使用 FastAPI 开发，具备自动文档与健康检查
• PostgreSQL 数据库配置与管理
• OpenSearch 混合搜索引擎设置
• Ollama 本地 LLM 服务配置
• 服务编排与健康监控
• 配备代码质量工具的专业开发环境

基础设施组件：

• FastAPI：支持异步的 REST 端点（端口 8000）

• PostgreSQL 16：论文元数据存储（端口 5432）

• OpenSearch 2.19：带仪表盘的搜索引擎（端口 9200、5601）

• Apache Airflow 3.0：工作流编排（端口 8080）

• Ollama：本地 LLM 服务器（端口 11434）

  启动第 1 周 Notebook

  uv run jupyter notebook notebooks/week1/week1_setup.ipynb

**完成指南：**请按照第 1 周 Notebook进行动手设置和验证。

博客文章：支撑 RAG 系统的基础设施 - 详细讲解与生产实践洞察

---

📚 第 2 周：数据摄取管线 ✅

**在第 1 周基础设施之上：**学习自动获取、处理和存储学术论文。

• 集成 arXiv API，具备限流和重试逻辑
• 使用 Docling 解析科学 PDF 文档
• 使用 Apache Airflow 构建自动化数据摄取管线
• 元数据提取与存储工作流
• 从 API 到数据库的完整论文处理流程

数据管线组件：

• MetadataFetcher：🎯 协调整个管线的主编排器

• ArxivClient：带限流和重试逻辑的论文获取组件

• PDFParserService：由 Docling 驱动的科学文档处理服务

• Airflow DAGs：每日自动摄取论文的工作流

• PostgreSQL Storage：结构化论文元数据与内容存储

  启动第 2 周 Notebook

  uv run jupyter notebook notebooks/week2/week2_arxiv_integration.ipynb

**完成指南：**请按照第 2 周 Notebook进行动手实现和验证。

博客文章：为 RAG 构建数据摄取管线 - arXiv API 集成与 PDF 处理

---

📚 第 3 周：关键词搜索优先——关键基础

**在第 1 至第 2 周的基础上：**实现专业 RAG 系统所依赖的关键词搜索基础。

• 为什么关键词搜索对 RAG 系统至关重要（基础优先方法）
• OpenSearch 索引管理、映射和搜索优化
• BM25 算法，以及高效关键词搜索背后的数学原理
• 使用 Query DSL 构建包含过滤和加权的复杂搜索查询
• 用于衡量相关性与性能的搜索分析
• 真实企业采用的生产模式

搜索基础设施组件：

• OpenSearch Service：src/services/opensearch/ - 专业搜索服务实现

• Search API：src/routers/search.py - 支持 BM25 评分的搜索 API 端点

• 学习材料：notebooks/week3/ - 完整的 OpenSearch 集成指南

• 质量指标：精确率、召回率和相关性评分

  启动第 3 周 Notebook

  uv run jupyter notebook notebooks/week3/week3_opensearch.ipynb

**完成指南：**请按照第 3 周 Notebook进行动手 OpenSearch 设置和 BM25 搜索实现。

博客文章：每个 RAG 系统都需要的搜索基础 - 使用 OpenSearch 完整实现 BM25

---

📚 第 4 周：分块与混合搜索——语义层

**在第 3 周基础上：**加入让搜索真正智能化的语义层。

• 采用基于章节的分块，实现智能文档切分
• 集成 Jina AI 并使用回退策略的生产级 Embeddings
• 运用 RRF 融合，实现关键词 + 语义检索的混合搜索
• 统一 API 设计：单一端点支持多种搜索模式
• 分析不同搜索方案的性能与权衡

混合搜索基础设施组件：

• Text Chunker：src/services/indexing/text_chunker.py - 支持重叠策略的章节感知分块

• Embeddings Service：src/services/embeddings/ - 使用 Jina AI 的生产级嵌入管线

• Hybrid Search API：src/routers/hybrid_search.py - 支持全部模式的统一搜索 API

• 学习材料：notebooks/week4/ - 完整混合搜索实现指南

  启动第 4 周 Notebook

  uv run jupyter notebook notebooks/week4/week4_hybrid_search.ipynb

**完成指南：**请按照第 4 周 Notebook进行动手实现和验证。

博客文章：让混合搜索奏效的分块策略 - 生产级分块与 RRF 融合实现

---

📚 第 5 周：集成 LLM 的完整 RAG 管线

**在第 4 周混合搜索基础上：**加入将搜索转化为智能对话的 LLM 层。

• 通过 Ollama 集成本地 LLM，实现完整数据隐私保护
• 通过减少 80% Prompt 优化性能（速度提升 6 倍）
• 使用 Server-Sent Events 实现流式响应，支持实时输出
• 标准端点和流式端点并存的双 API 设计
• 带高级参数控制的交互式 Gradio 界面

**完整 RAG

Phase 1 RAG Systems: arXiv Paper Curator

A Learner-Focused Journey into Production RAG Systems

Learn to build modern AI systems from the ground up through hands-on implementation

Master the most in-demand AI engineering skills: RAG (Retrieval-Augmented Generation)

This is a learner-focused project where you'll build a complete research assistant system that automatically fetches academic papers, understands their content, and answers your research questions using advanced RAG techniques.

The arXiv Paper Curator will teach you to build a production-grade RAG system using industry best practices. Unlike tutorials that jump straight to vector search, we follow the professional path : master keyword search foundations first, then enhance with vectors for hybrid retrieval.

🎯 The Professional Difference: We build RAG systems the way successful companies do - solid search foundations enhanced with AI, not AI-first approaches that ignore search fundamentals.

By the end of this course, you'll have your own AI research assistant and the deep technical skills to build production RAG systems for any domain.

• Week 1: Complete infrastructure with Docker, FastAPI, PostgreSQL, OpenSearch, and Airflow
• Week 2: Automated data pipeline fetching and parsing academic papers from arXiv
• Week 3: Production BM25 keyword search with filtering and relevance scoring
• Week 4: Intelligent chunking + hybrid search combining keywords with semantic understanding
• Week 5: Complete RAG pipeline with local LLM, streaming responses, and Gradio interface
• Week 6: Production monitoring with Langfuse tracing and Redis caching for optimized performance
• Week 7: Agentic RAG with LangGraph and Telegram Bot for mobile access

---

🏗️ System Architecture Evolution

Week 7: Agentic RAG & Telegram Bot Integration

Complete Week 7 architecture showing Telegram bot integration with the agentic RAG system

LangGraph Agentic RAG Workflow

Detailed LangGraph workflow showing decision nodes, document grading, and adaptive retrieval

Week 7 Code walkthrough + blog: Agentic RAG with LangGraph and Telegram

Key Innovations in Week 7:

• Intelligent Decision-Making : Agents evaluate and adapt retrieval strategies
• Document Grading : Automatic relevance assessment with semantic evaluation
• Query Rewriting : Adaptive query refinement when results are insufficient
• Guardrails : Out-of-domain detection prevents hallucination
• Mobile Access : Telegram bot for conversational AI on any device
• Transparency : Full reasoning step tracking for debugging and trust

---

• Docker Desktop (with Docker Compose)

• Python 3.12+

• UV Package Manager (Install Guide)

• 8GB+ RAM and 20GB+ free disk space

  1. Clone and setup

  git clone cd arxiv-paper-curator

  2. Configure environment (IMPORTANT!)

  cp .env.example .env

  The .env file contains all necessary configuration for OpenSearch,

  arXiv API, and service connections. Defaults work out of the box.

  You need to add Jina embeddings free api key and langfuse keys (check the blogs)

  3. Install dependencies

  uv sync

  4. Start all services

  docker compose up --build -d

  5. Verify everything works

  curl http://localhost:8000/api/v1/health

📥 Clone a specific week's release:

# Clone a specific week's code
git clone --branch <WEEK_TAG> https://github.com/jamwithai/arxiv-paper-curator
cd arxiv-paper-curator
uv sync
docker compose down -v
docker compose up --build -d

# Replace <WEEK_TAG> with: week1.0, week2.0, etc.

NOTE : Check airflow/simple_auth_manager_passwords.json.generated for Airflow username and password

---

📚 Week 1: Infrastructure Foundation ✅

Start here! Master the infrastructure that powers modern RAG systems.

• Complete infrastructure setup with Docker Compose
• FastAPI development with automatic documentation and health checks
• PostgreSQL database configuration and management
• OpenSearch hybrid search engine setup
• Ollama local LLM service configuration
• Service orchestration and health monitoring
• Professional development environment with code quality tools

Infrastructure Components:

• FastAPI : REST endpoints with async support (Port 8000)

• PostgreSQL 16 : Paper metadata storage (Port 5432)

• OpenSearch 2.19 : Search engine with dashboards (Ports 9200, 5601)

• Apache Airflow 3.0 : Workflow orchestration (Port 8080)

• Ollama : Local LLM server (Port 11434)

  Launch the Week 1 notebook

  uv run jupyter notebook notebooks/week1/week1_setup.ipynb

Completion Guide: Follow the Week 1 notebook for hands-on setup and verification steps.

Blog Post: The Infrastructure That Powers RAG Systems - Detailed walkthrough and production insights

---

📚 Week 2: Data Ingestion Pipeline ✅

Building on Week 1 infrastructure: Learn to fetch, process, and store academic papers automatically.

• arXiv API integration with rate limiting and retry logic
• Scientific PDF parsing using Docling
• Automated data ingestion pipelines with Apache Airflow
• Metadata extraction and storage workflows
• Complete paper processing from API to database

Data Pipeline Components:

• MetadataFetcher : 🎯 Main orchestrator coordinating the entire pipeline

• ArxivClient : Rate-limited paper fetching with retry logic

• PDFParserService : Docling-powered scientific document processing

• Airflow DAGs : Automated daily paper ingestion workflows

• PostgreSQL Storage : Structured paper metadata and content

  Launch the Week 2 notebook

  uv run jupyter notebook notebooks/week2/week2_arxiv_integration.ipynb

Completion Guide: Follow the Week 2 notebook for hands-on implementation and verification steps.

Blog Post: Building Data Ingestion Pipelines for RAG - arXiv API integration and PDF processing

---

📚 Week 3: Keyword Search First - The Critical Foundation

Building on Weeks 1-2 foundation: Implement the keyword search foundation that professional RAG systems rely on.

• Why keyword search is essential for RAG systems (foundation first approach)
• OpenSearch index management, mappings, and search optimization
• BM25 algorithm and the math behind effective keyword search
• Query DSL for building complex search queries with filters and boosting
• Search analytics for measuring relevance and performance
• Production patterns used by real companies

Search Infrastructure Components:

• OpenSearch Service : src/services/opensearch/ - Professional search service implementation

• Search API : src/routers/search.py - Search API endpoints with BM25 scoring

• Learning Materials : notebooks/week3/ - Complete OpenSearch integration guide

• Quality Metrics : Precision, recall, and relevance scoring

  Launch the Week 3 notebook

  uv run jupyter notebook notebooks/week3/week3_opensearch.ipynb

Completion Guide: Follow the Week 3 notebook for hands-on OpenSearch setup and BM25 search implementation.

Blog Post: The Search Foundation Every RAG System Needs - Complete BM25 implementation with OpenSearch

---

📚 Week 4: Chunking & Hybrid Search - The Semantic Layer

Building on Week 3 foundation: Add the semantic layer that makes search truly intelligent.

• Section-based chunking with intelligent document segmentation
• Production embeddings with Jina AI integration and fallback strategies
• Hybrid search mastery using RRF fusion for keyword + semantic retrieval
• Unified API design with single endpoint supporting multiple search modes
• Performance analysis and trade-offs between search approaches

Hybrid Search Infrastructure Components:

• Text Chunker : src/services/indexing/text_chunker.py - Section-aware chunking with overlap strategies

• Embeddings Service : src/services/embeddings/ - Production embedding pipeline with Jina AI

• Hybrid Search API : src/routers/hybrid_search.py - Unified search API supporting all modes

• Learning Materials : notebooks/week4/ - Complete hybrid search implementation guide

  Launch the Week 4 notebook

  uv run jupyter notebook notebooks/week4/week4_hybrid_search.ipynb

Completion Guide: Follow the Week 4 notebook for hands-on implementation and verification steps.

Blog Post: The Chunking Strategy That Makes Hybrid Search Work - Production chunking and RRF fusion implementation

---

📚 Week 5: Complete RAG Pipeline with LLM Integration

Building on Week 4 hybrid search: Add the LLM layer that turns search into intelligent conversation.

• Local LLM integration with Ollama for complete data privacy
• Performance optimization with 80% prompt reduction (6x speed improvement)
• Streaming implementation using Server-Sent Events for real-time responses
• Dual API design with standard and streaming endpoints
• Interactive Gradio interface with advanced parameter controls

Complete RAG Infrastructure Components:

• RAG Endpoints : src/routers/ask.py - Dual endpoints (/api/v1/ask + /api/v1/stream)

• Ollama Service : src/services/ollama/ - LLM client with optimized prompts

• System Prompt : src/services/ollama/prompts/rag_system.txt - Optimized for academic papers

• Gradio Interface : src/gradio_app.py - Interactive web UI with streaming support

• Launcher Script : gradio_launcher.py - Easy-launch script (runs on port 7861)

  Launch the Week 5 notebook

  uv run jupyter notebook notebooks/week5/week5_complete_rag_system.ipynb

  Launch Gradio interface

  uv run python gradio_launcher.py

  Open http://localhost:7861

Completion Guide: Follow the Week 5 notebook for hands-on LLM integration and RAG pipeline implementation.

Blog Post: The Complete RAG System - Complete RAG system with local LLM integration and optimization techniques

---

📚 Week 6: Production Monitoring and Caching

Building on Week 5 complete RAG system: Add observability, performance optimization, and production-grade monitoring.

• Langfuse integration for end-to-end RAG pipeline tracing
• Redis caching strategy with intelligent cache keys and TTL management
• Performance monitoring with real-time dashboards for latency and costs
• Production patterns for observability and optimization
• Cost analysis and LLM usage optimization (150-400x speedup with caching)

Production Infrastructure Components:

• Langfuse Service : src/services/langfuse/ - Complete tracing integration with RAG-specific metrics

• Cache Service : src/services/cache/ - Redis client with exact-match caching and graceful fallback

• Updated Endpoints : src/routers/ask.py - Integrated tracing and caching middleware

• Docker Config : docker-compose.yml - Added Redis service and Langfuse local instance

• Learning Materials : notebooks/week6/ - Complete monitoring and caching implementation guide

  Launch the Week 6 notebook

  uv run jupyter notebook notebooks/week6/week6_cache_testing.ipynb

Completion Guide: Follow the Week 6 notebook for hands-on Langfuse tracing and Redis caching implementation.

Blog Post: Production-ready RAG: Monitoring & Caching - Production-ready RAG with monitoring and caching

---

📚 Week 7: Agentic RAG with LangGraph and Telegram Bot

Building on Week 6 production system: Add intelligent reasoning, multi-step decision-making, and Telegram bot integration for mobile-first AI interactions.

• LangGraph workflows for state-based agent orchestration with decision nodes
• Guardrail implementation for query validation and domain boundary detection
• Document grading with semantic relevance evaluation
• Query rewriting for automatic query refinement and better retrieval
• Adaptive retrieval with multi-attempt retrieval and intelligent fallback
• Telegram bot integration with async operations and error handling
• Reasoning transparency by exposing agent decision-making process

Agentic RAG Infrastructure Components:

• Agent Nodes : src/services/agents/nodes/ - Guardrail, retrieve, grade, rewrite, and generate nodes

• Workflow Orchestration : src/services/agents/agentic_rag.py - LangGraph workflow coordination

• Telegram Bot : src/services/telegram/ - Command handlers and message processing

• Agentic Endpoint : src/routers/agentic_ask.py - Agentic RAG API endpoint

• Learning Materials : notebooks/week7/ - Week 7 learning materials and examples

  Launch the Week 7 notebook

  uv run jupyter notebook notebooks/week7/week7_agentic_rag.ipynb

Completion Guide: Follow the Week 7 notebook for hands-on LangGraph agentic RAG and Telegram bot implementation.

Blog Post: Agentic RAG with LangGraph and Telegram - Building intelligent agents with decision-making, adaptive retrieval, and mobile access

---

Setup:

cp .env.example .env
# Edit .env for your environment

Key Variables:

• JINA_API_KEY - Required for Week 4+ (hybrid search with embeddings)
• TELEGRAM__BOT_TOKEN - Required for Week 7 (Telegram bot integration)
• LANGFUSE__PUBLIC_KEY & LANGFUSE__SECRET_KEY - Optional for Week 6 (monitoring)

Complete Configuration: See .env.example for all available options and detailed documentation.

---

🔧 Reference & Development Guide

Service	Purpose	Status
FastAPI	REST API with automatic docs	✅ Ready
PostgreSQL 16	Paper metadata and content storage	✅ Ready
OpenSearch 2.19	Hybrid search engine (BM25 + Vector)	✅ Ready
Apache Airflow 3.0	Workflow automation	✅ Ready
Jina AI	Embedding generation (Week 4)	✅ Ready
Ollama	Local LLM serving (Week 5)	✅ Ready
Redis	High-performance caching (Week 6)	✅ Ready
Langfuse	RAG pipeline observability (Week 6)	✅ Ready

Development Tools: UV, Ruff, MyPy, Pytest, Docker Compose

arxiv-paper-curator/
├── src/                    # Main application code
│   ├── routers/            # API endpoints (search, ask, papers)
│   ├── services/           # Business logic (opensearch, ollama, agents, cache)
│   ├── models/             # Database models (SQLAlchemy)
│   ├── schemas/            # Pydantic validation schemas
│   └── config.py           # Environment configuration
├── notebooks/              # Weekly learning materials (week1-7)
├── airflow/                # Workflow orchestration (DAGs)
├── tests/                  # Test suite
└── compose.yml             # Docker service orchestration

📡 API Endpoints Reference

Endpoint	Method	Description	Week
/health	GET	Service health check	Week 1
/api/v1/papers	GET	List stored papers	Week 2
/api/v1/papers/{id}	GET	Get specific paper	Week 2
/api/v1/search	POST	BM25 keyword search	Week 3
/api/v1/hybrid-search/	POST	Hybrid search (BM25 + Vector)	Week 4

API Documentation: Visit http://localhost:8000/docs for interactive API explorer

Using the Makefile (Recommended)

# View all available commands
make help

# Quick workflow
make start         # Start all services
make health        # Check all services health
make test          # Run tests
make stop          # Stop services

Command	Description
make start	Start all services
make stop	Stop all services
make restart	Restart all services
make status	Show service status
make logs	Show service logs
make health	Check all services health
make setup	Install Python dependencies
make format	Format code
make lint	Lint and type check
make test	Run tests
make test-cov	Run tests with coverage
make clean	Clean up everything

Direct Commands (Alternative)

# If you prefer using commands directly
docker compose up --build -d    # Start services
docker compose ps               # Check status
docker compose logs            # View logs
uv run pytest                 # Run tests

Who	Why
AI/ML Engineers	Learn production RAG architecture beyond tutorials
Software Engineers	Build end-to-end AI applications with best practices
Data Scientists	Implement production AI systems using modern tools

---

Common Issues:

• Services not starting? Wait 2-3 minutes, check docker compose logs
• Port conflicts? Stop other services using ports 8000, 8080, 5432, 9200
• Memory issues? Increase Docker Desktop memory allocation

Get Help:

• Check the comprehensive Week 1 notebook troubleshooting section
• Review service logs: docker compose logs [service-name]
• Complete reset: docker compose down --volumes && docker compose up --build -d

---

This course is completely free! You'll only need minimal costs for optional services:

• Local Development: $0 (everything runs locally)
• Optional Cloud APIs: ~$2-5 for external LLM services (if chosen)

---

🎉 Ready to Start Your AI Engineering Journey?

Begin with the Week 1 setup notebook and build your first production RAG system!

For learners who want to master modern AI engineering

Built with love byShirin Khosravi Jam & Shantanu Ladhwe

---

---

MIT License - see LICENSE file for details.