🚀🤖 Crawl4AI：开源 LLM 友好型网络爬虫与抓取工具

Crawl4AI 将网页转换为干净、LLM 就绪的 Markdown，用于 RAG、代理和数据流水线。快速、可控，经 50k+ 星标社区的实战检验。

✨ 查看最新版本 v0.9.1

✨ v0.9.1 新特性：补丁版本，修复了 Docker、浏览器及核心模块共 12 个问题。为 PruningContentFilter 新增 preserve_classes/preserve_tags 白名单，修复 Windows 浏览器崩溃、Docker 认证网关界面、HTTP 超时单位不匹配等问题。发布说明 →

✨ 近期 v0.9.0：Docker API 服务器的重大安全默认版本。认证默认开启，服务器仅在未提供令牌时绑定回环地址，请求体现被视为不受信任的信任边界。发布说明 →

✨ 近期 v0.8.7：安全强化版本。修复了 Docker API 的关键漏洞（RCE、SSRF、认证绕过、文件写入、XSS、硬编码 JWT 密钥），新增 DomainMapper，并发布了抓取、深度爬取和 LLM 相关修复。发布说明 →

✨ 此前的 v0.8.0：崩溃恢复与预取模式！深度爬取崩溃恢复通过 resume_state 和 on_state_change 回调支持长时间运行的爬取任务。新增 prefetch=True 模式，使 URL 发现速度提升 5-10 倍。发布说明 →

✨ 此前的 v0.7.8：稳定性与错误修复版本！修复了 11 个问题，涵盖 Docker API 问题、LLM 提取改进、URL 处理修复和依赖项更新。发布说明 →

🤓 我的个人故事

我父亲让我在 Amstrad 上长大，从此我没停止过构建。读研时我专攻 NLP，并为研究编写爬虫。正是在那时我学会了提取数据的重要性。

2023 年，我需要将网页转为 Markdown。所谓的“开源”选项却要求注册账户、API 令牌并收费 16 美元，效果还不好。我怒由心生，几天内做出了 Crawl4AI，然后它就火了。现在它是 GitHub 上星标最多的爬虫。

我将其开源是为了可用性，任何人都可以无障碍使用。现在我正在构建平台以实现可负担性，让任何人都能运行严肃的爬取任务而不必破产。如果这引起了你的共鸣，欢迎加入，提供反馈，或只是用它去爬取一些精彩的内容。

开发者为什么选择 Crawl4AI

• LLM 就绪输出：智能 Markdown，包含标题、表格、代码、引用提示
• 实际运行快速：异步浏览器池、缓存、最少跳转
• 完全控制：会话、代理、Cookie、用户脚本、钩子
• 自适应智能：学习站点模式，只探索相关内容
• 随地部署：无需密钥，CLI 和 Docker，云友好

1. 安装 Crawl4AI：

# Install the package
pip install -U crawl4ai

# For pre release versions
pip install crawl4ai --pre

# Run post-installation setup
crawl4ai-setup

# Verify your installation
crawl4ai-doctor

如果遇到任何浏览器相关问题，可以手动安装它们：

python -m playwright install --with-deps chromium

2. 用 Python 运行一个简单的网页爬取：

import asyncio
from crawl4ai import *

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://www.nbcnews.com/business",
        )
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())

3. 或者使用新的命令行界面：

# Basic crawl with markdown output
crwl https://www.nbcnews.com/business -o markdown

# Deep crawl with BFS strategy, max 10 pages
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10

# Use LLM extraction with a specific question
crwl https://www.example.com/products -q "Extract all product prices"

🎉 赞助计划现已开放！ 在为 51K+ 开发者提供动力并经历 1 年成长后，Crawl4AI 为初创公司和企业推出专属支持。成为前 50 名创始赞助商，永久铭刻在我们的名人堂中。

Crawl4AI 是 GitHub 上排名第一的开源网页爬虫趋势项目。您的支持可让它保持独立、创新，并对社区免费——同时您也可直接获得高级权益。

🤝 赞助层级

• 🌱 信仰者（$5/月） — 加入数据民主化运动
• 🚀 构建者（$50/月） — 优先支持与早期功能体验
• 💼 成长团队（$500/月） — 每两周同步与优化帮助
• 🏢 数据基础设施合作伙伴（$2000/月） — 全面合作与专属支持
  可定制安排 — 详见SPONSORS.md 获取详情及联系方式

为什么赞助？
无速率限制的 API。无锁定。构建并拥有您的数据流水线，并获得 Crawl4AI 创建者的直接指导。

查看所有层级与权益 →

📝 Markdown 生成

• 🧹 干净 Markdown：生成格式准确、结构清晰的 Markdown。
• 🎯 适配 Markdown：基于启发式过滤去除噪声和无关部分，适合 AI 处理。
• 🔗 引用与参考文献：将页面链接转换为带编号的参考文献列表和清晰的引用。
• 🛠️ 自定义策略：用户可根据特定需求创建自己的 Markdown 生成策略。
• 📚 BM25 算法：使用基于 BM25 的过滤提取核心信息并去除无关内容。

📊 结构化数据提取

• 🤖 LLM 驱动提取：支持所有 LLM（开源和闭源）进行结构化数据提取。
• 🧱 分块策略：实现基于主题、正则表达式、句子级的分块，用于有针对性的内容处理。
• 🌌 余弦相似度：根据用户查询找到相关的内容块，实现语义提取。
• 🔎 基于 CSS 的提取：使用 XPath 和 CSS 选择器快速进行基于模式的数据提取。
• 🔧 模式定义：定义自定义模式，从重复模式中提取结构化 JSON。

🌐 浏览器集成

• 🖥️ 托管浏览器：使用用户自有浏览器并完全控制，避免机器人检测。
• 🔄 远程浏览器控制：连接到 Chrome DevTools Protocol 进行远程大规模数据提取。
• 👤 浏览器剖析器：创建并管理具有已保存认证状态、Cookie 和设置的持久配置文件。
• 🔒 会话管理：保留浏览器状态并复用以进行多步爬取。
• 🧩 代理支持：无缝连接需要认证的代理以安全访问。
• ⚙️ 完全浏览器控制：修改标头、Cookie、用户代理等，实现定制化爬取设置。
• 🌍 多浏览器支持：兼容 Chromium、Firefox 和 WebKit。
• 📐 动态视口调整：自动调整浏览器视口以匹配页面内容，确保完整渲染并捕获所有元素。

🔎 爬取与抓取

• 🖼️ 媒体支持：提取图片、音频、视频以及 srcset 和 picture 等响应式图片格式。
• 🚀 动态爬取：执行 JS 并等待异步或同步加载，以提取动态内容。
• 📸 截图：爬取过程中捕获页面截图，用于调试或分析。
• 📂 原始数据爬取：直接处理原始 HTML（raw:）或本地文件（file://）。
• 🔗 全面链接提取：提取内部链接、外部链接和嵌入的 iframe 内容。
• 🛠️ 可定制钩子：在每一步定义钩子，定制爬取行为（支持字符串和函数 API）。
• 💾 缓存：缓存数据以提高速度并避免重复获取。
• 📄 元数据提取：从网页中检索结构化元数据。
• 📡 IFrame 内容提取：无缝提取嵌入的 iframe 内容。
• 🕵️ 懒加载处理：等待图片完全加载，确保不会因懒加载而遗漏内容。
• 🔄 全页扫描：模拟滚动以加载并捕获所有动态内容，非常适合无限滚动页面。

🚀 部署

• 🐳 Docker 化设置：优化的 Docker 镜像，搭配 FastAPI 服务器，便于部署。
• 🔑 安全认证：内置 JWT 令牌认证，保障 API 安全。
• 🔄 API 网关：一键部署，附带安全令牌认证，适合基于 API 的工作流。
• 🌐 可扩展架构：为大规模生产而设计，优化服务器性能。
• ☁️ 云部署：适用于主流云平台的即用型配置。

🎯 附加功能

• 🕶️ 隐身模式：通过模仿真实用户避免机器人检测。
• 🏷️ 基于标签的内容提取：根据自定义标签、标头或元数据细化爬取。
• 🔗 链接分析：提取并分析所有链接以进行详细的数据探索。
• 🛡️ 错误处理：稳健的错误管理，确保执行顺畅。
• 🔐 CORS 与静态服务：支持基于文件系统的缓存和跨域请求。
• 📖 清晰文档：简化和更新的指南，便于入门和高级使用。
• 🙌 社区认可：感谢贡献者和拉取请求，保持透明。

✨ 在这里试用

✨ 访问我们的文档网站

Crawl4AI 提供灵活的安装选项以适应不同用例。您可以将其安装为 Python 包或使用 Docker。

🐍 使用 pip

选择最适合您需求的安装选项：

用于基本的网页爬取和抓取任务：

pip install crawl4ai
crawl4ai-setup # Setup the browser

默认情况下，这将安装 Crawl4AI 的异步版本，使用 Playwright 进行网页爬取。

👉 注意：当您安装 Crawl4AI 时，crawl4ai-setup 应会自动安装并设置 Playwright。但是，如果您遇到任何与 Playwright 相关的错误，可以使用以下方法之一手动安装：

1. 通过命令行：

2. 如果上述方法不起作用，请尝试使用这个更具体的命令：

   python -m playwright install chromium
   

这种第二种方法在某些情况下已被证明更加可靠。

---

安装同步版本

同步版本已被弃用，将在未来版本中移除。如果您需要使用 Selenium 的同步版本：

pip install crawl4ai[sync]

---

对于计划修改源代码的贡献者：

git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai
pip install -e .                    # Basic installation in editable mode

安装可选功能：

pip install -e ".[torch]"           # With PyTorch features
pip install -e ".[transformer]"     # With Transformer features
pip install -e ".[cosine]"          # With cosine similarity features
pip install -e ".[sync]"            # With synchronous crawling (Selenium)
pip install -e ".[all]"             # Install all optional features

🐳 Docker 部署

🚀 现已可用！ 我们完全重新设计的 Docker 实现来了！这个新的解决方案使部署比以往更高效、更无缝。

新的 Docker 实现包括：

• 实时监控仪表板，提供实时系统指标和浏览器池可见性

• 浏览器池，具有页面预热功能，可加快响应时间

• 交互式演练场，用于测试和生成请求代码

• MCP 集成，可直接连接到 Claude Code 等 AI 工具

• 全面的 API 端点，包括 HTML 提取、截图、PDF 生成和 JavaScript 执行

• 多架构支持，具备自动检测功能（AMD64/ARM64）

• 优化的资源，具有改进的内存管理

  Pull and run the latest release

  docker pull unclecode/crawl4ai:latest docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest

  Visit the monitoring dashboard at http://localhost:11235/dashboard

  Or the playground at http://localhost:11235/playground

运行快速测试（两种 Docker 选项均可）：

import requests

# Submit a crawl job
response = requests.post(
    "http://localhost:11235/crawl",
    json={"urls": ["https://example.com"], "priority": 10}
)
if response.status_code == 200:
    print("Crawl job submitted successfully.")
    
if "results" in response.json():
    results = response.json()["results"]
    print("Crawl job completed. Results:")
    for result in results:
        print(result)
else:
    task_id = response.json()["task_id"]
    print(f"Crawl job submitted. Task ID:: {task_id}")
    result = requests.get(f"http://localhost:11235/task/{task_id}")

更多示例，请参阅我们的 Docker 示例。有关高级配置、监控功能和生产部署，请参阅我们的自托管指南。

---

🔬 高级用法示例 🔬

你可以在 docs/examples 目录中查看项目结构。接下来

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.

Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.

✨ Check out latest update v0.9.1

✨ New in v0.9.1 : Patch release with 12 bug fixes across Docker, browser, and core. Adds preserve_classes/preserve_tags whitelist for PruningContentFilter, fixes Windows browser crash, Docker auth gate UI, HTTP timeout unit mismatch, and more. Release notes →

✨ Recent v0.9.0: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. Release notes →

✨ Recent v0.8.7: Security-hardening release. Fixes critical Docker API vulnerabilities (RCE, SSRF, auth bypass, file write, XSS, hardcoded JWT secret), adds DomainMapper, and ships scraping, deep-crawl, and LLM fixes. Release notes →

✨ Previous v0.8.0: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with resume_state and on_state_change callbacks for long-running crawls. New prefetch=True mode for 5-10x faster URL discovery. Release notes →

✨ Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. Release notes →

🤓 My Personal Story

I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.

In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.

I made it open source for availability , anyone can use it without a gate. Now I’m building the platform for affordability , anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.

Why developers pick Crawl4AI

• LLM ready output , smart Markdown with headings, tables, code, citation hints
• Fast in practice , async browser pool, caching, minimal hops
• Full control , sessions, proxies, cookies, user scripts, hooks
• Adaptive intelligence , learns site patterns, explores only what matters
• Deploy anywhere , zero keys, CLI and Docker, cloud friendly

1. Install Crawl4AI:

# Install the package
pip install -U crawl4ai

# For pre release versions
pip install crawl4ai --pre

# Run post-installation setup
crawl4ai-setup

# Verify your installation
crawl4ai-doctor

If you encounter any browser-related issues, you can install them manually:

python -m playwright install --with-deps chromium

2. Run a simple web crawl with Python:

import asyncio
from crawl4ai import *

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://www.nbcnews.com/business",
        )
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())

3. Or use the new command-line interface:

# Basic crawl with markdown output
crwl https://www.nbcnews.com/business -o markdown

# Deep crawl with BFS strategy, max 10 pages
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10

# Use LLM extraction with a specific question
crwl https://www.example.com/products -q "Extract all product prices"

🎉 Sponsorship Program Now Open! After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support for startups and enterprises. Be among the first 50 Founding Sponsors for permanent recognition in our Hall of Fame.

Crawl4AI is the #1 trending open-source web crawler on GitHub. Your support keeps it independent, innovative, and free for the community — while giving you direct access to premium benefits.

🤝 Sponsorship Tiers

• 🌱 Believer ($5/mo) — Join the movement for data democratization
• 🚀 Builder ($50/mo) — Priority support & early access to features
• 💼 Growing Team ($500/mo) — Bi-weekly syncs & optimization help
• 🏢 Data Infrastructure Partner ($2000/mo) — Full partnership with dedicated support
  Custom arrangements available - seeSPONSORS.md for details & contact

Why sponsor?
No rate-limited APIs. No lock-in. Build and own your data pipeline with direct guidance from the creator of Crawl4AI.

See All Tiers & Benefits →

📝 Markdown Generation

• 🧹 Clean Markdown : Generates clean, structured Markdown with accurate formatting.
• 🎯 Fit Markdown : Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing.
• 🔗 Citations and References : Converts page links into a numbered reference list with clean citations.
• 🛠️ Custom Strategies : Users can create their own Markdown generation strategies tailored to specific needs.
• 📚 BM25 Algorithm : Employs BM25-based filtering for extracting core information and removing irrelevant content.

📊 Structured Data Extraction

• 🤖 LLM-Driven Extraction : Supports all LLMs (open-source and proprietary) for structured data extraction.
• 🧱 Chunking Strategies : Implements chunking (topic-based, regex, sentence-level) for targeted content processing.
• 🌌 Cosine Similarity : Find relevant content chunks based on user queries for semantic extraction.
• 🔎 CSS-Based Extraction : Fast schema-based data extraction using XPath and CSS selectors.
• 🔧 Schema Definition : Define custom schemas for extracting structured JSON from repetitive patterns.

🌐 Browser Integration

• 🖥️ Managed Browser : Use user-owned browsers with full control, avoiding bot detection.
• 🔄 Remote Browser Control : Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.
• 👤 Browser Profiler : Create and manage persistent profiles with saved authentication states, cookies, and settings.
• 🔒 Session Management : Preserve browser states and reuse them for multi-step crawling.
• 🧩 Proxy Support : Seamlessly connect to proxies with authentication for secure access.
• ⚙️ Full Browser Control : Modify headers, cookies, user agents, and more for tailored crawling setups.
• 🌍 Multi-Browser Support : Compatible with Chromium, Firefox, and WebKit.
• 📐 Dynamic Viewport Adjustment : Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements.

🔎 Crawling & Scraping

• 🖼️ Media Support : Extract images, audio, videos, and responsive image formats like srcset and picture.
• 🚀 Dynamic Crawling : Execute JS and wait for async or sync for dynamic content extraction.
• 📸 Screenshots : Capture page screenshots during crawling for debugging or analysis.
• 📂 Raw Data Crawling : Directly process raw HTML (raw:) or local files (file://).
• 🔗 Comprehensive Link Extraction : Extracts internal, external links, and embedded iframe content.
• 🛠️ Customizable Hooks : Define hooks at every step to customize crawling behavior (supports both string and function-based APIs).
• 💾 Caching : Cache data for improved speed and to avoid redundant fetches.
• 📄 Metadata Extraction : Retrieve structured metadata from web pages.
• 📡 IFrame Content Extraction : Seamless extraction from embedded iframe content.
• 🕵️ Lazy Load Handling : Waits for images to fully load, ensuring no content is missed due to lazy loading.
• 🔄 Full-Page Scanning : Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages.

🚀 Deployment

• 🐳 Dockerized Setup : Optimized Docker image with FastAPI server for easy deployment.
• 🔑 Secure Authentication : Built-in JWT token authentication for API security.
• 🔄 API Gateway : One-click deployment with secure token authentication for API-based workflows.
• 🌐 Scalable Architecture : Designed for mass-scale production and optimized server performance.
• ☁️ Cloud Deployment : Ready-to-deploy configurations for major cloud platforms.

🎯 Additional Features

• 🕶️ Stealth Mode : Avoid bot detection by mimicking real users.
• 🏷️ Tag-Based Content Extraction : Refine crawling based on custom tags, headers, or metadata.
• 🔗 Link Analysis : Extract and analyze all links for detailed data exploration.
• 🛡️ Error Handling : Robust error management for seamless execution.
• 🔐 CORS & Static Serving: Supports filesystem-based caching and cross-origin requests.
• 📖 Clear Documentation : Simplified and updated guides for onboarding and advanced usage.
• 🙌 Community Recognition : Acknowledges contributors and pull requests for transparency.

✨ Play around with this

✨ Visit our Documentation Website

Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.

🐍 Using pip

Choose the installation option that best fits your needs:

For basic web crawling and scraping tasks:

pip install crawl4ai
crawl4ai-setup # Setup the browser

By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.

👉 Note : When you install Crawl4AI, the crawl4ai-setup should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:

1. Through the command line:

2. If the above doesn't work, try this more specific command:

   python -m playwright install chromium
   

This second method has proven to be more reliable in some cases.

---

Installation with Synchronous Version

The sync version is deprecated and will be removed in future versions. If you need the synchronous version using Selenium:

pip install crawl4ai[sync]

---

For contributors who plan to modify the source code:

git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai
pip install -e .                    # Basic installation in editable mode

Install optional features:

pip install -e ".[torch]"           # With PyTorch features
pip install -e ".[transformer]"     # With Transformer features
pip install -e ".[cosine]"          # With cosine similarity features
pip install -e ".[sync]"            # With synchronous crawling (Selenium)
pip install -e ".[all]"             # Install all optional features

🐳 Docker Deployment

🚀 Now Available! Our completely redesigned Docker implementation is here! This new solution makes deployment more efficient and seamless than ever.

The new Docker implementation includes:

• Real-time Monitoring Dashboard with live system metrics and browser pool visibility

• Browser pooling with page pre-warming for faster response times

• Interactive playground to test and generate request code

• MCP integration for direct connection to AI tools like Claude Code

• Comprehensive API endpoints including HTML extraction, screenshots, PDF generation, and JavaScript execution

• Multi-architecture support with automatic detection (AMD64/ARM64)

• Optimized resources with improved memory management

  Pull and run the latest release

  docker pull unclecode/crawl4ai:latest docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest

  Visit the monitoring dashboard at http://localhost:11235/dashboard

  Or the playground at http://localhost:11235/playground

Run a quick test (works for both Docker options):

import requests

# Submit a crawl job
response = requests.post(
    "http://localhost:11235/crawl",
    json={"urls": ["https://example.com"], "priority": 10}
)
if response.status_code == 200:
    print("Crawl job submitted successfully.")
    
if "results" in response.json():
    results = response.json()["results"]
    print("Crawl job completed. Results:")
    for result in results:
        print(result)
else:
    task_id = response.json()["task_id"]
    print(f"Crawl job submitted. Task ID:: {task_id}")
    result = requests.get(f"http://localhost:11235/task/{task_id}")

For more examples, see our Docker Examples. For advanced configuration, monitoring features, and production deployment, see our Self-Hosting Guide.

---

🔬 Advanced Usage Examples 🔬

You can check the project structure in the directory docs/examples. Over there, you can find a variety of examples; here, some popular examples are shared.

📝 Heuristic Markdown Generation with Clean and Fit Markdown

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilter, BM25ContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

async def main():
    browser_config = BrowserConfig(
        headless=True,  
        verbose=True,
    )
    run_config = CrawlerRunConfig(
        cache_mode=CacheMode.ENABLED,
        markdown_generator=DefaultMarkdownGenerator(
            content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed", min_word_threshold=0)
        ),
        # markdown_generator=DefaultMarkdownGenerator(
        #     content_filter=BM25ContentFilter(user_query="WHEN_WE_FOCUS_BASED_ON_A_USER_QUERY", bm25_threshold=1.0)
        # ),
    )
    
    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://docs.micronaut.io/4.9.9/guide/",
            config=run_config
        )
        print(len(result.markdown.raw_markdown))
        print(len(result.markdown.fit_markdown))

if __name__ == "__main__":
    asyncio.run(main())

🖥️ Executing JavaScript & Extract Structured Data without LLMs

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy
import json

async def main():
    schema = {
    "name": "KidoCode Courses",
    "baseSelector": "section.charge-methodology .w-tab-content > div",
    "fields": [
        {
            "name": "section_title",
            "selector": "h3.heading-50",
            "type": "text",
        },
        {
            "name": "section_description",
            "selector": ".charge-content",
            "type": "text",
        },
        {
            "name": "course_name",
            "selector": ".text-block-93",
            "type": "text",
        },
        {
            "name": "course_description",
            "selector": ".course-content-text",
            "type": "text",
        },
        {
            "name": "course_icon",
            "selector": ".image-92",
            "type": "attribute",
            "attribute": "src"
        }
    ]
}

    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)

    browser_config = BrowserConfig(
        headless=False,
        verbose=True
    )
    run_config = CrawlerRunConfig(
        extraction_strategy=extraction_strategy,
        js_code=["""(async () => {const tabs = document.querySelectorAll("section.charge-methodology .tabs-menu-3 > div");for(let tab of tabs) {tab.scrollIntoView();tab.click();await new Promise(r => setTimeout(r, 500));}})();"""],
        cache_mode=CacheMode.BYPASS
    )
        
    async with AsyncWebCrawler(config=browser_config) as crawler:
        
        result = await crawler.arun(
            url="https://www.kidocode.com/degrees/technology",
            config=run_config
        )

        companies = json.loads(result.extracted_content)
        print(f"Successfully extracted {len(companies)} companies")
        print(json.dumps(companies[0], indent=2))


if __name__ == "__main__":
    asyncio.run(main())

📚 Extracting Structured Data with LLMs

import os
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from crawl4ai import LLMExtractionStrategy
from pydantic import BaseModel, Field

class OpenAIModelFee(BaseModel):
    model_name: str = Field(..., description="Name of the OpenAI model.")
    input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
    output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")

async def main():
    browser_config = BrowserConfig(verbose=True)
    run_config = CrawlerRunConfig(
        word_count_threshold=1,
        extraction_strategy=LLMExtractionStrategy(
            # Here you can use any provider that Litellm library supports, for instance: ollama/qwen2
            # provider="ollama/qwen2", api_token="no-token", 
            llm_config = LLMConfig(provider="openai/gpt-4o", api_token=os.getenv('OPENAI_API_KEY')), 
            schema=OpenAIModelFee.schema(),
            extraction_type="schema",
            instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens. 
            Do not miss any models in the entire content. One extracted model JSON format should look like this: 
            {"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}."""
        ),            
        cache_mode=CacheMode.BYPASS,
    )
    
    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url='https://openai.com/api/pricing/',
            config=run_config
        )
        print(result.extracted_content)

if __name__ == "__main__":
    asyncio.run(main())

🤖 Using Your own Browser with Custom User Profile

import os, sys
from pathlib import Path
import asyncio, time
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def test_news_crawl():
    # Create a persistent user data directory
    user_data_dir = os.path.join(Path.home(), ".crawl4ai", "browser_profile")
    os.makedirs(user_data_dir, exist_ok=True)

    browser_config = BrowserConfig(
        verbose=True,
        headless=True,
        user_data_dir=user_data_dir,
        use_persistent_context=True,
    )
    run_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS
    )
    
    async with AsyncWebCrawler(config=browser_config) as crawler:
        url = "ADDRESS_OF_A_CHALLENGING_WEBSITE"
        
        result = await crawler.arun(
            url,
            config=run_config,
            magic=True,
        )
        
        print(f"Successfully crawled {url}")
        print(f"Content length: {len(result.markdown)}")

---

Version 0.9.1 Release Highlights - Bug Fixes & PruningContentFilter Whitelist

A patch release with 12 bug fixes and one new feature. The new preserve_classes / preserve_tags parameters for PruningContentFilter let you whitelist CSS classes or HTML tags that should never be pruned — useful for protecting short metadata elements like author names and timestamps.

Bug fixes span Docker (auth gate UI, supervisord/redis dirs, FastAPI compatibility, redis auth), browser (Windows channel crash, context snapshot leak), core (HTTP timeout unit mismatch, best-first ordering), and extraction (html2text table attributes).

Full v0.9.1 Release Notes →

Version 0.9.0 Release Highlights - Secure-by-Default Docker Server

A major, secure-by-default release of the Docker API server. The out-of-the-box deployment is hardened with defense in depth: authentication is on by default, the server binds loopback unless you give it a token, and the network request body is treated as an untrusted trust boundary.

Migration Guide → · Full v0.9.0 Release Notes →

Version 0.8.7 Release Highlights - Security Hardening, DomainMapper & Community Fixes

A security-hardening release. Fixes critical Docker API vulnerabilities (AST sandbox escape RCE, hook sandbox RCE, hardcoded JWT secret, SSRF on webhook and crawl endpoints, arbitrary file write, monitor auth bypass, stored XSS, and unauthenticated JS execution), adds the DomainMapper feature, and ships a batch of scraping, deep-crawl, and LLM fixes. If you self-host the Docker API, upgrade immediately.

Full v0.8.7 Release Notes →

Version 0.8.6 - Security Hotfix: litellm Supply Chain Fix

Replaced litellm dependency with unclecode-litellm due to a PyPI supply chain compromise affecting the original package. If you're on v0.8.5 or earlier, upgrade immediately.

Version 0.8.5 Release Highlights - Anti-Bot Detection, Shadow DOM & 60+ Bug Fixes

Our biggest release since v0.8.0. Anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes.

• 🛡️ Anti-Bot Detection & Proxy Escalation:

  • 3-tier detection: known vendors, generic block indicators, structural integrity checks
  • Automatic retry with proxy chain and fallback fetch function

  from crawl4ai import CrawlerRunConfig from crawl4ai.async_configs import ProxyConfig

  config = CrawlerRunConfig( proxy_config=[ProxyConfig.DIRECT, ProxyConfig(server="http://my-proxy:8080")], max_retries=2, fallback_fetch_function=my_web_unlocker, )

• 🌑 Shadow DOM Flattening :

  • Extract content hidden inside shadow DOM components

  config = CrawlerRunConfig(flatten_shadow_dom=True)

• 🛑 Deep Crawl Cancellation :

  • Stop long crawls gracefully with cancel() or should_cancel callback
  • Works with BFS, DFS, and BestFirst strategies

• ⚙️ Config Defaults API :

  • set_defaults() / get_defaults() / reset_defaults() on BrowserConfig and CrawlerRunConfig

• 🔒 Critical Security Fixes :

  • RCE via deserialization in Docker /crawl endpoint — removed eval(), added allowlist
  • Redis CVE-2025-49844 (CVSS 10.0) — upgraded to 7.2.7

• 60+ Bug Fixes across browser management, proxy, deep crawling, extraction, CLI, and Docker

Full v0.8.5 Release Notes →

Version 0.8.0 Release Highlights - Crash Recovery & Prefetch Mode

This release introduces crash recovery for deep crawls, a new prefetch mode for fast URL discovery, and critical security fixes for Docker deployments.

• 🔄 Deep Crawl Crash Recovery :

  • on_state_change callback fires after each URL for real-time state persistence
  • resume_state parameter to continue from a saved checkpoint
  • JSON-serializable state for Redis/database storage
  • Works with BFS, DFS, and Best-First strategies

  from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

  strategy = BFSDeepCrawlStrategy( max_depth=3, resume_state=saved_state, # Continue from checkpoint on_state_change=save_to_redis, # Called after each URL )

• ⚡ Prefetch Mode for Fast URL Discovery :

  • prefetch=True skips markdown, extraction, and media processing
  • 5-10x faster than full processing
  • Perfect for two-phase crawling: discover first, process selectively

  config = CrawlerRunConfig(prefetch=True) result = await crawler.arun("https://example.com", config=config)

  Returns HTML and links only - no markdown generation

• 🔒 Security Fixes (Docker API) :

  • Hooks disabled by default (CRAWL4AI_HOOKS_ENABLED=false)
  • file:// URLs blocked on API endpoints to prevent LFI
  • __import__ removed from hook execution sandbox

Full v0.8.0 Release Notes →

Version 0.7.8 Release Highlights - Stability & Bug Fix Release

This release focuses on stability with 11 bug fixes addressing issues reported by the community. No new features, but significant improvements to reliability.

Full v0.7.8 Release Notes →

Version 0.7.7 Release Highlights - The Self-Hosting & Monitoring Update

• 📊 Real-time Monitoring Dashboard : Interactive web UI with live system metrics and browser pool visibility

  # Access the monitoring dashboard
  # Visit: http://localhost:11235/dashboard
  
  # Real-time metrics include:
  # - System health (CPU, memory, network, uptime)
  # - Active and completed request tracking
  # - Browser pool management (permanent/hot/cold)
  # - Janitor cleanup events
  # - Error monitoring with full context
  

• 🔌 Comprehensive Monitor API : Complete REST API for programmatic access to all monitoring data

  import httpx
  
  async with httpx.AsyncClient() as client:
      # System health
      health = await client.get("http://localhost:11235/monitor/health")
  
      # Request tracking
      requests = await client.get("http://localhost:11235/monitor/requests")
  
      # Browser pool status
      browsers = await client.get("http://localhost:11235/monitor/browsers")
  
      # Endpoint statistics
      stats = await client.get("http://localhost:11235/monitor/endpoints/stats")
  

• ⚡ WebSocket Streaming : Real-time updates every 2 seconds for custom dashboards

• 🔥 Smart Browser Pool : 3-tier architecture (permanent/hot/cold) with automatic promotion and cleanup

• 🧹 Janitor System : Automatic resource management with event logging

• 🎮 Control Actions : Manual browser management (kill, restart, cleanup) via API

• 📈 Production Metrics : 6 critical metrics for operational excellence with Prometheus integration

• 🐛 Critical Bug Fixes :

  • Fixed async LLM extraction blocking issue (#1055)
  • Enhanced DFS deep crawl strategy (#1607)
  • Fixed sitemap parsing in AsyncUrlSeeder (#1598)
  • Resolved browser viewport configuration (#1495)
  • Fixed CDP timing with exponential backoff (#1528)
  • Security update for pyOpenSSL (>=25.3.0)

Full v0.7.7 Release Notes →

Version 0.7.5 Release Highlights - The Docker Hooks & Security Update

• 🔧 Docker Hooks System : Complete pipeline customization with user-provided Python functions at 8 key points

• ✨ Function-Based Hooks API (NEW) : Write hooks as regular Python functions with full IDE support:

  from crawl4ai import hooks_to_string
  from crawl4ai.docker_client import Crawl4aiDockerClient
  
  # Define hooks as regular Python functions
  async def on_page_context_created(page, context, **kwargs):
      """Block images to speed up crawling"""
      await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
      await page.set_viewport_size({"width": 1920, "height": 1080})
      return page
  
  async def before_goto(page, context, url, **kwargs):
      """Add custom headers"""
      await page.set_extra_http_headers({'X-Crawl4AI': 'v0.7.5'})
      return page
  
  # Option 1: Use hooks_to_string() utility for REST API
  hooks_code = hooks_to_string({
      "on_page_context_created": on_page_context_created,
      "before_goto": before_goto
  })
  
  # Option 2: Docker client with automatic conversion (Recommended)
  client = Crawl4aiDockerClient(base_url="http://localhost:11235")
  results = await client.crawl(
      urls=["https://httpbin.org/html"],
      hooks={
          "on_page_context_created": on_page_context_created,
          "before_goto": before_goto
      }
  )
  # ✓ Full IDE support, type checking, and reusability!
  

• 🤖 Enhanced LLM Integration : Custom providers with temperature control and base_url configuration

• 🔒 HTTPS Preservation : Secure internal link handling with preserve_https_for_internal_links=True

• 🐍 Python 3.10+ Support : Modern language features and enhanced performance

• 🛠️ Bug Fixes : Resolved multiple community-reported issues including URL processing, JWT authentication, and proxy configuration

Full v0.7.5 Release Notes →

Version 0.7.4 Release Highlights - The Intelligent Table Extraction & Performance Update

• 🚀 LLMTableExtraction : Revolutionary table extraction with intelligent chunking for massive tables:

  from crawl4ai import LLMTableExtraction, LLMConfig
  
  # Configure intelligent table extraction
  table_strategy = LLMTableExtraction(
      llm_config=LLMConfig(provider="openai/gpt-4.1-mini"),
      enable_chunking=True,           # Handle massive tables
      chunk_token_threshold=5000,     # Smart chunking threshold
      overlap_threshold=100,          # Maintain context between chunks
      extraction_type="structured"    # Get structured data output
  )
  
  config = CrawlerRunConfig(table_extraction_strategy=table_strategy)
  result = await crawler.arun("https://complex-tables-site.com", config=config)
  
  # Tables are automatically chunked, processed, and merged
  for table in result.tables:
      print(f"Extracted table: {len(table['data'])} rows")
  

• ⚡ Dispatcher Bug Fix : Fixed sequential processing bottleneck in arun_many for fast-completing tasks

• 🧹 Memory Management Refactor : Consolidated memory utilities into main utils module for cleaner architecture

• 🔧 Browser Manager Fixes : Resolved race conditions in concurrent page creation with thread-safe locking

• 🔗 Advanced URL Processing : Better handling of raw:// URLs and base tag link resolution

• 🛡️ Enhanced Proxy Support : Flexible proxy configuration supporting both dict and string formats

Full v0.7.4 Release Notes →

Version 0.7.3 Release Highlights - The Multi-Config Intelligence Update

from crawl4ai import CrawlerRunConfig, MatchMode, CacheMode
  
  configs = [
      # Documentation sites - aggressive caching
      CrawlerRunConfig(
          url_matcher=["*docs*", "*documentation*"],
          cache_mode=CacheMode.WRITE_ONLY,
          markdown_generator_options={"include_links": True}
      ),
      
      # News/blog sites - fresh content
      CrawlerRunConfig(
          url_matcher=lambda url: 'blog' in url or 'news' in url,
          cache_mode=CacheMode.BYPASS
      ),
      
      # Fallback for everything else
      CrawlerRunConfig()
  ]
  
  results = await crawler.arun_many(urls, config=configs)
  # Each URL gets the perfect configuration automatically

• 🧠 Memory Monitoring : Track and optimize memory usage during crawling:

  from crawl4ai.memory_utils import MemoryMonitor
  
  monitor = MemoryMonitor()
  monitor.start_monitoring()
  
  results = await crawler.arun_many(large_url_list)
  
  report = monitor.get_report()
  print(f"Peak memory: {report['peak_mb']:.1f} MB")
  print(f"Efficiency: {report['efficiency']:.1f}%")
  # Get optimization recommendations
  

• 📊 Enhanced Table Extraction : Direct DataFrame conversion from web tables:

  result = await crawler.arun("https://site-with-tables.com")
  
  # New way - direct table access
  if result.tables:
      import pandas as pd
      for table in result.tables:
          df = pd.DataFrame(table['data'])
          print(f"Table: {df.shape[0]} rows × {df.shape[1]} columns")
  

• 💰 GitHub Sponsors : 4-tier sponsorship system for project sustainability

• 🐳 Docker LLM Flexibility : Configure providers via environment variables

Full v0.7.3 Release Notes →

Version 0.7.0 Release Highlights - The Adaptive Intelligence Update

• 🧠 Adaptive Crawling : Your crawler now learns and adapts to website patterns automatically:

  config = AdaptiveConfig(
      confidence_threshold=0.7, # Min confidence to stop crawling
      max_depth=5, # Maximum crawl depth
      max_pages=20, # Maximum number of pages to crawl
      strategy="statistical"
  )
  
  async with AsyncWebCrawler() as crawler:
      adaptive_crawler = AdaptiveCrawler(crawler, config)
      state = await adaptive_crawler.digest(
          start_url="https://news.example.com",
          query="latest news content"
      )
  # Crawler learns patterns and improves extraction over time
  

• 🌊 Virtual Scroll Support : Complete content extraction from infinite scroll pages:

  scroll_config = VirtualScrollConfig(
      container_selector="[data-testid='feed']",
      scroll_count=20,
      scroll_by="container_height",
      wait_after_scroll=1.0
  )
  
  result = await crawler.arun(url, config=CrawlerRunConfig(
      virtual_scroll_config=scroll_config
  ))
  

• 🔗 Intelligent Link Analysis : 3-layer scoring system for smart link prioritization:

  link_config = LinkPreviewConfig(
      query="machine learning tutorials",
      score_threshold=0.3,
      concurrent_requests=10
  )
  
  result = await crawler.arun(url, config=CrawlerRunConfig(
      link_preview_config=link_config,
      score_links=True
  ))
  # Links ranked by relevance and quality
  

• 🎣 Async URL Seeder : Discover thousands of URLs in seconds:

  seeder = AsyncUrlSeeder(SeedingConfig(
      source="sitemap+cc",
      pattern="*/blog/*",
      query="python tutorials",
      score_threshold=0.4
  ))
  
  urls = await seeder.discover("https://example.com")
  

• ⚡ Performance Boost : Up to 3x faster with optimized resource handling and memory efficiency

Read the full details in our 0.7.0 Release Notes or check the CHANGELOG.

Version Numbering in Crawl4AI

Crawl4AI follows standard Python version numbering conventions (PEP 440) to help users understand the stability and features of each release.

📈 Version Numbers Explained

Our version numbers follow this pattern: MAJOR.MINOR.PATCH (e.g., 0.4.3)

We use different suffixes to indicate development stages:

• dev (0.4.3dev1): Development versions, unstable

• a (0.4.3a1): Alpha releases, experimental features

• b (0.4.3b1): Beta releases, feature complete but needs testing

• rc (0.4.3): Release candidates, potential final version

• Regular installation (stable version):

• Install pre-release versions:

  pip install crawl4ai --pre
  

• Install specific version:

  pip install crawl4ai==0.4.3b1
  

We use pre-releases to:

• Test new features in real-world scenarios
• Gather feedback before final releases
• Ensure stability for production users
• Allow early adopters to try new features

For production environments, we recommend using the stable version. For testing new features, you can opt-in to pre-releases using the --pre flag.

📖 Documentation & Roadmap

🚨 Documentation Update Alert : We're undertaking a major documentation overhaul next week to reflect recent updates and improvements. Stay tuned for a more comprehensive and up-to-date guide!

For current documentation, including installation instructions, advanced features, and API reference, visit our Documentation Website.

To check our development plans and upcoming features, visit our Roadmap.

📈 Development TODOs

We welcome contributions from the open-source community. Check out our contribution guidelines for more information.

I'll help modify the license section with badges. For the halftone effect, here's a version with it:

Here's the updated license section:

This project is licensed under the Apache License 2.0, attribution is recommended via the badges below. See the Apache 2.0 License file for details.

When using Crawl4AI, you must include one of the following attribution methods:

📈 1. Badge Attribution (Recommended) Add one of these badges to your README, documentation, or website: Theme	Badge
Disco Theme (Animated)
Night Theme (Dark with Neon)
Dark Theme (Classic)
Light Theme (Classic)

HTML code for adding the badges:

<!-- Disco Theme (Animated) -->
<a href="https://github.com/unclecode/crawl4ai">
  <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-disco.svg" alt="Powered by Crawl4AI" width="200"/>
</a>

<!-- Night Theme (Dark with Neon) -->
<a href="https://github.com/unclecode/crawl4ai">
  <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-night.svg" alt="Powered by Crawl4AI" width="200"/>
</a>

<!-- Dark Theme (Classic) -->
<a href="https://github.com/unclecode/crawl4ai">
  <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-dark.svg" alt="Powered by Crawl4AI" width="200"/>
</a>

<!-- Light Theme (Classic) -->
<a href="https://github.com/unclecode/crawl4ai">
  <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-light.svg" alt="Powered by Crawl4AI" width="200"/>
</a>

<!-- Simple Shield Badge -->
<a href="https://github.com/unclecode/crawl4ai">
  <img src="https://img.shields.io/badge/Powered%20by-Crawl4AI-blue?style=flat-square" alt="Powered by Crawl4AI"/>
</a>

📖 2. Text Attribution Add this line to your documentation: This project uses Crawl4AI (<https://github.com/unclecode/crawl4ai>) for web data extraction.

If you use Crawl4AI in your research or project, please cite:

@software{crawl4ai2024,
  author = {UncleCode},
  title = {Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub Repository},
  howpublished = {\url{https://github.com/unclecode/crawl4ai}},
  commit = {Please use the commit hash you're working with}
}

Text citation format:

UncleCode. (2024). Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper [Computer software]. 
GitHub. https://github.com/unclecode/crawl4ai

For questions, suggestions, or feedback, feel free to reach out:

Happy Crawling! 🕸️🚀

Our mission is to unlock the value of personal and enterprise data by transforming digital footprints into structured, tradeable assets. Crawl4AI empowers individuals and organizations with open-source tools to extract and structure data, fostering a shared data economy.

We envision a future where AI is powered by real human knowledge, ensuring data creators directly benefit from their contributions. By democratizing data and enabling ethical sharing, we are laying the foundation for authentic AI advancement.

🔑 Key Opportunities

• Data Capitalization : Transform digital footprints into measurable, valuable assets.
• Authentic AI Data : Provide AI systems with real human insights.
• Shared Economy : Create a fair data marketplace that benefits data creators.

🚀 Development Pathway

1. Open-Source Tools : Community-driven platforms for transparent data extraction.
2. Digital Asset Structuring : Tools to organize and value digital knowledge.
3. Ethical Data Marketplace : A secure, fair platform for exchanging structured data.

For more details, see our full mission statement.

🌟 Current Sponsors

These companies provide core infrastructure and technology that power Crawl4AI’s capabilities — from web access and proxy networks to AI tooling and data pipelines.

Company	About
	Massive is a web access API backed by millions of volunteer devices in 195+ countries. AI agents, models, and data pipelines use it to reach any site on the internet, reliably, in real time, and at scale.

🏢 Enterprise Sponsors

Our enterprise sponsors support Crawl4AI and help scale it to power production-grade data pipelines.

Company	About	Sponsorship Tier
	Helps engineers and buyers find, compare, and source electronic & industrial parts in seconds, with specs, pricing, lead times & alternatives.	🥇 Gold
	Kidocode is a hybrid technology and entrepreneurship school for kids aged 5–18, offering both online and on-campus education.	🥇 Gold
	Singapore-based Aleph Null is Asia’s leading edtech hub, dedicated to student-centric, AI-driven education—empowering learners with the tools to thrive in a fast-changing world.	🥇 Gold

---

💼 Become a Strategic Partner or Sponsor

Interested in partnering with Crawl4AI?

Whether you’re a proxy provider, AI infrastructure company, cloud platform, or an organization looking to support the Crawl4AI ecosystem, we’d love to hear from you.

📩 Contact: hello@crawl4ai.com

🧑‍🤝 Individual Sponsors

A heartfelt thanks to our individual supporters! Every contribution helps us keep our opensource mission alive and thriving!

Want to join them? Sponsor Crawl4AI →