现代网络的轻松抓取

العربيه | Español | Português (Brasil) | Français | Deutsch | 简体中文 | 日本語 | Русский | 한국어

选择方法 · 抓取器 · 爬虫 · 代理轮换 · CLI · MCP

Scrapling 是一个自适应的网页抓取框架，可以处理从单次请求到全量爬取的一切工作。

它的解析器会从网站变化中学习，并在页面更新时自动重新定位你的元素。它的抓取器开箱即用地绕过 Cloudflare Turnstile 等反爬系统。而它的爬虫框架让你能够扩展至并发、多会话的爬取，支持暂停/恢复和自动代理轮换——所有这些仅需几行 Python 代码。一个库，不妥协。

极速爬取，附带实时统计和流式传输。由网页抓取者为自己和普通用户打造，每个人都能找到所需。

from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)  # 在雷达下抓取网站！
products = p.css('.product', auto_save=True)                                        # 抓取能够承受网站设计变化的数据！
products = p.css('.product', adaptive=True)                                         # 之后，如果网站结构改变，传入 `adaptive=True` 来找到它们！

或者扩展为完整的爬虫

from scrapling.spiders import Spider, Response

class MySpider(Spider):
  name = "demo"
  start_urls = ["https://example.com/"]

  async def parse(self, response: Response):
      for item in response.css('.product'):
          yield {"title": item.css('h2::text').get()}

MySpider().start()

铂金赞助商

	Proxidize 为抓取、浏览器自动化、SEO 监控、AI 代理和数据采集提供移动和住宅代理。使用代码 scrapling20 享 8 折。
	ColdProxy 提供住宅和数据中心代理，用于稳定的网页抓取、公开数据采集以及覆盖 195+ 个国家的地理定位测试。
	Scrapling 处理 Cloudflare Turnstile。对于企业级防护，Hyper Solutions 提供了 API 端点，可为 Akamai、DataDome、Kasada 和 Incapsula 生成有效的反机器人令牌。简单的 API 调用，无需浏览器自动化。
	嘿，我们构建了 BirdProxies，因为代理不应该复杂或过于昂贵。在 195+ 个地点提供快速住宅和 ISP 代理，公平定价和真正支持。
在我们的着陆页试玩 FlappyBird 游戏，赢取免费流量！
	Evomi：住宅代理低至 $0.49/GB。提供完全伪装的 Chromium 抓取浏览器，住宅 IP，自动验证码解决以及反爬绕过。
Scraper API 提供无忧结果。可集成 MCP 和 N8N。
	TikHub.io 提供 900+ 稳定 API，覆盖 TikTok、X、YouTube、Instagram 等 16+ 平台，以及 4000 万+ 数据集。
还提供 折扣 AI 模型 —— Claude、GPT、GEMINI 等，最高可享 71% 折扣。
	合上笔记本电脑，你的爬虫继续运行。
PetroSky VPS —— 为不间断自动化构建的云服务器。提供 Windows 和 Linux 机器，完全控制。起步价 €6.99/月。
	阅读《网页抓取俱乐部》对 Scrapling 的完整评测（2025 年 11 月），这是专注于网页抓取的 #1 通讯。
	Swiftproxy 提供可扩展的住宅代理，拥有 8000 万+ IP，覆盖 195+ 个国家，提供快速、可靠的连接，自动轮换和强大的防封禁性能。可免费试用。
	9Proxy 提供住宅代理，低至 $0.018/IP 或 $0.68/GB。2000 万+ IP，覆盖 90+ 个国家。支持固定或轮换会话，可通过桌面或移动应用管理。
	NodeMaven —— 可靠的代理提供商，市场最高品质 IP。使用促销代码 SCRAPLING35 享受代理 35% 折扣。

想要在此展示您的广告？点击这里

赞助商

想要在此展示您的广告？点击这里 并选择适合您的等级！

---

爬虫——完整的抓取框架

• 🕷️ 类 Scrapy 风格的爬虫 API：使用 start_urls、异步 parse 回调和 Request/Response 对象定义爬虫。
• ⚡ 并发抓取：可配置并发限制、按域名限速和下载延迟。
• 🔄 多会话支持：统一接口，在单个爬虫中处理 HTTP 请求和隐蔽无头浏览器——按会话 ID 将请求路由到不同会话。
• 💾 暂停和恢复：基于检查点的抓取持久化。按 Ctrl+C 优雅关闭；重新启动即从上次中断处继续。
• 📡 流式模式：通过 async for item in spider.stream() 实时流式传输抓取到的条目，并提供实时统计——非常适合 UI、管道和长时间运行的爬取。
• 🛡️ 被屏蔽请求检测：自动检测并重试被屏蔽的请求，并可自定义逻辑。
• 🤖 Robots.txt 合规：可选的 robots_txt_obey 标志，遵守 Disallow 规则。

Effortless Web Scraping for the Modern Web

العربيه | Español | Português (Brasil) | Français | Deutsch | 简体中文 | 日本語 | Русский | 한국어

Selection methods · Fetchers · Spiders · Proxy Rotation · CLI · MCP

Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl.

Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation - all in a few lines of Python. One library, zero compromises.

Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.

from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)  # Fetch website under the radar!
products = p.css('.product', auto_save=True)                                        # Scrape data that survives website design changes!
products = p.css('.product', adaptive=True)                                         # Later, if the website structure changes, pass `adaptive=True` to find them!

Or scale up to full crawls

from scrapling.spiders import Spider, Response

class MySpider(Spider):
  name = "demo"
  start_urls = ["https://example.com/"]

  async def parse(self, response: Response):
      for item in response.css('.product'):
          yield {"title": item.css('h2::text').get()}

MySpider().start()

Platinum Sponsors

	Proxidize provides mobile and residential proxies for scraping, browser automation, SEO monitoring, AI agents, and data collection. Use codescrapling20 for 20% off.
	ColdProxy provides residential and datacenter proxies for stable web scraping, public data collection, and geo-targeted testing across 195+ countries.
	Scrapling handles Cloudflare Turnstile. For enterprise-grade protection, Hyper Solutions provides API endpoints that generate valid antibot tokens for Akamai , DataDome , Kasada , and Incapsula. Simple API calls, no browser automation required.
	Hey, we built BirdProxies because proxies shouldn't be complicated or overpriced. Fast residential and ISP proxies in 195+ locations, fair pricing, and real support.
Try our FlappyBird game on the landing page for free data!
	Evomi : residential proxies from $0.49/GB. Scraping browser with fully spoofed Chromium, residential IPs, auto CAPTCHA solving, and anti-bot bypass.
Scraper API for hassle-free results. MCP and N8N integrations are available.
	TikHub.io provides 900+ stable APIs across 16+ platforms including TikTok, X, YouTube & Instagram, with 40M+ datasets.
Also offers DISCOUNTED AI models - Claude, GPT, GEMINI & more up to 71% off.
	Close your laptop. Your scrapers keep running.
PetroSky VPS - cloud servers built for nonstop automation. Windows and Linux machines with full control. From €6.99/mo.
	Read a full review of Scrapling on The Web Scraping Club (Nov 2025), the #1 newsletter dedicated to Web Scraping.
	Swiftproxy provides scalable residential proxies with 80M+ IPs across 195+ countries, delivering fast, reliable connections, automatic rotation, and strong anti-block performance. Free trial available.
	9Proxy provides residential proxies from just $0.018/IP or $0.68/GB. 20M+ IPs across 90+ countries. Sticky or rotating sessions, managed from desktop or mobile app.
	NodeMaven - reliable proxy provider with the highest quality IP on the market. Use promo code SCRAPLING35 for 35% discount on proxies.

Do you want to show your ad here? Clickhere

Sponsors

Do you want to show your ad here? Clickhere and choose the tier that suits you!

---

Spiders - A Full Crawling Framework

• 🕷️ Scrapy-like Spider API : Define spiders with start_urls, async parse callbacks, and Request/Response objects.
• ⚡ Concurrent Crawling : Configurable concurrency limits, per-domain throttling, and download delays.
• 🔄 Multi-Session Support : Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID.
• 💾 Pause & Resume: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off.
• 📡 Streaming Mode : Stream scraped items as they arrive via async for item in spider.stream() with real-time stats - ideal for UI, pipelines, and long-running crawls.
• 🛡️ Blocked Request Detection : Automatic detection and retry of blocked requests with customizable logic.
• 🤖 Robots.txt Compliance : Optional robots_txt_obey flag that respects Disallow, Crawl-delay, and Request-rate directives with per-domain caching.
• 🧪 Development Mode : Cache responses to disk on the first run and replay them on subsequent runs - iterate on your parse() logic without re-hitting the target servers.
• 📦 Built-in Export : Export results through hooks and your own pipeline or the built-in JSON/JSONL with result.items.to_json() / result.items.to_jsonl() respectively.

Advanced Websites Fetching with Session Support

• HTTP Requests : Fast and stealthy HTTP requests with the Fetcher class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.
• Dynamic Loading : Fetch dynamic websites with full browser automation through the DynamicFetcher class supporting Playwright's Chromium and Google's Chrome.
• Anti-bot Bypass : Advanced stealth capabilities with StealthyFetcher and fingerprint spoofing. Can easily bypass all types of Cloudflare's Turnstile/Interstitial with automation.
• Session Management : Persistent session support with FetcherSession, StealthySession, and DynamicSession classes for cookie and state management across requests.
• Proxy Rotation : Built-in ProxyRotator with cyclic or custom rotation strategies across all session types, plus per-request proxy overrides.
• Domain & Ad Blocking: Block requests to specific domains (and their subdomains) or enable built-in ad blocking (~3,500 known ad/tracker domains) in browser-based fetchers.
• DNS Leak Prevention : Optional DNS-over-HTTPS support to route DNS queries through Cloudflare's DoH, preventing DNS leaks when using proxies.
• Async Support : Complete async support across all fetchers and dedicated async session classes.

Adaptive Scraping & AI Integration

• 🔄 Smart Element Tracking : Relocate elements after website changes using intelligent similarity algorithms.
• 🎯 Smart Flexible Selection : CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
• 🔍 Find Similar Elements : Automatically locate elements similar to found elements.
• 🤖 MCP Server to be used with AI : Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features powerful, custom capabilities that leverage Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. (demo video)

High-Performance & battle-tested Architecture

• 🚀 Lightning Fast : Optimized performance outperforming most Python scraping libraries.
• 🔋 Memory Efficient : Optimized data structures and lazy loading for a minimal memory footprint.
• ⚡ Fast JSON Serialization : 10x faster than the standard library.
• 🏗️ Battle tested : Not only does Scrapling have 92% test coverage and full type hints coverage, but it has been used daily by hundreds of Web Scrapers over the past year.

Developer/Web Scraper Friendly Experience

• 🎯 Interactive Web Scraping Shell : Optional built-in IPython shell with Scrapling integration, shortcuts, and new tools to speed up Web Scraping scripts development, like converting curl requests to Scrapling requests and viewing requests results in your browser.
• 🚀 Use it directly from the Terminal : Optionally, you can use Scrapling to scrape a URL without writing a single line of code!
• 🛠️ Rich Navigation API : Advanced DOM traversal with parent, sibling, and child navigation methods.
• 🧬 Enhanced Text Processing : Built-in regex, cleaning methods, and optimized string operations.
• 📝 Auto Selector Generation : Generate robust CSS/XPath selectors for any element.
• 🔌 Familiar API : Similar to Scrapy/BeautifulSoup with the same pseudo-elements used in Scrapy/Parsel.
• 📘 Complete Type Coverage : Full type hints for excellent IDE support and code completion. The entire codebase is automatically scanned with PyRight and MyPy with each change.
• 🔋 Ready Docker image : With each release, a Docker image containing all browsers is automatically built and pushed.

Let's give you a quick glimpse of what Scrapling can do without deep diving.

HTTP requests with session support

from scrapling.fetchers import Fetcher, FetcherSession

with FetcherSession(impersonate='chrome') as session:  # Use latest version of Chrome's TLS fingerprint
    page = session.get('https://quotes.toscrape.com/', stealthy_headers=True)
    quotes = page.css('.quote .text::text').getall()

# Or use one-off requests
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()

Advanced stealth mode

from scrapling.fetchers import StealthyFetcher, StealthySession

with StealthySession(headless=True, solve_cloudflare=True) as session:  # Keep the browser open until you finish
    page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)
    data = page.css('#padded_content a').getall()

# Or use one-off request style, it opens the browser for this request, then closes it after finishing
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
data = page.css('#padded_content a').getall()

Full browser automation

from scrapling.fetchers import DynamicFetcher, DynamicSession

with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session:  # Keep the browser open until you finish
    page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
    data = page.xpath('//span[@class="text"]/text()').getall()  # XPath selector if you prefer it

# Or use one-off request style, it opens the browser for this request, then closes it after finishing
page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()

Build full crawlers with concurrent requests, multiple session types, and pause/resume:

from scrapling.spiders import Spider, Request, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
    concurrent_requests = 10
    
    async def parse(self, response: Response):
        for quote in response.css('.quote'):
            yield {
                "text": quote.css('.text::text').get(),
                "author": quote.css('.author::text').get(),
            }
            
        next_page = response.css('.next a')
        if next_page:
            yield response.follow(next_page[0].attrib['href'])

result = QuotesSpider().start()
print(f"Scraped {len(result.items)} quotes")
result.items.to_json("quotes.json")

Use multiple session types in a single spider:

from scrapling.spiders import Spider, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession

class MultiSessionSpider(Spider):
    name = "multi"
    start_urls = ["https://example.com/"]
    
    def configure_sessions(self, manager):
        manager.add("fast", FetcherSession(impersonate="chrome"))
        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
    
    async def parse(self, response: Response):
        for link in response.css('a::attr(href)').getall():
            # Route protected pages through the stealth session
            if "protected" in link:
                yield Request(link, sid="stealth")
            else:
                yield Request(link, sid="fast", callback=self.parse)  # explicit callback

Pause and resume long crawls with checkpoints by running the spider like this:

QuotesSpider(crawldir="./crawl_data").start()

Press Ctrl+C to pause gracefully - progress is saved automatically. Later, when you start the spider again, pass the same crawldir, and it will resume from where it stopped.

Advanced Parsing & Navigation

from scrapling.fetchers import Fetcher

# Rich element selection and navigation
page = Fetcher.get('https://quotes.toscrape.com/')

# Get quotes with multiple selection methods
quotes = page.css('.quote')  # CSS selector
quotes = page.xpath('//div[@class="quote"]')  # XPath
quotes = page.find_all('div', {'class': 'quote'})  # BeautifulSoup-style
# Same as
quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote')  # and so on...
# Find element by text content
quotes = page.find_by_text('quote', tag='div')

# Advanced navigation
quote_text = page.css('.quote')[0].css('.text::text').get()
quote_text = page.css('.quote').css('.text::text').getall()  # Chained selectors
first_quote = page.css('.quote')[0]
author = first_quote.next_sibling.css('.author::text')
parent_container = first_quote.parent

# Element relationships and similarity
similar_elements = first_quote.find_similar()
below_elements = first_quote.below_elements()

You can use the parser right away if you don't want to fetch websites like below:

from scrapling.parser import Selector

page = Selector("<html>...</html>")

And it works precisely the same way!

Async Session Management Examples

import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession

async with FetcherSession(http3=True) as session:  # `FetcherSession` is context-aware and can work in both sync/async patterns
    page1 = session.get('https://quotes.toscrape.com/')
    page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')

# Async session usage
async with AsyncStealthySession(max_pages=2) as session:
    tasks = []
    urls = ['https://example.com/page1', 'https://example.com/page2']
    
    for url in urls:
        task = session.fetch(url)
        tasks.append(task)
    
    print(session.get_pool_stats())  # Optional - The status of the browser tabs pool (busy/free/error)
    results = await asyncio.gather(*tasks)
    print(session.get_pool_stats())

Scrapling includes a powerful command-line interface:

Launch the interactive Web Scraping shell

Extract pages to a file directly without programming (Extracts the content inside the body tag by default). If the output file ends with .txt, then the text content of the target will be extracted. If it ends in .md, it will be a Markdown representation of the HTML content; if it ends in .html, it will be the HTML content itself.

scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome'  # All elements matching the CSS selector '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare

Note

There are many additional features, but we want to keep this page concise, including the MCP server and the interactive Web Scraping Shell. Check out the full documentation here

Scrapling isn't just powerful-it's also blazing fast. The following benchmarks compare Scrapling's parser with the latest versions of other popular libraries.

Text Extraction Speed Test (5000 nested elements)

| Library | Time (ms) | vs Scrapling

---|---|---|---
1 | Scrapling | 2.02 | 1.0x
2 | Parsel/Scrapy | 2.04 | 1.01
3 | Raw Lxml | 2.54 | 1.257
4 | PyQuery | 24.17 | ~12x
5 | Selectolax | 82.63 | ~41x
6 | MechanicalSoup | 1549.71 | ~767.1x
7 | BS4 with Lxml | 1584.31 | ~784.3x
8 | BS4 with html5lib | 3391.91 | ~1679.1x

Element Similarity & Text Search Performance

Scrapling's adaptive element finding capabilities significantly outperform alternatives:

Library	Time (ms)	vs Scrapling
Scrapling	2.39	1.0x
AutoScraper	12.45	5.209x

All benchmarks represent averages of 100+ runs. See benchmarks.py for methodology.

Scrapling requires Python 3.10 or higher:

Important

This installation only includes the parser engine and its dependencies, without any fetchers or commandline dependencies. So importing anything from scrapling.fetchers or scrapling.spiders, like in the examples above, will raise ModuleNotFoundError with this installation alone. If you are going to use any of the fetchers or spiders, install the fetchers' dependencies first as shown below.

1. If you are going to use any of the extra features below, the fetchers, or their classes, you will need to install fetchers' dependencies and their browser dependencies as follows:

   pip install "scrapling[fetchers]"
   
   scrapling install           # normal install
   scrapling install  --force  # force reinstall
   

This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.

Or you can install them from the code instead of running a command like this:

     from scrapling.cli import install
     
     install([], standalone_mode=False)          # normal install
     install(["--force"], standalone_mode=False) # force reinstall

2. Extra features:

   • Install the MCP server feature:

     pip install "scrapling[ai]"
     

   • Install shell features (Web Scraping shell and the extract command):

     pip install "scrapling[shell]"
     

   • Install everything:

     pip install "scrapling[all]"
     

Remember that you need to install the browser dependencies with scrapling install after any of these extras (if you didn't already)

You can also install a Docker image with all extras and browsers with the following command from DockerHub:

docker pull pyd4vinci/scrapling

Or download it from the GitHub registry:

docker pull ghcr.io/d4vinci/scrapling:latest

This image is automatically built and pushed using GitHub Actions and the repository's main branch.

We welcome contributions! Please read our contributing guidelines before getting started.

Caution

This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international data scraping and privacy laws. The authors and contributors are not responsible for any misuse of this software. Always respect the terms of service of websites and robots.txt files.

If you have used our library for research purposes please quote us with the following reference:

  @misc{scrapling,
    author = {Karim Shoair},
    title = {Scrapling},
    year = {2024},
    url = {https://github.com/D4Vinci/Scrapling},
    note = {An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!}
  }

This work is licensed under the BSD-3-Clause License.

This project includes code adapted from:

• Parsel (BSD License)-Used for translator submodule

---

Designed & crafted with ❤️ by Karim Shoair.