# Open-Source Web Scraping Tools: The 42 Most-Starred GitHub Repos, by Category (2026)


The most-starred open-source web scraping tools on GitHub are Firecrawl (199,913 stars), browser-use (120,388), Scrapling (100,136), Playwright (99,642), and Puppeteer (95,816), as of November 9, 2026. Microsoft's MarkItDown (194,504) sits second by stars, but it converts documents to Markdown rather than fetching pages; we keep it in the list because so many AI scraping pipelines use it for that conversion step. We pulled live data from the GitHub API for 52 candidate repos, kept the 42 that met our rules, and sorted them into 8 categories, from AI-ready crawlers to the HTML parsers underneath them. For each one we list stars and license, and for the most-used libraries we compare GitHub stars with last month's PyPI and npm downloads.

The pattern in the data is hard to miss. Tools created in 2024 or later make up 17 of the 42 repos but hold 58% of the stars. Downloads tell a different story, and both numbers are below.

> **Key Takeaways**
>
> - **Firecrawl is the most-starred web scraping repo on GitHub**, at 199,913 stars. Six of the top ten repos were created in 2024 or later.
> - **AI-era tools dominate attention:** the 17 repos created since January 2024 hold 958,003 of the 1,664,365 stars in this list (58%).
> - **Stars and downloads disagree.** Scrapling has 100,136 stars to Scrapy's 65,286, but Scrapy had 4.30 million PyPI downloads last month against Scrapling's 0.92 million.
> - **Check the license before you ship.** Six of the 42 use copyleft licenses (AGPL-3.0 or MPL-2.0), including Firecrawl and Skyvern.
> - **None of the self-hosted crawlers, browsers, or agents brings its own IP addresses.** Their requests leave from the machine the tool runs on, which is why most of them expose a proxy setting.

## How we ranked open-source web scraping tools on GitHub

We started from GitHub's search for the `web-scraping` topic and for crawler and scraper repos with more than 15,000 stars, then added the browser automation, parsing, and MCP projects that scrapers depend on. We screened out platform-specific tools, such as single-site downloaders and social media scrapers, which gave 52 general-purpose candidates. On November 9, 2026, our script called the GitHub REST API for each one and recorded stars, forks, license, language, creation date, and last push.

A candidate made the list if it met all four rules:

1. **At least 5,000 stars.**
2. **An open-source license** detected by GitHub.
3. **A push in the last 12 months**, so the project is still maintained.
4. **Created at least 12 months ago**, so the star count reflects more than a launch spike.

Download counts come from [pypistats.org](https://pypistats.org/) for Python packages and the [npm registry API](https://github.com/npm/registry/blob/main/docs/download-counts.md) for JavaScript ones, both for the last 30 days. Stars measure attention, not usage, so we report both.

![Bar chart of the 15 most-starred open-source web scraping tools on GitHub on November 9, 2026, led by Firecrawl at 199,913 stars; tools created since 2024 are highlighted.](assets/top-starred.png "The 15 most-starred repos in our list. Orange bars mark repos created in 2024 or later. Source: GitHub API, November 9, 2026.")

## AI-ready crawlers and extraction

These tools fetch a page and hand back something a language model can read, usually Markdown or structured JSON. Nearly everything here is new: every repo except Maxun was created in 2024.

| Tool | Stars | License | What it does |
|---|---|---|---|
| [Firecrawl](https://github.com/firecrawl/firecrawl) | 199,913 | AGPL-3.0 | Scrape, crawl, and search API that returns LLM-ready data. Self-host the repo or use the hosted cloud |
| [MarkItDown](https://github.com/microsoft/markitdown) | 194,504 | MIT | Microsoft's Python tool for converting files and office documents to Markdown. It converts content you already have rather than fetching it |
| [Crawl4AI](https://github.com/unclecode/crawl4ai) | 87,124 | Apache-2.0 | Python crawler with LLM-friendly Markdown output. Self-host it, or use Crawl4AI Cloud |
| [ScrapeGraphAI](https://github.com/ScrapeGraphAI/Scrapegraph-ai) | 32,646 | MIT | Python library that uses LLM-driven pipelines to extract structured data from a page based on a prompt |
| [Maxun](https://github.com/getmaxun/maxun) | 18,183 | AGPL-3.0 | Turns a website into a structured API, with recorded robots for extraction plus site monitoring |
| [Jina Reader](https://github.com/jina-ai/reader) | 12,247 | Apache-2.0 | Converts any URL to LLM-friendly text by adding the `r.jina.ai/` prefix |

Crawl4AI lists Massive as a strategic partner for web access in its [README](https://github.com/unclecode/crawl4ai#-current-sponsors), and we covered that partnership in our [Crawl4AI spotlight](https://www.joinmassive.com/blog/crawl4ai-partner-spotlight). If you're choosing a format for model input, our post on [how HTML-to-Markdown cuts agent token costs](https://www.joinmassive.com/blog/skip-the-browser-how-html-to-markdown-cuts-agent-token-costs-by-80) explains why Markdown is the default output here.

## Crawling frameworks

Frameworks handle the parts of a crawl that aren't parsing: queues, retries, concurrency, deduplication, and export. Scrapy has done this in Python since 2010. Scrapling, created in October 2024, now has more stars.

| Tool | Stars | License | Language | What it does |
|---|---|---|---|---|
| [Scrapling](https://github.com/D4Vinci/Scrapling) | 100,136 | BSD-3-Clause | Python | Framework that covers single requests through full crawls, with HTTP and browser-based fetchers |
| [Scrapy](https://github.com/scrapy/scrapy) | 65,286 | BSD-3-Clause | Python | Fast, high-level crawling framework built around spiders, item pipelines, and middleware |
| [Crawlee](https://github.com/apify/crawlee) | 26,390 | Apache-2.0 | TypeScript | Apify's Node.js crawling library. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP, with proxy rotation |
| [Colly](https://github.com/gocolly/colly) | 25,581 | Apache-2.0 | Go | Scraper and crawler framework for Go |
| [Katana](https://github.com/projectdiscovery/katana) | 17,815 | MIT | Go | ProjectDiscovery's crawling and spidering framework, common in security reconnaissance |
| [Crawlee for Python](https://github.com/apify/crawlee-python) | 9,655 | Apache-2.0 | Python | The Python port of Crawlee, working with Parsel, BeautifulSoup, Playwright, and raw HTTP |
| [Ferret](https://github.com/MontFerret/ferret) | 6,025 | Apache-2.0 | Go | Declarative query language and Go runtime for structured extraction |

## Browser automation

When a page builds its content with JavaScript, you need a real browser. These libraries drive one. Playwright and Puppeteer are within 4,000 stars of each other, but on npm Playwright pulled 521 million downloads last month to Puppeteer's 50 million, most likely because test suites also install it in CI.

| Tool | Stars | License | Language | What it does |
|---|---|---|---|---|
| [Playwright](https://github.com/microsoft/playwright) | 99,642 | Apache-2.0 | TypeScript | Drives Chromium, Firefox, and WebKit with one API, with bindings for Python, Java, and .NET |
| [Puppeteer](https://github.com/puppeteer/puppeteer) | 95,816 | Apache-2.0 | TypeScript | JavaScript API for Chrome and Firefox |
| [Lightpanda](https://github.com/lightpanda-io/browser) | 41,649 | AGPL-3.0 | Zig | A headless browser built from scratch for AI and automation, rather than a stripped-down Chrome |
| [Selenium](https://github.com/SeleniumHQ/selenium) | 34,577 | Apache-2.0 | Java | The original WebDriver automation framework, with bindings in most languages |
| [chromedp](https://github.com/chromedp/chromedp) | 13,355 | MIT | Go | Drives Chrome through the DevTools Protocol from Go |
| [SeleniumBase](https://github.com/seleniumbase/SeleniumBase) | 13,233 | MIT | Python | Python APIs for web automation and testing, with modes for sites that detect bots |
| [Rod](https://github.com/go-rod/rod) | 7,145 | MIT | Go | DevTools Protocol driver for automation and scraping in Go |
| [Pydoll](https://github.com/autoscrape-labs/pydoll) | 7,136 | MIT | Python | Automates Chromium without WebDriver, with realistic interactions |

## AI browser agents

Agents put a language model in charge of the browser: it reads the page, decides what to click, and repeats until the task is done. This is the newest category, and its leader, browser-use, has the third-highest star count in the list, behind Firecrawl and MarkItDown. We compared three of these frameworks in depth in [browser-use vs Stagehand vs Skyvern](https://www.joinmassive.com/blog/browser-use-vs-stagehand-vs-skyvern-choosing-an-agent-browser-framework).

| Tool | Stars | License | What it does |
|---|---|---|---|
| [browser-use](https://github.com/browser-use/browser-use) | 120,388 | MIT | Python library for AI browser agents, plus a hosted cloud with stealth browsers |
| [Stagehand](https://github.com/browserbase/stagehand) | 25,990 | MIT | Browserbase's SDK for extracting data and acting on any site, mixing code with natural-language steps |
| [Skyvern](https://github.com/Skyvern-AI/skyvern) | 23,367 | AGPL-3.0 | Automates browser-based workflows with LLMs and computer vision |
| [Nanobrowser](https://github.com/nanobrowser/nanobrowser) | 14,266 | Apache-2.0 | Chrome extension that runs an AI agent in your own logged-in browser |

## Stealth and fingerprinting

Sites check more than your IP address. They read the TLS handshake, HTTP/2 settings, and dozens of browser properties to decide whether a client is a person or a script. These projects change what that fingerprint says. The most-starred repo in this space, CloakBrowser, and another stealth browser, camofox-browser, were both created in 2026, so they fail our 12-month age rule and are listed separately below.

| Tool | Stars | License | What it does |
|---|---|---|---|
| [Camoufox](https://github.com/daijro/camoufox) | 13,132 | MPL-2.0 | Modified Firefox build for automation, with fingerprint controls |
| [curl_cffi](https://github.com/lexiforest/curl_cffi) | 6,827 | MIT | Python HTTP client that can impersonate browser TLS, JA3, and HTTP/2 fingerprints |
| [Patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright) | 5,126 | Apache-2.0 | Patched version of Playwright that removes common automation signals |

A matching fingerprint still arrives from whatever IP address you run on. A browser that looks like Chrome on a home laptop but connects from a cloud data center is a common reason scrapers get blocked, which we covered in [why AI agents get blocked on datacenter IPs](https://www.joinmassive.com/blog/why-ai-agents-get-blocked-on-datacenter-ips-and-how-to-fix-it).

## Parsing and content extraction

Once you have the HTML, something has to pull out the parts you want. These libraries sit underneath most of the tools above. One gap in a star ranking: Beautiful Soup, probably the best-known Python parser, is developed on [Launchpad](https://code.launchpad.net/beautifulsoup) rather than GitHub, so it has no star count to compare.

| Tool | Stars | License | Language | What it does |
|---|---|---|---|---|
| [Cheerio](https://github.com/cheeriojs/cheerio) | 30,536 | MIT | TypeScript | Fast HTML and XML parsing with a jQuery-style API |
| [Newspaper](https://github.com/codelucas/newspaper) | 15,183 | MIT | Python | Extracts article text and metadata from news pages |
| [goquery](https://github.com/PuerkitoBio/goquery) | 14,991 | BSD-3-Clause | Go | jQuery-style HTML traversal for Go |
| [Readability](https://github.com/mozilla/readability) | 11,529 | Apache-2.0 | JavaScript | The standalone version of Firefox Reader View's article extractor |
| [jsoup](https://github.com/jhy/jsoup) | 11,413 | MIT | Java | Java HTML parser for cleaning, editing, and scraping |
| [Trafilatura](https://github.com/adbar/trafilatura) | 7,104 | Apache-2.0 | Python | Gathers main text and metadata, with output as CSV, JSON, HTML, Markdown, TXT, or XML |

## Monitoring and no-code

Not every scraping job is a one-time crawl. These tools watch pages over time or let non-developers build scrapers visually.

| Tool | Stars | License | What it does |
|---|---|---|---|
| [Huginn](https://github.com/huginn/huginn) | 50,444 | MIT | Self-hosted agents that monitor the web and act on what they find |
| [EasySpider](https://github.com/NaiboWang/EasySpider) | 44,770 | AGPL-3.0 | Visual, no-code crawler designer |
| [changedetection.io](https://github.com/dgtlmoon/changedetection.io) | 35,662 | Apache-2.0 | Website change detection and alerts for price drops, restocks, and content changes |
| [AutoScraper](https://github.com/alirezamika/autoscraper) | 8,080 | MIT | Learns scraping rules from example values you give it |

## MCP servers for scraping

MCP servers expose browsing and scraping as tools that an AI agent can call. Each of the four that passed our rules is a year or more old, and the Playwright one is heavily used: `@playwright/mcp` had 29.3 million npm downloads last month.

| Tool | Stars | License | What it does |
|---|---|---|---|
| [Chrome DevTools MCP](https://github.com/ChromeDevTools/chrome-devtools-mcp) | 55,003 | Apache-2.0 | Gives coding agents control of a live Chrome through DevTools |
| [Playwright MCP](https://github.com/microsoft/playwright-mcp) | 39,361 | Apache-2.0 | Microsoft's MCP server for browser automation with Playwright |
| [Apify MCP Server](https://github.com/apify/apify-mcp-server) | 15,350 | MIT | Connects agents to the ready-made scrapers and crawlers in the Apify Store |
| [Firecrawl MCP Server](https://github.com/firecrawl/firecrawl-mcp-server) | 7,785 | MIT | Firecrawl's search, scrape, crawl, map, and agent tools for any MCP client |

## Stars vs downloads: which tools people actually install

A star is a bookmark. A download is closer to a deployment, although download counts also include CI runs, mirrors, and installs pulled in as a dependency of another package. We matched the main Python crawling and agent libraries with their PyPI packages:

| Tool | GitHub stars | PyPI downloads, last 30 days | Downloads per star |
|---|---|---|---|
| browser-use | 120,388 | 8,061,974 | 67 |
| Scrapy | 65,286 | 4,303,709 | 66 |
| Crawl4AI | 87,124 | 1,275,996 | 15 |
| Scrapling | 100,136 | 919,855 | 9 |
| Crawlee for Python | 9,655 | 818,726 | 85 |

Scrapy's more than 15 years of production use show here. It has a third fewer stars than Scrapling and about 4.7 times the installs. Crawlee for Python has the fewest stars in the table and the highest ratio, which suggests a user base that installs more than it stars. The newer tools are growing from a smaller installed base, so the gap is a snapshot, not a verdict. Firecrawl is missing from the table because its PyPI package, `firecrawl-py`, is the client for the hosted API rather than the self-hosted crawler; it had 5.34 million downloads last month. For browser drivers the scale is different again: Selenium had 28.0 million PyPI downloads and Playwright 97.0 million on PyPI on top of its npm numbers.

## New, fast-rising, and left out

Ten candidates missed our rules. Four of them are new repos that are already large, which is worth watching:

- **Created in 2026, under 12 months old:** CloakBrowser (32,789 stars, a stealth Chromium), Obscura (30,905, a headless browser for agents), moli (23,110, a Rust headless browser), and camofox-browser (12,211). They are too new for us to judge how well they will be maintained.
- **No push in the last 12 months:** GPT Crawler, undetected-chromedriver, and Postlight Parser.
- **Under 5,000 stars:** spider-rs, Apify's fingerprint-suite, and Scrapy's Parsel selector library.

We also left out platform-specific tools that would otherwise rank high, such as MediaCrawler (about 70,700 stars), which targets Chinese social platforms, and the video downloader you-get (about 56,900).

## Licenses: what you can ship

Most of this list uses permissive licenses (MIT, Apache-2.0, BSD-3-Clause) that let you embed the code in a commercial product. Six repos use copyleft licenses: Firecrawl, Maxun, Skyvern, EasySpider, and Lightpanda are AGPL-3.0, and Camoufox is MPL-2.0. The AGPL's network clause can require you to publish your changes if you run a modified version as a service, which matters if you plan to self-host and modify. Several of these projects also sell a hosted version, which is a separate commercial agreement. Read the license file and ask your legal team before you build on a copyleft repo.

## Where proxies fit in an open-source scraping stack

Every crawler, browser driver, and agent above fetches from the machine it runs on (the parsers only process HTML you already have). On a laptop that is a home IP address. On a cloud server it is a data center IP, and many sites rate-limit or block those ranges. The tools solve fetching, parsing, and orchestration, and the network address is left to you.

That's why most of them take a proxy setting. In Scrapy you set `proxy` in a request's `meta`, Playwright accepts a `proxy` option when it launches a browser, and Crawlee has a `ProxyConfiguration` class with rotation built in. Massive's [residential proxies](https://www.joinmassive.com/solutions/proxies-for-web-scraping) plug into any of these over HTTP, HTTPS, or SOCKS5, from real consumer devices in 195+ countries, with country, state, and city targeting. Every device in the network opted in through the Massive SDK. If you'd rather skip running a browser, the [Web Render API](https://www.joinmassive.com/blog/introducing-massive-web-render-api) renders the page and returns HTML or Markdown.

## Picking from the list

If you're starting a new project, the open-source web scraping tools on GitHub sort into a short decision. For a large Python crawl, start with Scrapy or Scrapling. For Node.js, start with Crawlee. When pages need JavaScript, add Playwright. When the output feeds an LLM, Firecrawl and Crawl4AI return Markdown directly, and when an agent has to click through a site, browser-use and Stagehand lead. Whichever you choose, plan the network layer early: the tool decides how you fetch, and the IP address decides whether you get an answer.

## Frequently Asked Questions

### What is the most-starred web scraping repo on GitHub?

Firecrawl, with 199,913 stars on November 9, 2026. It is followed by browser-use (120,388), Scrapling (100,136), and Playwright (99,642), which is a browser automation library that many scrapers use. Microsoft's MarkItDown has 194,504 stars but converts documents to Markdown rather than scraping.

### What is the best open-source web scraping tool?

It depends on the job. Scrapy and Scrapling suit large Python crawls, Crawlee suits Node.js, Playwright or Puppeteer handle JavaScript-heavy pages, and Firecrawl or Crawl4AI return Markdown for LLMs. For agents that click through pages, browser-use and Stagehand are the most-starred options.

### Is Scrapy still worth learning in 2026?

Yes. Scrapy had 4.30 million PyPI downloads in the 30 days to November 9, 2026, more than Scrapling, Crawl4AI, and Crawlee for Python combined. It is maintained, BSD-licensed, and has a large base of existing spiders, plugins, and middleware.

### Are open-source web scraping tools free for commercial use?

Most are. MIT, Apache-2.0, and BSD-3-Clause allow commercial use with attribution. AGPL-3.0 projects such as Firecrawl and Skyvern add obligations when you run modified code as a network service. The license covers the code, not the data you collect, so check each site's terms and the law where you operate.

## Sources

- GitHub REST API, repository metadata for 52 candidate repos, fetched November 9, 2026 (first-party snapshot; script and raw JSON kept with this post)
- pypistats.org, [recent downloads API](https://pypistats.org/api/), and npm, [download counts API](https://github.com/npm/registry/blob/main/docs/download-counts.md), last-30-day figures fetched November 9, 2026
- Crawl4AI, [README, "Current Sponsors"](https://github.com/unclecode/crawl4ai#-current-sponsors), read November 9, 2026
- Beautiful Soup, [project on Launchpad](https://code.launchpad.net/beautifulsoup)
- Scrapy documentation, [HttpProxyMiddleware](https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.httpproxy); Playwright documentation, [proxy settings](https://playwright.dev/docs/network#http-proxy); Crawlee documentation, [proxy management](https://crawlee.dev/js/docs/guides/proxy-management)
- GNU Project, [GNU Affero General Public License v3](https://www.gnu.org/licenses/agpl-3.0.html)