Web Scraper User Agents: What 17 Open-Source Tools Send by Default (We Tested)

A web scraper's user agent is the User-Agent header it sends with every request, and most open-source tools pick one for you. To see which, we pointed 17 popular scraping tools at a server we control on November 9, 2026, with default settings, and logged every header they sent. Ten of them name themselves, such as python-requests/2.34.2 or Scrapy/2.20.0. Two announce a headless browser. Four send a string that looks like an ordinary desktop browser. And one sends no user agent at all.

We also served a robots.txt that disallowed one page. Across the 19 setups we ran, only one read robots.txt before crawling: Scrapy with ROBOTSTXT_OBEY = True, the value scrapy startproject writes into new projects. Here is what each tool sent, why it matters, and how to change it.

Key Takeaways

• 10 of 17 tools identify themselves by name in their default user agent, including requests, HTTPX, aiohttp, Scrapy, and Axios.
• Playwright and Puppeteer say HeadlessChrome in the user agent, and Playwright repeats it in the sec-ch-ua header, unless you override them.
• Four tools default to a browser-like string: Scrapling, Crawl4AI, and both versions of Crawlee. Each can be set to a custom or rotating user agent.
• Only 1 of 19 setups read robots.txt. Three of the four link-following crawler setups fetched the page our robots.txt disallowed.
• Every tool lets you set the user agent, and every link-following crawler we tested has a robots.txt option, listed in the robots.txt table below.

How we tested

We wrote a small Python server that listens on 127.0.0.1 and logs the path and every header of each request, in the order received. Its home page links to two normal pages and one page under /private/, and its robots.txt says:

User-agent: *
Disallow: /private/

Each Python tool ran in its own fresh environment, and the Node.js tools ran from one npm project, with each third-party library at its latest PyPI or npm release, using the shortest code its documentation shows for fetching a page. For the four link-following setups (Scrapy twice and both versions of Crawlee), we started at the home page and let them crawl. We ran Scrapy twice: once as a bare script, which uses Scrapy's built-in defaults, and once with ROBOTSTXT_OBEY = True, the value scrapy startproject writes into a new project's settings file. We also ran curl_cffi twice, with and without its browser impersonation option.

Two limits apply. The server spoke plain HTTP/1.1 on the local machine, so we could not see TLS or HTTP/2 fingerprints, which sites also check. And Trafilatura refused to connect at all, because its downloader blocks non-public addresses as an SSRF protection. We took its user agent from its source code instead.

Default web scraper user agents, tool by tool

"Headers" is the number of request headers on the first page request. Browser-like tools send 11 to 18, because they add headers such as sec-ch-ua client hints and Sec-Fetch-* that a real browser sends. Library defaults send 2 to 7. The counts for curl_cffi with impersonation (17) and Scrapling (18) include three HTTP/2 upgrade headers that curl adds on plain HTTP, an artifact of our test server.

Ten tools tell the site exactly what they are

HTTP libraries name themselves by default, usually with a version number. That is honest, and it is also the easiest thing for a site to filter: a rule that matches python-requests or axios catches every default request from those libraries. Scrapy goes a step further and includes a URL, +https://scrapy.org, following the convention search engine crawlers use so site owners can find out who is visiting.

If you run a crawler at scale, that convention is worth keeping. A descriptive user agent gives robots.txt rules a name to match (RFC 9309), and its contact URL lets a site owner reach you instead of blocking you. Changing it is one line in every library: headers={"User-Agent": ...} in requests and HTTPX, the USER_AGENT setting in Scrapy, or a headers option in Axios and fetch. A descriptive one in requests looks like this:

import requests
UA = ("PriceBot/1.0 "
      "(+https://example.com/bot)")
r = requests.get(
    "https://example.com",
    headers={"User-Agent": UA},
)

One outlier: curl_cffi sends no User-Agent header at all unless you ask it to impersonate a browser. With impersonate="chrome", it sent 17 headers, with a Chrome 152 user agent and Chrome's client hints. Impersonation is the library's main purpose, so the bare default probably isn't what most people run.

Two headless browsers say so

Playwright and Puppeteer launch Chromium in headless mode by default, and that browser announces itself. Both sent a user agent containing HeadlessChrome/156, and Playwright's sec-ch-ua client hint also listed "HeadlessChrome";v="156". A site that looks for that token sees a browser automation tool in the first request.

Both let you replace the user agent: user_agent on a Playwright browser context, or page.setUserAgent() in Puppeteer. The client hint headers come from the browser itself, so a matching override needs more than the user agent string. Patched builds such as Patchright exist for this reason, which we listed in the open-source web scraping stack.

Four tools look like a browser out of the box

Scrapling, Crawl4AI, and both versions of Crawlee send a browser-style user agent by default, with the extra headers to match. Scrapling's fetcher sent Chrome 152 on Windows with 18 headers, close to what curl_cffi's impersonation produced, and added a Referer of https://www.google.com/ by default. Crawlee generates a browser fingerprint for its sessions: our Node.js run rotated through browser identities across its four requests (Edge 147 on Windows first, then Chrome 147 on Windows and macOS), and our Python run stayed on Firefox 128 on Linux.

Crawl4AI's default user agent is a fixed Chrome-on-Linux string, sent with matching sec-ch-ua hints, and you can set it to match whatever browser and platform you run. Its BrowserConfig takes a user_agent string, or user_agent_mode="random" to pick from a pool (Crawl4AI docs). Whatever tool you use, if you run it on a different machine or browser version than its default string describes, align the two.

Only one setup read robots.txt

None of the HTTP libraries read robots.txt, and that's expected: they fetch the URL you give them. Robots.txt matters most for crawlers that discover URLs on their own, and that's where the defaults differ:

Scrapy's documentation explains the split: the built-in default is False "for historical reasons," but the option is enabled in the settings.py that scrapy startproject generates (Scrapy docs). So a Scrapy project started the standard way obeys robots.txt, and a one-file spider run with scrapy runspider does not. In our test, the bare script fetched the disallowed page and the project setup skipped it after requesting /robots.txt first.

Crawlee's option makes the crawler fetch each domain's robots.txt and keeps disallowed URLs out of the queue that enqueueLinks builds (Crawlee docs). Crawl4AI's check_robots_txt defaults to False and, when enabled, "checks and respects robots.txt rules before crawling" (Crawl4AI docs). Robots.txt isn't binding on its own in the US, though in the EU it can work as a text and data mining opt-out, as we covered in is robots.txt legally binding. But the EDPB's draft scraping guidelines tell scrapers to skip sites that clearly object through robots.txt, as we covered in our EDPB guidelines breakdown, so it's worth deciding on purpose rather than inheriting a default.

What a web scraper user agent can't hide

A web scraper user agent is one header. Sites also read the TLS handshake, HTTP/2 settings, header order, JavaScript-visible browser properties, and the IP address the request comes from. A perfect Chrome user agent sent from a cloud data center range can still look like automation, a common reason scrapers get blocked, as we explained in why AI agents get blocked on datacenter IPs.

That's the layer Massive handles. Every tool in our table accepts a proxy, and Massive's residential proxies route requests through real consumer devices in 195+ countries whose owners opted in through the Massive SDK, over HTTP, HTTPS, or SOCKS5. If you'd rather not manage a browser or its fingerprint at all, the Web Render API renders the page and returns HTML or Markdown.

Sources

• Massive, user agent and robots.txt test of 17 open-source tools (19 setups) against a local logging server, November 9, 2026 (first-party test; server, scripts, and raw logs available on request)
• Scrapy documentation, settings: ROBOTSTXT_OBEY and USER_AGENT
• Crawlee documentation, BasicCrawlerOptions (Node.js) and BasicCrawlerOptions (Python)
• Crawl4AI documentation, browser and crawler parameters
• IETF, RFC 9309: Robots Exclusion Protocol