# Web Scraper User Agents: What 17 Open-Source Tools Send by Default (We Tested)
A web scraper's user agent is the `User-Agent` header it sends with every request, and most open-source tools pick one for you. To see which, we pointed 17 popular scraping tools at a server we control on November 9, 2026, with default settings, and logged every header they sent. Ten of them name themselves, such as `python-requests/2.34.2` or `Scrapy/2.20.0`. Two announce a headless browser. Four send a string that looks like an ordinary desktop browser. And one sends no user agent at all.
We also served a robots.txt that disallowed one page. Across the 19 setups we ran, only one read robots.txt before crawling: Scrapy with `ROBOTSTXT_OBEY = True`, the value `scrapy startproject` writes into new projects. Here is what each tool sent, why it matters, and how to change it.
> **Key Takeaways**
>
> - **10 of 17 tools identify themselves by name** in their default user agent, including requests, HTTPX, aiohttp, Scrapy, and Axios.
> - **Playwright and Puppeteer say `HeadlessChrome`** in the user agent, and Playwright repeats it in the `sec-ch-ua` header, unless you override them.
> - **Four tools default to a browser-like string:** Scrapling, Crawl4AI, and both versions of Crawlee. Each can be set to a custom or rotating user agent.
> - **Only 1 of 19 setups read robots.txt.** Three of the four link-following crawler setups fetched the page our robots.txt disallowed.
> - **Every tool lets you set the user agent**, and every link-following crawler we tested has a robots.txt option, listed in the robots.txt table below.
## How we tested
We wrote a small Python server that listens on `127.0.0.1` and logs the path and every header of each request, in the order received. Its home page links to two normal pages and one page under `/private/`, and its robots.txt says:
ROBOTS.TXT
User-agent: *
Disallow: /private/
Each Python tool ran in its own fresh environment, and the Node.js tools ran from one npm project, with each third-party library at its latest PyPI or npm release, using the shortest code its documentation shows for fetching a page. For the four link-following setups (Scrapy twice and both versions of Crawlee), we started at the home page and let them crawl. We ran Scrapy twice: once as a bare script, which uses Scrapy's built-in defaults, and once with `ROBOTSTXT_OBEY = True`, the value `scrapy startproject` writes into a new project's settings file. We also ran curl_cffi twice, with and without its browser impersonation option.
Two limits apply. The server spoke plain HTTP/1.1 on the local machine, so we could not see TLS or HTTP/2 fingerprints, which sites also check. And Trafilatura refused to connect at all, because its downloader blocks non-public addresses as an SSRF protection. We took its user agent from its source code instead.
## Default web scraper user agents, tool by tool
| Tool (version) | Default user agent | Headers | robots.txt |
|---|---|---|---|
| curl (8.21.0) | `curl/8.21.0` | 3 | Not read |
| Python urllib (3.12) | `Python-urllib/3.12` | 4 | Not read |
| requests (2.34.2) | `python-requests/2.34.2` | 5 | Not read |
| HTTPX (0.28.1) | `python-httpx/0.28.1` | 5 | Not read |
| aiohttp (3.14.5) | `Python/3.12 aiohttp/3.14.5` | 4 | Not read |
| curl_cffi (0.17.0) | No user agent sent | 2 | Not read |
| curl_cffi with `impersonate="chrome"` | Chrome 152 on Windows | 17 | Not read |
| Scrapy (2.20.0), bare script | `Scrapy/2.20.0 (+https://scrapy.org)` | 5 | Not read; fetched the disallowed page |
| Scrapy, `ROBOTSTXT_OBEY = True` | `Scrapy/2.20.0 (+https://scrapy.org)` | 5 | Read; skipped the disallowed page |
| Scrapling Fetcher (0.5.1) | Chrome 152 on Windows | 18 | Not read |
| Crawlee for Python (1.11.1) | Firefox 128 on Linux | 11 | Not read; fetched the disallowed page |
| Crawlee for Node.js (3.18.2) | Edge 147 on Windows (first request; varies per request) | 15 | Not read; fetched the disallowed page |
| Crawl4AI (0.9.5) | Chrome 116 on Linux | 14 | Not read |
| Playwright (1.64.1) | `HeadlessChrome/156` on Windows | 13 | Not read |
| Puppeteer (25.16.0) | `HeadlessChrome/156` on Windows | 14 | Not read |
| Trafilatura (2.3.1) | `trafilatura/2.3.1 (+https://github.com/adbar/trafilatura)` | n/a | No request sent |
| Newspaper3k (0.2.8) | `newspaper/0.2.8` | 5 | Not read |
| Node.js fetch (24.19) | `node` | 7 | Not read |
| Axios (1.21.0) | `axios/1.21.0` | 5 | Not read |
"Headers" is the number of request headers on the first page request. Browser-like tools send 11 to 18, because they add headers such as `sec-ch-ua` client hints and `Sec-Fetch-*` that a real browser sends. Library defaults send 2 to 7. The counts for curl_cffi with impersonation (17) and Scrapling (18) include three HTTP/2 upgrade headers that curl adds on plain HTTP, an artifact of our test server.
## Ten tools tell the site exactly what they are
HTTP libraries name themselves by default, usually with a version number. That is honest, and it is also the easiest thing for a site to filter: a rule that matches `python-requests` or `axios` catches every default request from those libraries. Scrapy goes a step further and includes a URL, `+https://scrapy.org`, following the convention search engine crawlers use so site owners can find out who is visiting.
If you run a crawler at scale, that convention is worth keeping. A descriptive user agent gives robots.txt rules a name to match ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)), and its contact URL lets a site owner reach you instead of blocking you. Changing it is one line in every library: `headers={"User-Agent": ...}` in requests and HTTPX, the `USER_AGENT` setting in Scrapy, or a `headers` option in Axios and fetch. A descriptive one in requests looks like this:
PYTHON
import requests
UA = ("PriceBot/1.0 "
"(+https://example.com/bot)")
r = requests.get(
"https://example.com",
headers={"User-Agent": UA},
)
One outlier: curl_cffi sends no `User-Agent` header at all unless you ask it to impersonate a browser. With `impersonate="chrome"`, it sent 17 headers, with a Chrome 152 user agent and Chrome's client hints. Impersonation is the library's main purpose, so the bare default probably isn't what most people run.
## Two headless browsers say so
Playwright and Puppeteer launch Chromium in headless mode by default, and that browser announces itself. Both sent a user agent containing `HeadlessChrome/156`, and Playwright's `sec-ch-ua` client hint also listed `"HeadlessChrome";v="156"`. A site that looks for that token sees a browser automation tool in the first request.
Both let you replace the user agent: `user_agent` on a Playwright browser context, or `page.setUserAgent()` in Puppeteer. The client hint headers come from the browser itself, so a matching override needs more than the user agent string. Patched builds such as Patchright exist for this reason, which we listed in [the open-source web scraping stack](https://www.joinmassive.com/blog/open-source-web-scraping-tools).
## Four tools look like a browser out of the box
Scrapling, Crawl4AI, and both versions of Crawlee send a browser-style user agent by default, with the extra headers to match. Scrapling's fetcher sent Chrome 152 on Windows with 18 headers, close to what curl_cffi's impersonation produced, and added a `Referer` of `https://www.google.com/` by default. Crawlee generates a browser fingerprint for its sessions: our Node.js run rotated through browser identities across its four requests (Edge 147 on Windows first, then Chrome 147 on Windows and macOS), and our Python run stayed on Firefox 128 on Linux.
Crawl4AI's default user agent is a fixed Chrome-on-Linux string, sent with matching `sec-ch-ua` hints, and you can set it to match whatever browser and platform you run. Its `BrowserConfig` takes a `user_agent` string, or `user_agent_mode="random"` to pick from a pool ([Crawl4AI docs](https://docs.crawl4ai.com/api/parameters/)). Whatever tool you use, if you run it on a different machine or browser version than its default string describes, align the two.
## Only one setup read robots.txt
None of the HTTP libraries read robots.txt, and that's expected: they fetch the URL you give them. Robots.txt matters most for crawlers that discover URLs on their own, and that's where the defaults differ:
| Crawler | Reads robots.txt by default? | How to turn it on |
|---|---|---|
| Scrapy, bare script | No | `ROBOTSTXT_OBEY = True` |
| Scrapy, `startproject` project | Yes | Already on in the generated `settings.py` (we tested the same setting directly) |
| Crawlee for Node.js | No | `respectRobotsTxtFile: true` |
| Crawlee for Python | No | `respect_robots_txt_file=True` |
| Crawl4AI | No | `check_robots_txt=True` on `CrawlerRunConfig` |
Scrapy's documentation explains the split: the built-in default is `False` "for historical reasons," but the option is enabled in the `settings.py` that `scrapy startproject` generates ([Scrapy docs](https://docs.scrapy.org/en/latest/topics/settings.html)). So a Scrapy project started the standard way obeys robots.txt, and a one-file spider run with `scrapy runspider` does not. In our test, the bare script fetched the disallowed page and the project setup skipped it after requesting `/robots.txt` first.
Crawlee's option makes the crawler fetch each domain's robots.txt and keeps disallowed URLs out of the queue that `enqueueLinks` builds ([Crawlee docs](https://crawlee.dev/js/api/basic-crawler/interface/BasicCrawlerOptions)). Crawl4AI's `check_robots_txt` defaults to `False` and, when enabled, "checks and respects robots.txt rules before crawling" ([Crawl4AI docs](https://docs.crawl4ai.com/api/parameters/)). Robots.txt isn't binding on its own in the US, though in the EU it can work as a text and data mining opt-out, as we covered in [is robots.txt legally binding](https://www.joinmassive.com/blog/is-robots-txt-legally-binding). But the EDPB's draft scraping guidelines tell scrapers to skip sites that clearly object through robots.txt, as we covered in [our EDPB guidelines breakdown](https://www.joinmassive.com/blog/edpb-web-scraping-guidelines), so it's worth deciding on purpose rather than inheriting a default.
## What a web scraper user agent can't hide
A web scraper user agent is one header. Sites also read the TLS handshake, HTTP/2 settings, header order, JavaScript-visible browser properties, and the IP address the request comes from. A perfect Chrome user agent sent from a cloud data center range can still look like automation, a common reason scrapers get blocked, as we explained in [why AI agents get blocked on datacenter IPs](https://www.joinmassive.com/blog/why-ai-agents-get-blocked-on-datacenter-ips-and-how-to-fix-it).
That's the layer Massive handles. Every tool in our table accepts a proxy, and Massive's [residential proxies](https://www.joinmassive.com/solutions/proxies-for-web-scraping) route requests through real consumer devices in 195+ countries whose owners opted in through the Massive SDK, over HTTP, HTTPS, or SOCKS5. If you'd rather not manage a browser or its fingerprint at all, the [Web Render API](https://www.joinmassive.com/blog/introducing-massive-web-render-api) renders the page and returns HTML or Markdown.
## Frequently Asked Questions
### What user agent does Python requests use?
`python-requests/` followed by the library version, for example `python-requests/2.34.2` in our November 2026 test. Set a different one with `requests.get(url, headers={"User-Agent": "..."})` or on a `Session` object's headers.
### What is the best user agent for web scraping?
There isn't one string that works everywhere. For a crawler you run at scale, a descriptive user agent that names your bot and links to a contact page is the transparent choice. If you need a browser-like one, it should match the browser and platform your other signals present, and it should be a current version.
### Does Scrapy respect robots.txt?
Only if `ROBOTSTXT_OBEY` is `True`. New projects created with `scrapy startproject` turn it on in `settings.py`, but Scrapy's built-in default is `False`, so a spider run on its own with `scrapy runspider` ignores robots.txt unless you set it.
### How do I see what user agent my scraper sends?
Point it at a server you control and log the request headers, as we did, or at a header-echo endpoint. Check the `User-Agent` header and the `sec-ch-ua` client hints, since headless browsers can announce themselves in both.
## Sources
- Massive, user agent and robots.txt test of 17 open-source tools (19 setups) against a local logging server, November 9, 2026 (first-party test; server, scripts, and raw logs available on request)
- Scrapy documentation, [settings: ROBOTSTXT_OBEY and USER_AGENT](https://docs.scrapy.org/en/latest/topics/settings.html)
- Crawlee documentation, [BasicCrawlerOptions (Node.js)](https://crawlee.dev/js/api/basic-crawler/interface/BasicCrawlerOptions) and [BasicCrawlerOptions (Python)](https://crawlee.dev/python/api/class/BasicCrawlerOptions)
- Crawl4AI documentation, [browser and crawler parameters](https://docs.crawl4ai.com/api/parameters/)
- IETF, [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)