# EDPB Web Scraping Guidelines: What the GDPR Draft Means for AI Data Collection

The European Data Protection Board (EDPB) is finalising draft guidelines on web scraping. Guidelines 03/2026, adopted on July 7, 2026, explain how the GDPR applies when a company scrapes the web to train generative AI. Public comments close on **October 30, 2026** at 23:59 CET ([EDPB consultation page](https://www.edpb.europa.eu/public-consultations/guidelines-032026-on-web-scraping-in-the-context-of-generative-ai_en)).

The draft gives robots.txt, ai.txt, and CAPTCHA a real role in the legal test. It tells the company responsible for the scraping to leave out websites that "clearly oppose the scraping of their content" through those measures. So we checked what 47 major news sites signal today. 42 block at least one AI training crawler in robots.txt. None of the 38 we could read publishes an ai.txt file.

> **Key Takeaways**
>
> - The EDPB's draft Guidelines 03/2026 cover private companies that scrape the web, or hire a scraper, to train generative AI. Comments close **October 30, 2026**. Reed Smith does not expect the final version before the end of 2026.
> - Consent is "most probably not" a usable legal basis. Most companies will rely on legitimate interest, which has three conditions: a real interest, necessity, and a balancing test.
> - robots.txt, ai.txt, and CAPTCHA count. The draft says to exclude sites that clearly oppose scraping, and that a missing robots.txt file "does not amount to consent."
> - In our check of 47 news sites, **42 block at least one AI training crawler** site-wide, 18 block all four we tested, and **none of the 38 readable sites serve an ai.txt file**.
> - The draft also asks for a public list of sources, with domains and collection dates where possible. Teams that log what they collect, and when, will find most of these steps easier.

*This post summarizes a draft regulatory text. It is not legal advice.*

## What are the EDPB web scraping guidelines?

The EDPB is the body of EU data protection authorities that issues guidance on how to apply the GDPR. Its draft Guidelines 03/2026 "on web scraping in the context of generative AI" set out what the GDPR requires when personal data ends up in scraped training data ([EDPB, Guidelines 03/2026](https://www.edpb.europa.eu/system/files/2026-07/edpb_guidelines_2020603_webscraping_v1_en_0.pdf)).

The scope is narrower than the title suggests:

- **Who:** private entities only (paragraph 6).
- **What:** scraping to train or fine-tune generative AI, either done in-house or by a contracted party, and the reuse of a dataset someone else scraped (paragraph 3).
- **Not covered:** data brokers that scrape and hold datasets without training models themselves, and an organisation's own data (paragraphs 4 and 5). The EDPB notes that those situations are still subject to data protection law.

The draft defines web scraping as automated extraction from "publicly available web services," and lists public registers, open-data portals, news outlets, social media, forums, and blogs as examples. It adds that "many of these websites are likely to contain personal data" (paragraph 7). It also separates targeted scraping, run against defined collection criteria, from untargeted crawling that follows every link it finds. The second, it says, "increases the risk that the controller has limited knowledge of what personal data are collected" (paragraph 8).

## Is web scraping GDPR compliant?

It can be, but the draft sets a high bar. The GDPR applies whenever scraping involves personal data, and in a "mixed" dataset it applies to the personal data part (paragraph 13). Public availability does not change that.

**Consent is effectively out.** The draft says consent "would most probably not be an applicable legal basis," because scrapers have no direct relationship with the people whose data they collect. Posting data on a public page "does not mean that the data subjects gave their consent to the scraping." And "the absence or non-applicability of a robots.txt file on a web site does not amount to consent within the meaning of the GDPR" (paragraphs 44 and 45).

**Legitimate interest is the realistic basis,** and it has three cumulative conditions (paragraph 47):

| Condition | What the draft asks | Paragraphs |
|---|---|---|
| 1. Legitimate interest | A lawful, clearly articulated, real and present interest. Examples the EDPB has said may qualify: a conversational agent, fraud detection, threat detection. | 48 to 50 |
| 2. Necessity | Scraping personal data must serve the purpose, with no equally effective, less intrusive option. Narrow collection criteria, pseudonymised data, or synthetic data can help. | 51 and 52 |
| 3. Balancing test | The company's interest must not be overridden by people's rights and reasonable expectations. The more sensitive the data and the larger the scrape, the harder it gets. | 53 to 64 |

The balancing test is where most of the practical detail sits. The draft points to the "chilling effect" of indiscriminate collection, the difficulty of exercising rights against a third-party scraper, and the fact that "personal data cannot be easily deleted from a model" once it is trained (paragraphs 54 and 60).

## What do the guidelines say about robots.txt, ai.txt, and CAPTCHA?

They come up in several places. Paragraph 11 lists robots.txt and CAPTCHA in the first step of a typical scraping project, and paragraph 45 says a missing robots.txt file is not consent. Three more bear directly on the controller:

1. **As a data minimisation step.** Before collection, the controller should "exclude from the collection websites which clearly oppose the scraping of their content," through measures "such as the use of robots authentication to view content, robots.txt or ai.txt files, or CAPTCHA" (paragraph 37).
2. **In reasonable expectations.** If people know a site uses these measures "and a website is nevertheless scraped, it is less likely that they can expect the processing of their personal data by a scraping entity" (paragraph 64).
3. **In the worked examples.** On a platform that blocks scraping with robots.txt and CAPTCHA and says it does not allow AI training, users "cannot reasonably expect third parties to scrape that data for that purpose." On an open platform that tells users their content may be scraped, they can (examples 4 and 5).

Login walls come up too. Paragraph 37 lists "robots authentication to view content" among the opposition measures, and one optional mitigation is to exclude personal data that is "only accessible when logged into the site" (paragraph 66). The practical reading for data teams: a robots.txt disallow aimed at AI training crawlers is evidence a regulator can weigh, even though, as a text file, it does not technically stop anyone. Crawlers that follow [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) check it before they crawl, and the draft treats ignoring it as a point against the scraper in the GDPR analysis.

## What do 47 news sites signal to scrapers today?

On October 26, 2026, we downloaded the robots.txt files of 50 major news sites from the US, UK, Europe, Canada, and Australia, plus Al Jazeera. 47 returned a valid file. We applied the group-matching rules in RFC 9309 (a group that names the crawler wins; otherwise the `*` group applies) and counted a crawler as blocked when the rules that apply to it disallow the whole site. We checked four AI training crawlers: OpenAI's GPTBot, Common Crawl's CCBot, Anthropic's ClaudeBot, and Google-Extended, the token Google uses for AI training controls.

On October 27, we requested `/ai.txt` from the same 47 sites, on the bare domain and on `www`. ai.txt is a robots.txt-style file for AI training preferences, published by Spawning in 2023 ([Spawning](https://spawning.substack.com/p/aitxt-a-new-way-for-websites-to-set)). It is a convention, not a ratified standard. We counted a file as present only if it returned a non-HTML file with at least one user-agent, allow, or disallow line.

![Chart of what 47 news sites signal to scrapers: 42 of 47 block at least one AI training crawler in robots.txt, 18 block all four checked, 5 block none; 0 of 38 readable sites serve an ai.txt file, and 9 more refused the request or timed out.](assets/edpb-scraping-signals.png "robots.txt checked October 26, 2026; ai.txt checked October 27, 2026.")

Three results stood out:

1. **Opposition through robots.txt is the norm.** 42 of 47 sites disallow at least one of the four AI training crawlers across the whole site, and 18 disallow all four. For those 18, the file reads as clear opposition to AI training. For the 24 that block only some, the draft does not say whether a crawler-specific rule counts as opposition to everyone.
2. **The block is often crawler by crawler.** 41 sites name at least one of the four crawlers in their own group. Paragraph 37 asks whether a site "clearly" opposes scraping. It does not say a rule has to name your bot, so treating a block aimed at another crawler as irrelevant to you is a risky reading.
3. **ai.txt has not caught on.** None of the 38 sites that gave a readable answer serves an ai.txt file: 35 returned a 404 and 3 returned an HTML page. The other 9 refused our request (401, 402, or 403) or timed out, so we can't rule them out.

A robots.txt rule is a request, not a lock, so these counts show what publishers asked for, not what any crawler does. Our survey scripts and per-site results are available on request.

## Controller, processor, or joint controller: who is responsible?

The draft says the role of each company "should be analysed on a case-by-case basis," and that "the organisation performing the scraping is not necessarily the controller" (paragraphs 16 and 17). It describes three common setups:

- **A scraper builds a dataset on your instructions.** If an AI developer sets the sources and data categories, the scraper "may be regarded as a processor" and the developer is the controller (paragraph 18).
- **You train on a dataset someone else scraped.** Each company is responsible for its own processing, and the scraper "is not, in principle, responsible for the re-use of the data" (paragraph 19).
- **Two companies decide together.** If they set the purposes and collection criteria jointly, they are joint controllers (paragraph 20).

For buyers of scraped or licensed datasets, the transparency section adds a practical ask: tell people how to contact the controller that did the scraping, and link to it where possible (paragraph 32).

## What should data teams do now?

The draft lists measures for each stage. Most of them map to decisions a data team already makes when it scopes a crawl:

| Stage | What the draft suggests | Paragraph |
|---|---|---|
| Before collection | Define precise collection criteria. Map and inventory the data. Filter out categories you don't need (for example financial or location data). Exclude sites mainly used by minors, and sites that clearly oppose scraping through robots.txt, ai.txt, or CAPTCHA. Consider synthetic data. | 37 |
| During and after | Filter identifiers by format with regular expressions, such as phone or social security numbers. Anonymise or pseudonymise where feasible. Delete unnecessary personal data as soon as possible. | 38, 66 |
| Accuracy | Scrape from reliable, maintained sources. Timestamp what you collect. Validate formats and spot-check samples before training. | 42 |
| Transparency | If you can't inform people one by one, publish the information instead: data categories, purposes, legal basis, and the sources, including the crawler's characteristics. Where possible, list domains and URLs in a searchable format with collection dates. | 30, 31 |
| Rights | Consider a prior right to object, or an opt-out list that lets people exclude themselves before collection. | 66 |

Special categories of data, such as health, political opinion, or ethnic origin, get their own section. Incidental collection may be defensible under the reasoning of the EU Court of Justice's GC & Others ruling, but only where the processing resembles a search engine's and the collection is truly incidental and residual. The EDPB stresses the ruling is not "a general exemption" (paragraphs 72 and 73). Even then, the controller needs at least filters before collection, deletion as soon as the data is found, extraction testing during training, and output monitoring after release (paragraph 74).

Several of these steps depend on records, not on legal judgment: which sources you collected from, which crawler identity you used, what each site's robots.txt said, and when. A team that keeps that log can publish a source list and answer a regulator's question. A team that doesn't will struggle with both. Our coverage of [Cloudflare's AI crawler defaults](https://www.joinmassive.com/blog/cloudflare-ai-crawler-defaults) points the same way: sites are separating training, search, and agent traffic, and asking each to identify itself.

Massive sees the sourcing side of this. Every device on our network opted in through the Massive SDK, as our [explainer on how that opt-in works](https://www.joinmassive.com/blog/how-massives-opt-in-network-works) describes. Our guides to [ethical web scraping](https://www.joinmassive.com/blog/ethical-web-scraping) and [proxy vendor compliance](https://www.joinmassive.com/blog/proxy-vendor-compliance-soc2-gdpr) cover the questions to ask any data or network provider. None of this is legal advice; for a specific project, talk to a data protection lawyer.

## How do you comment on the draft?

The consultation is open until 23:59 CET on October 30, 2026, on the [EDPB's consultation page](https://www.edpb.europa.eu/public-consultations/guidelines-032026-on-web-scraping-in-the-context-of-generative-ai_en). The EDPB adopted draft [Guidelines 02/2026 on anonymisation](https://www.edpb.europa.eu/public-consultations/guidelines-022026-on-anonymisation_en) on the same day, with the same deadline. Reed Smith does not expect the final web scraping version before the end of 2026 ([Reed Smith](https://www.reedsmith.com/our-insights/blogs/technology-law-dispatch/102nbqu/edpb-web-scraping-guidelines-for-ai-making-the-impossible-possible/)), and the text can still change.

## Frequently Asked Questions

### What are the EDPB web scraping guidelines?

They are draft Guidelines 03/2026 from the European Data Protection Board, adopted on July 7, 2026, on how the GDPR applies to web scraping for training generative AI. They cover private companies that scrape the web themselves, hire a scraper, or reuse a scraped dataset.

### Is web scraping legal in Europe?

Web scraping is not banned in the EU, but scraping personal data must comply with the GDPR. Under the EDPB's draft, consent is unlikely to work, so most companies would need to pass the three-part legitimate interest test and apply safeguards such as narrow collection criteria, filtering, and a published list of sources. Copyright and website terms raise separate questions.

### Does robots.txt matter under the GDPR?

Yes, according to the draft. A robots.txt file that opposes scraping is a signal the scraper should respect, and scraping a site anyway makes it less likely people expected their data to be processed. A missing robots.txt file "does not amount to consent."

### When is the deadline for comments on the EDPB guidelines?

October 30, 2026, at 23:59 CET, through the EDPB's public consultation page. Reed Smith does not expect the final version before the end of 2026.

### Do the guidelines apply to web scraping that isn't for AI?

The draft focuses on scraping to train generative AI. Other scraping that involves personal data is still covered by the GDPR, but the draft's specific guidance was written for AI training.

## The bottom line

- The EDPB's draft web scraping guidelines close for comment on October 30, 2026. Legitimate interest is the likely legal basis, and it comes with a three-part test.
- robots.txt, ai.txt, and CAPTCHA count as opposition. 42 of 47 news sites block at least one AI training crawler; none of the 38 we could read uses ai.txt.
- Keep a dated record of sources, crawler identity, and robots.txt rules. Most of the draft's transparency and minimisation steps start there.

## Sources

- European Data Protection Board, ["Guidelines 03/2026 on web scraping in the context of generative AI," version 1.0](https://www.edpb.europa.eu/system/files/2026-07/edpb_guidelines_2020603_webscraping_v1_en_0.pdf), adopted July 7, 2026
- European Data Protection Board, [public consultation page for Guidelines 03/2026](https://www.edpb.europa.eu/public-consultations/guidelines-032026-on-web-scraping-in-the-context-of-generative-ai_en), open July 8 to October 30, 2026
- European Data Protection Board, [public consultation page for Guidelines 02/2026 on anonymisation](https://www.edpb.europa.eu/public-consultations/guidelines-022026-on-anonymisation_en)
- Spawning, ["ai.txt: A new way for websites to set permissions for AI"](https://spawning.substack.com/p/aitxt-a-new-way-for-websites-to-set), 2023
- Reed Smith, ["EDPB web scraping guidelines for AI: Making the impossible possible?"](https://www.reedsmith.com/our-insights/blogs/technology-law-dispatch/102nbqu/edpb-web-scraping-guidelines-for-ai-making-the-impossible-possible/), July 14, 2026
- IETF, [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html), September 2022
- Massive, robots.txt survey of 47 news sites (October 26, 2026) and ai.txt check of the same sites (October 27, 2026), first-party tests; scripts and per-site results available on request