A weathered wooden sign on a lawn reading please keep off the grass in English and Japanese, with the headline Is robots.txt legally binding and the Massive logo.
All Posts

Is robots.txt Legally Binding? What US and EU Courts Have Ruled

Franklin Uche
Franklin Uche · Community Lead
Open markdown

Is robots.txt legally binding? In the United States, not on its own. In December 2025, a federal judge in New York held that robots.txt directives "are merely requests and do not effectively control access to copyrighted works," as quoted by law professor Eric Goldman (Technology & Marketing Law Blog).

In the European Union, the answer is different for AI training. A German appeals court ruled the same month that an opt-out from text and data mining only counts if it is machine-readable, and commentators read the ruling as pointing to robots.txt-style signals. So the common myth, that robots.txt is either a law or a meaningless text file, is wrong both ways. This post covers what each court decided, where robots.txt does carry legal weight, and what we found in the robots.txt files of 47 news sites.

This post explains public rulings. It is not legal advice.

Key Takeaways
  • The robots.txt standard itself, RFC 9309, says: "These rules are not a form of access authorization."
  • In Ziff Davis v. OpenAI (S.D.N.Y., December 15, 2025), the court held that ignoring robots.txt is not circumvention under the DMCA, comparing it to a "keep off the grass" sign.
  • In Kneschke v. LAION (Hamburg Higher Regional Court, December 10, 2025), the court held that a text-and-data-mining opt-out must be machine-readable to count under EU law. A Danish court followed that reasoning in 2026.
  • The EU's General-Purpose AI Code of Practice commits signatories to use crawlers that follow robots.txt.
  • In our check of 47 news sites, 15 put licensing terms or text-and-data-mining reservations in robots.txt comments, which crawlers skip. Only 3 used a machine-readable signal.

Where does the myth come from?

Robots.txt looks official. It sits at a fixed address on almost every site, and crawlers read it before anything else. Many files also contain legal-sounding comments. The BBC's file, for example, lists "No text and data mining (TDM) under Article 4 of the EU Directive on Copyright in the Digital Single Market" as a comment.

But the protocol was written as a courtesy system. RFC 9309, published in September 2022, standardized a practice from 1994. It describes rules that crawlers are "requested to honor," and states plainly that the rules "are not a form of access authorization" (RFC 9309). Whether a court gives those requests legal force depends on the law being applied.

Is robots.txt legally binding in the United States?

Not by itself. The clearest recent ruling is Ziff Davis v. OpenAI. Ziff Davis argued that OpenAI broke the DMCA's anti-circumvention rule by ignoring its robots.txt files. Judge Sidney Stein rejected that claim on December 15, 2025. The court said robots.txt files "do not 'effectively control' access to that content any more than a sign requesting that visitors 'keep off the grass' effectively controls access to a lawn" (Technology & Marketing Law Blog). Three days later, the court refused to let Ziff Davis amend that claim.

The ruling is narrow. It covers one statute, DMCA section 1201, and it is a district court decision, not an appeals ruling. Law professor Eric Goldman wrote that the opinion "doesn't answer the most important questions about robots.txt." Two other legal theories remain open:

  • Computer fraud law. After hiQ Labs v. LinkedIn and the Supreme Court's Van Buren decision, collecting public pages is generally not "unauthorized access" under the Computer Fraud and Abuse Act. RFC 9309's own wording makes robots.txt a weak basis for that claim. Our explainer on scraping versus hacking covers these cases.
  • Trespass to chattels and contract. In eBay v. Bidder's Edge (N.D. Cal. 2000), the court blocked a crawler under trespass to chattels. That case rested on eBay's user agreement, which barred robots, and on Bidder's Edge ignoring eBay's requests to stop (Berkeley Center for Law & Technology). Goldman notes the Ziff Davis ruling "casts a little doubt" on treating robots.txt as a gate for trespass claims.

The practical reading: in the US, robots.txt is evidence of what a site wanted. A contract, a direct notice, or a copyright claim is what turns that preference into a legal risk.

Is robots.txt legally binding in the EU?

For AI training, it can be the thing that makes an opt-out valid. EU copyright law allows text and data mining of lawfully accessible content, unless the rights holder has reserved those rights "in an appropriate manner, such as machine-readable means" for online content. That rule is Article 4(3) of the Digital Single Market Directive (EUR-Lex).

In Kneschke v. LAION, the Hamburg Higher Regional Court decided on December 10, 2025 that a written prohibition on a stock photo site did not count as an opt-out. The law prescribes no format, but the reservation must be understandable by machines, and this natural-language one "could not be automatically identified or processed" (Hogan Lovells AI case tracker). Commentators read that as pointing rights holders to "technical measures like robots.txt or metadata protocols" (newmind Journal). The case is now pending at Germany's Federal Court of Justice, so this may not be the last word.

A Danish court followed the same reasoning. In BoligPortal v. ReData, the Eastern High Court overturned an injunction against a scraper because the listing site's TDM reservation sat in a privacy policy linked from its footer. The court cited the Hamburg ruling and found a human-readable reservation insufficient (Plesner).

The EU AI Act adds another layer for model makers. The General-Purpose AI Code of Practice, published July 10, 2025, asks signatories to respect machine-readable rights reservations, including by "employing web-crawlers that follow robots.txt" (Freshfields). The Code is voluntary, but signing it is one way providers show they meet the AI Act's copyright duties.

Where robots.txt carries legal weight, as of October 2026.

Myth vs. fact: what robots.txt does and doesn't do

Claim Fact Source
Robots.txt is a law It is an IETF standard for crawler courtesy, and says it is "not a form of access authorization" RFC 9309
Ignoring robots.txt is hacking A US court held it is not DMCA circumvention; public pages are generally not "unauthorized access" under the CFAA Ziff Davis v. OpenAI (2025); hiQ, Van Buren
Robots.txt means nothing legally In the EU, it can be the machine-readable opt-out that blocks text and data mining for AI Kneschke v. LAION (2025)
A comment in robots.txt is an opt-out Crawlers skip comments. EU courts have rejected human-readable reservations, such as one in a privacy policy, and a comment is the same kind of text BoligPortal v. ReData (2026)
Blocking one AI crawler blocks a company's AI use Companies run several crawlers; each needs its own rule Meta's crawler documentation

What do news sites actually put in robots.txt?

On October 26, 2026, we tried to download the robots.txt files of 50 major news sites; 47 returned a valid file. We parsed those 47 to see which AI crawlers they block, a trend we covered in The Closing Web: AI Crawler Blocking and Agent Access. We then looked at how each site expresses legal terms:

  • 38 of 47 disallow ClaudeBot site-wide (35 by name) and 39 disallow Common Crawl's CCBot (36 by name). The rest block every crawler they don't list. AI crawler blocks are now standard.
  • 15 of 47 state licensing terms, licensing contacts, or reservations against text and data mining or AI use in comment lines. The New York Times, Financial Times, Reuters, and The Wall Street Journal point to licensing contacts. The BBC, Chicago Tribune, Al Jazeera, and Der Spiegel state Article 4 or AI reservations as comments.
  • Only 3 of 47 use a machine-readable licensing or usage signal. The Guardian publishes a License: line from Really Simple Licensing (RSL), an open standard for machine-readable license terms. Corriere della Sera and The Atlantic use Cloudflare's Content-Signal line, a robots.txt extension, with ai-train=no.

That gap matters under the EU rulings above. A comment is text for people. A Disallow rule for a named crawler, or a structured signal, is text for machines. For 13 of the 15 sites, the comment is the only place in the file where the terms appear. In every one of these files, though, the comments sit next to Disallow rules, either for named AI crawlers or for all crawlers by default.

What should data teams and AI builders do?

For teams that collect web data, the safe course is to treat robots.txt as a meaningful signal, not a technicality:

  1. Read it and log it. Save the robots.txt file and the date you fetched it, for every source you collect from at scale.
  2. Identify your crawler. Use a clear user agent, so a site's rules can actually address you.
  3. Separate uses. If you might train models on collected content, check EU sources for TDM reservations before you do.
  4. Check terms and notices. In the US, contract terms and direct cease-and-desist letters carry more legal weight than robots.txt alone.

Massive's network is built on the same idea from the other direction: every device on it opted in through the Massive SDK. Our post on whether residential proxies are legal explains why sourcing and use decide legality. For the wider legal picture on AI training data, see our roundup of AI training data lawsuits.

The bottom line

  • In the US, robots.txt is a request. Courts look to contracts, notices, and copyright for the binding part.
  • In the EU, robots.txt can be the machine-readable opt-out that limits text and data mining for AI.
  • If your terms matter to you, state them in a form machines can act on, not only in comments.

Sources

Frequently Asked Questions

Is robots.txt legally binding?+

Not on its own in the United States, where a federal court held in 2025 that robots.txt directives are requests, not access controls. In the European Union, a robots.txt rule can serve as a valid machine-readable opt-out from text and data mining for AI training.

Is it illegal to ignore robots.txt?+

Ignoring robots.txt is not illegal by itself in the US. Legal risk comes from other things, such as breaking a site's contract terms after notice, copying protected content, or causing harm to a site's servers. In the EU, ignoring a robots.txt opt-out can mean text and data mining is not covered by the copyright exception.

Is robots.txt legally enforceable?+

A site cannot sue over robots.txt as such. It can use robots.txt as evidence in copyright, contract, or trespass claims, and in the EU it can make a text-and-data-mining reservation effective.

Does a comment in robots.txt count as an opt-out?+

Probably not in the EU. Courts in Hamburg and Denmark have held that a reservation must be machine-readable. Crawlers ignore comment lines, so a rule that names the crawler, or a structured signal, is the safer form.

Does robots.txt stop AI training?+

Only for crawlers that read and follow it, and only for the user agents it names. Many AI companies run several crawlers for training, search, and user-requested fetches, so each needs its own group.