sota.io
Join the waitlist
2026-07-27·13 min read·sota.io team

EDPB's Web Scraping Guidelines for Generative AI: The Legitimate Interest Test You Actually Have to Pass

EDPB's Web Scraping Guidelines for Generative AI: The Legitimate Interest Test You Actually Have to Pass

On 7 July 2026, the European Data Protection Board adopted Guidelines 03/2026 on web scraping in the context of generative AI — the first EDPB document that walks through, provision by provision, what happens when you point a crawler at the open web to build a training dataset. Version 1.0 is open for public consultation until 30 October 2026, 23:59 CET.

If your product trains, fine-tunes, or builds retrieval pipelines on scraped web content — or if you resell a dataset that someone else scraped — this document is the closest thing to an operating manual EU data protection authorities have published for you. It will not be final until after the consultation, and some wording may shift. But the core legal reasoning (Article 6(1)(f) as the realistic legal basis, a strict three-part test, robots.txt/ai.txt as evidence of reasonable expectations) closely follows the EDPB's own Guidelines 1/2024 on Article 6(1)(f) adopted in October 2024, so it is very unlikely to reverse. Treat this as the operative reading today, not a draft you can ignore until the consultation window closes.

This post breaks the guidelines down into what a developer building or buying a scraping pipeline actually needs to check, with the paragraph numbers so you can verify each claim against the source PDF yourself.

What the Guidelines Cover — and What They Don't

The guidelines define two typical scenarios (paragraph 3):

  1. An organisation scrapes data from the internet itself, or contracts another party to do it, in the context of developing generative AI.
  2. An organisation obtains and reuses a dataset that has already been scraped by another organisation — a data broker.

They explicitly do not address the role of data brokers who scrape and hold datasets without training AI themselves (paragraph 4), and they focus only on data acquired from sources external to the organisation — not an org's own first-party data (paragraph 5). They also cover only web scraping by private entities (paragraph 6), since that is who mostly does this for generative AI.

The GDPR applies the moment scraping involves processing operations — collection, storage, organisation, retrieval — on data that identifies or can be linked to an identifiable individual, directly or indirectly (paragraph 12). Crucially, if you scrape a "mixed" dataset containing both personal and non-personal data, GDPR governs the personal-data portion of the whole dataset — you don't get to argue the dataset is "mostly non-personal" and skip compliance (paragraph 13).

Two definitions worth internalizing because the guidelines use them throughout:

The guidelines also break the scraping pipeline into four steps that map directly onto how most crawling infrastructure is actually built: defining collection criteria → extraction → cleaning → structuring and storing (paragraph 11). Each step is where a different GDPR principle attaches — collection criteria is where minimisation lives, cleaning is where accuracy lives, and so on. Design your compliance controls around this pipeline shape, not as a bolt-on audit afterward.

Who Is the Controller? (It's Not Automatically the Scraper)

If you buy scraped data or outsource the scraping, the guidelines set out a case-by-case test (paragraphs 16-20):

The practical implication: buying a "GDPR-compliant" dataset from a vendor does not transfer their compliance work to you. You are a separate controller the moment you decide the purposes of your training run, and you inherit the full obligations below — lawfulness, transparency, minimisation, accuracy, and legal basis — for your own processing.

Why Consent Is Dead as a Legal Basis Here

The guidelines are blunt about this (paragraphs 44-45): consent under Article 6(1)(a) is "most probably not" applicable, because scraping at scale collects data indirectly, without a direct relationship to the data subject, at a scale that makes obtaining individual consent from every person in the dataset effectively impossible. And critically: the absence of a robots.txt file blocking your crawler does not amount to consent. Non-opposition is not consent under the GDPR — don't build a compliance argument that treats "they didn't block us" as "they agreed."

That leaves legitimate interest under Article 6(1)(f) as the realistic path for most AI-training scraping by private entities (paragraph 43). The rest of this post is about what that legal basis actually demands.

The Three-Part Legitimate Interest Test

Article 6(1)(f) requires three cumulative conditions (paragraph 47). Fail any one, and you cannot rely on legitimate interest.

Condition 1 — A real, articulated interest

The interest must be lawful, "clearly and precisely articulated and real and present, not speculative" (paragraph 49). The EDPB gives three examples that have previously qualified: developing a conversational agent, detecting fraudulent content, improving threat detection (paragraph 50). For general-purpose model development where the eventual use case isn't fixed yet, you should still articulate the objective — commercial, public-interest, or scientific research — and whether it's internal or external to your organisation.

Write this down. "We might need training data eventually" is speculative. "We are training a domain-specific summarization model for legal documents, internal use, commercial purpose" is articulated.

Condition 2 — Necessity

Two sub-questions (paragraph 51): does the processing actually serve the interest, and is there an equally effective, less intrusive way to pursue it? For AI training specifically, the guidelines call out narrowing your collection criteria and using pseudonymised or synthetic data as ways to satisfy necessity instead of scraping broadly (paragraph 52). If you're crawling the entire open web when a curated, filtered subset would train the model just as well, that's a necessity problem, not just a minimisation nicety.

Condition 3 — The balancing test

This is where most of the operational work lives (paragraphs 53-66). You weigh the data subject's interests, rights, and freedoms against your legitimate interest, considering:

The nature of the data (paragraphs 55-56) — special categories and criminal-record data get maximum weight, but so does other "highly private" data like location or financial information, and data about minors. The more sensitive, the more it weighs against you.

The context of processing (paragraphs 57-58) — large volume, large number of data subjects, and indiscriminate collection all increase the interference. Since scraped web content is rarely deleted, historical depth of the dataset matters too.

Consequences for data subjects (paragraphs 59-61) — a key one for AI specifically: "given the current technical state of the art, once the model is trained, personal data cannot be easily deleted from a model" (paragraph 60). That irreversibility is itself a factor the EDPB weighs against you.

Reasonable expectations (paragraphs 62-64) — this is the section with the most direct engineering relevance. Reasonable expectations don't automatically exist just because you published a privacy notice (paragraph 63). Instead, the EDPB looks at:

Two contrasting examples the EDPB gives directly (paragraph 64, Examples 4-5): a platform that is freely accessible and communicates that scraping is possible to its users supports a reasonable-expectation finding for the controller. A platform that blocks scraping via robots.txt and CAPTCHA, and expressly states it doesn't allow AI training use of its data, means users cannot reasonably expect third-party AI scraping — and a controller ignoring those signals weakens their own balancing-test position.

robots.txt and ai.txt are not a technicality here — they are evidence the regulator explicitly weighs in your favor or against you. If your crawler infrastructure doesn't check and respect these files as a hard gate, you are giving up a factor the guidelines put directly in your control.

A Pre-Scrape Compliance Gate You Can Actually Build

The guidelines list concrete "mitigating measures" a controller should implement before, during, and after collection (paragraphs 37-38, 65-66). Turned into engineering requirements, before collection you should:

During and after collection:

A minimal pre-scrape gate implementing the parts you control programmatically looks like this:

import re
import time
from urllib.robotparser import RobotFileParser
from urllib.parse import urlparse

SENSITIVE_URL_PATTERNS = [
    r"/kids?/", r"/minors?/", r"/children/",     # sites structurally aimed at minors
    r"/health-records?/", r"/patients?/",         # likely Art.9 special-category data
]

PII_STRIP_PATTERNS = {
    "email": re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+"),
    "phone": re.compile(r"\b(?:\+?\d[\d\-\s]{7,}\d)\b"),
    "national_id": re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),  # adapt per jurisdiction
}

def robots_and_ai_txt_allow(url: str, user_agent: str) -> bool:
    """EDPB Guidelines 03/2026, paragraph 64: a site's robots.txt/ai.txt
    opposition to scraping is direct evidence against a reasonable-
    expectation finding. Hard-block on disallow, don't just log it."""
    parsed = urlparse(url)
    base = f"{parsed.scheme}://{parsed.netloc}"
    rp = RobotFileParser()
    rp.set_url(f"{base}/robots.txt")
    try:
        rp.read()
        if not rp.can_fetch(user_agent, url):
            return False
    except Exception:
        pass  # unreachable robots.txt: apply your own conservative default
    # ai.txt is not yet a formal standard, but the EDPB treats it as a
    # signal equivalent to robots.txt (para 37, 64) — check it the same way
    return True  # extend with an ai.txt parser using the same disallow logic

def pre_scrape_gate(url: str, user_agent: str) -> bool:
    if any(re.search(p, url) for p in SENSITIVE_URL_PATTERNS):
        return False
    return robots_and_ai_txt_allow(url, user_agent)

def clean_record(text: str, source_url: str) -> dict:
    for label, pattern in PII_STRIP_PATTERNS.items():
        text = pattern.sub(f"[REDACTED_{label.upper()}]", text)
    return {
        "text": text,
        "source_url": source_url,
        "collected_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }

This is a starting skeleton, not a compliance certificate — the guidelines expect a documented, case-by-case assessment (paragraph 73), not just code. But every check above corresponds to a specific paragraph the EDPB will look for if a supervisory authority ever asks how you determined the balancing test came out in your favor.

Transparency: When You Can Skip Individual Notices

Article 14 normally requires informing data subjects when you didn't collect data directly from them. The guidelines acknowledge that individually notifying millions of scraped data subjects is often "impossible or requires disproportionate effort" — and Article 14(5)(b) provides exactly that exemption (paragraph 25). But this exemption is not a blanket pass: the impossibility/disproportionate-effort standard applies "in particular" to archiving, scientific/historical research, or statistical purposes under Article 89(1) safeguards, and "should not be routinely relied upon" outside those contexts (paragraph 27).

Where you do rely on it, Article 14(5)(b) itself requires one thing you must always do: make the information publicly available (paragraph 30) — typically a privacy notice on your site that includes:

If you're reusing a dataset scraped by someone else, EDPB good practice is to also link to the original scraper's own disclosure and explain the conditions under which the data was originally collected (paragraph 32).

Special Category Data: You Can't Fully Prevent It, But You Must Try

Article 9 prohibits processing special categories of data (racial/ethnic origin, political opinion, health, sexual orientation, etc.) without a derogation under Article 9(2), in addition to your Article 6 legal basis. The problem the EDPB acknowledges directly: it's often impossible to know whether a scrape picked up special-category data until after the scraping already happened (paragraph 68).

The guidelines borrow reasoning from the CJEU's GC and Others v CNIL judgment (C-136/17), originally about search engines, holding that the Article 9(1) prohibition applies to a controller only "within the framework of his responsibilities, powers and capabilities" (paragraph 70-72). For web scraping, the EDPB says this reasoning can apply — case by case, not as a blanket exemption — where four conditions are met (paragraph 73):

If those four hold, you still need lifecycle measures across four phases (paragraph 74): filter categories and exclude structurally sensitive sites before collection; delete any special-category data identified after collection, including on a data subject's request; train with resistance to extraction attacks and apply output filters during model development; and continuously monitor and filter production outputs after deployment, with model unlearning as a longer-term mitigation as the technology matures. You also need to be able to demonstrate, per the accountability principle, that all four conditions were relevant to your specific case and that your measures were actually effective (paragraph 75) — and re-verify that regularly as your scraping and model evolve (paragraph 76).

This is not a loophole. It's a narrow, fact-specific defense that only works if you can show your filters were real and your monitoring is ongoing — not a checkbox you tick once.

What to Do This Quarter

  1. Map your training-data pipeline against the four-step model (collection criteria → extraction → cleaning → structuring/storing) and identify which GDPR principle attaches to each step.
  2. Write down your legitimate interest — specific, not speculative — for every dataset you're building or buying.
  3. Add a hard robots.txt/ai.txt/CAPTCHA gate to any crawler you operate. Don't just log violations, block on them — this is a documented factor in the balancing test, not a nice-to-have.
  4. Audit any purchased or reused datasets — you are a separate controller for your own training run, and the seller's compliance work does not cover you.
  5. Publish (or update) a public scraping/training-data notice with source categories, legal basis, and — to the extent feasible — a list of scraped domains, per Article 14(5)(b).
  6. Document your special-category mitigation lifecycle if there's any realistic chance your dataset touches Article 9 data — before/after collection, during training, and in production output filtering.
  7. If you operate an EU-facing product, consider submitting feedback before the consultation closes on 30 October 2026 — the guidelines are still version 1.0.

Related on this blog: how Article 10 training data governance and Article 10 data provenance logging intersect with the EU AI Act's own high-risk training-data rules, how GPAI copyright compliance for training data covers the separate text-and-data-mining opt-out regime under Directive 2019/790, and our breakdown of the EDPB's common breach notification template if a scraping pipeline incident ever needs Article 33 reporting.

Primary Sources

EU-Native Hosting

Ready to move to EU-sovereign infrastructure?

sota.io is a German-hosted PaaS — no CLOUD Act exposure, no US jurisdiction, full GDPR compliance by design. Deploy your first app in minutes.

Join the waitlistView plans