The Scraping Era Is Over

For most of AI’s recent history, the default approach to training data has been simple: take what you can find. Scrape the web. Download whatever is publicly accessible. Normalize later. Litigate if necessary.

The scale of this practice is hard to overstate. Common Crawl’s January 2025 corpus alone covered roughly 3 billion pages across 39 million domains.[1] LAION-5B, one of the most influential open image-text datasets in generative AI, contained 5.85 billion image-text pairs.[2] Cloudflare reported that by mid-2025, AI training crawling accounted for nearly 80% of all AI bot activity on its network, up from 72% the year before.[3] TollBit reported that scraping on publisher sites in its network rose 87% quarter over quarter, and that 26 million scrapes bypassing robots.txt were observed in a single month.[4]

This was not a marginal practice at the edges of the industry. It was central to how the field grew. And for a while, it seemed to work. Models got bigger, benchmarks went up, and nobody asked too many questions about where the data came from.

That era is ending.

Three things happened at once, and together they are reshaping the economics and ethics of AI training data.

The legal ground shifted. There are now over 70 copyright infringement lawsuits pending against AI companies in the United States alone.[5] In 2025, Anthropic settled with a group of authors for $1.5 billion, one of the largest copyright settlements in U.S. history, after a court found that while training on lawfully obtained books may qualify as fair use, acquiring those books from pirated sources does not.[6] Warner Music settled with AI music generator Suno and pivoted into a licensing partnership.[7] Disney, Universal, and Warner Bros. sued AI video generators for training on their copyrighted content.[5] On March 9, 2026, the court overseeing the consolidated OpenAI litigation ordered the company to produce training data details, a discovery ruling that could expose the sourcing practices behind the industry's most widely used models.[13] Courts are signaling that the question is shifting from "is AI training fair use?" to "how was the data obtained, and was the source legitimate?"[6]

Globally, the picture is fragmenting. The EU AI Act now requires general-purpose AI providers to publish detailed summaries of their training data. The UK backed away from a permissive opt-out framework after opposition from publishers and the creative industry. The U.S. Copyright Office released a major report on generative AI training in early 2026, and legal experts are warning of a decade of uncertainty ahead.[8]

The provenance problem became visible. A Nature Machine Intelligence audit found that more than 70% of licenses for popular datasets on GitHub and Hugging Face were "unspecified."[9] The Data Provenance Initiative's multimodal audit found that by amount of content, less than 0.1% of text, 5.4% of speech, and 0.6% of video in widely used datasets were actually usable for commercial purposes under both their licenses and their underlying source terms.[10] A separate ICLR audit of nearly 4,000 datasets found that while fewer than one-third are restrictively licensed on paper, more than 80% of the underlying source content carries non-commercial restrictions.[10]

That is not a healthy ecosystem. It is a documentation failure at industrial scale. And it means that companies training on these datasets are carrying legal and operational risk they may not even be aware of.

The quality argument caught up. This is the part that gets less attention but matters most for where the industry is heading. Raw scraped data is not simply "more data." It is duplicated, unevenly structured, contaminated by boilerplate, mixed with spam, and only weakly aligned with downstream training objectives. The same research literature powering state-of-the-art open datasets increasingly says it plainly: curation matters, and it matters a lot. FineWeb, DataComp, SoftDedup, and D4 all show that better filtering, deduplication, and source selection improve model performance while reducing compute.[11]

So the scrape-everything approach is not just legally fragile and ethically strained. It is technically lazy. And as training costs scale into the hundreds of millions of dollars per run, that laziness becomes expensive.

And here is the part almost nobody is talking about yet: scraping has a ceiling, and the AI industry just hit it. Text could be scraped. Images could be scraped. Video could be pulled from the web, but look at what you actually get when you scrape gameplay from streaming platforms: picture-in-picture webcam feeds, heads-up displays, donation banners, chat overlays, notification popups, branded watermarks. That is not clean training data. It is noise layered on top of signal, and every contaminated frame wastes compute or degrades the model. Even if you strip the overlays, you still have screen-level footage with no information about what is happening underneath. The data that world models actually need was never on the internet to begin with. Engine telemetry. Physics contacts. Player inputs synchronized to the frame. Camera position and rotation in 3D space. Action-outcome pairs across interactive environments. None of that exists in a Twitch stream or a YouTube clip. It lives inside game engines, and it can only be captured there, at the engine level, by teams that know how to do it.

The licensing market is not mature yet, but even the biggest players are conceding that high-value data cannot remain an unpriced commons forever.

OpenAI has signed publisher partnerships with the AP, Hearst, Conde Nast, the Guardian, Schibsted, and Stack Overflow. Shutterstock licensed its full library to OpenAI under a multi-year agreement. The New York Times signed a licensing deal with Amazon in 2025. Reddit licensed its data to Google and OpenAI while simultaneously suing Anthropic for scraping without a license. Google, Apple, and others are quietly building content licensing teams that did not exist two years ago.[12]

The pattern is clear. Litigation leads to negotiation. Negotiation leads to structured licensing. And structured licensing leads, inevitably, to a market where data quality, provenance, and rights clarity become differentiators rather than afterthoughts.

This is the argument we care about most, because it is the one the industry has been slowest to internalize.

Rights-cleared data is not just data that will not get you sued. It is data with a known source, a defined scope, a documented chain of custody, and a relationship with the people who created it. That relationship means you can go back to the source when you need more. You can negotiate for specific coverage, specific formats, specific distributions. You can build refresh cycles into your pipeline instead of hoping the next web crawl catches what you missed.

For world models in particular, where training data needs to include not just images or video but action labels, engine telemetry, synchronized sensor streams, and structured metadata across dozens of categories, the difference between scraped media and purpose-built licensed data is not incremental. It is structural. Scraped video does not carry player inputs. It does not include camera state or physics contacts. It has no frame-level annotations. It cannot be replayed, queried, or filtered by scene type, weather condition, or interaction pattern.

Licensed capture from structured environments can do all of those things. And it can do them because the data relationship was designed from the start, not extracted after the fact.

Video game publishers and developers operate some of the most complex, structured, interactive digital environments ever built. Open worlds with realistic physics. Dynamic weather and lighting. Emergent player behavior across millions of sessions. Environments where every action produces a measurable consequence and every frame carries spatial, temporal, and causal information. A recent paper from University College London described game-generated data as "an untapped resource for advanced AI training," noting properties like causal relationship preservation, multimodal temporal alignment, and emergent complexity generation that address fundamental limitations in current approaches.[14]

This is exactly what the next generation of AI needs to learn from. And right now, the game industry is getting almost zero value from it.

The licensing deals making headlines are between AI companies and text publishers: newspapers, wire services, social platforms. Those deals matter. But they are licensing flat content for language models. The multimodal, interactive, spatially rich data that world models require is a different category entirely, and the game industry is the only sector that produces it at scale.

The challenge for game companies is that turning game content into high-quality training data is a specialized discipline. Professional capture means recording at the engine level: removing the HUD, stripping overlays, isolating the clean visual feed while simultaneously capturing synchronized input streams, camera telemetry, physics state, and scene metadata. Done well, the result is pristine, high-signal data where every frame carries usable information. Done poorly, or scraped from the surface, the result is ten times more data than you actually need, most of it redundant, contaminated, or structurally useless for training. The signal-to-noise difference between professional multimodal capture and bulk-scraped gameplay is not a quality preference. It is a direct multiplier on model efficiency, fidelity, and training cost.

And a single title is not enough. World models need diversity: different environments, physics systems, visual styles, interaction patterns. That means working across publishers, across genres, across engines. It means AI-specific licensing frameworks, enrichment pipelines, metadata standards, and buyer relationships that most game companies have no reason to build. That is a different business. The more natural path is a partner that already understands both sides: the game industry's content, IP protections, and business model, and the AI industry's data requirements, pipeline expectations, and willingness to pay for quality.

We started Origin Lab because we come from these worlds. We have spent our careers inside games, interactive media, and AI, and we understand both what game environments contain and what the AI industry needs from them.

At Origin Lab, every source is licensed before capture begins. Every dataset ships with a full chain of custody, from source through capture, enrichment, and delivery. Rights holders are fairly participating in the value their IP is helping to create. This is not a compliance posture. It is a product design choice. We believe that data built on consent, structure, and compensation produces better training outcomes than data built on extraction and hope.

The AI industry spent its first decade treating data as a free resource. The next decade will reward the companies that treat it as an earned relationship. Not because regulators forced them to, but because the models that learn from better data will outperform the ones that do not.

That is the bet we are making. And in our next post, we will make the financial case: why bad data is one of the most expensive mistakes in modern AI, and what CEOs and model trainers should be asking before the next expensive training run.