Perspective

How AI is changing web scraping

Abby Grills· CEO, RiveterPublished Aug 29, 2026 · Updated Aug 29, 2026

Web scraping has been a stable craft for twenty years. You find the page, you locate the value in the markup, you write down where it sits, you handle the exceptions, and you re-do that work every time the site changes. The tooling improved enormously over that period. The shape of the job did not.

It’s changing now — not because extraction got easier, but because five different things moved at once, in different directions, and the combination is what makes this feel like a discontinuity rather than an upgrade.

1. From parsing to understanding

The traditional scraper is a set of coordinates. div.pricing > span.amount isn’t a description of a price; it’s a description of where a price happened to sit on a particular Tuesday. The knowledge is positional, which is why it’s so fragile — nothing about a redesign changes what the price is, but everything about it changes where the price lives.

Model-based extraction targets the value semantically. “The monthly price of the mid-tier plan” survives a redesign because it was never a statement about the DOM in the first place.

The consequence people underrate: when extraction logic is derived from a description rather than hand-written, it can be regenerated rather than repaired. The description is the durable artifact; the selectors become disposable output. That’s a different maintenance model, not a better version of the old one — and it’s the mechanism behind self-healing extraction, which is otherwise an easy phrase to use without meaning anything.

It isn’t free. Selectors are deterministic and cheap; comprehension is neither. Which leads directly to the third shift.

2. The unit of work changed

The old unit was “fetch this page.” Success was an HTTP 200 and a non-empty parse.

The new unit is “answer this question.” The page is an implementation detail — possibly several pages, possibly a PDF three clicks off the homepage, possibly a source nobody specified in advance.

This re-defines failure, and the re-definition is the important part. Under the old unit, a job that fetched every requested page succeeded. Under the new one, a job can fetch every page flawlessly and still fail, because the cells are empty. Fill rate becomes the metric, and it’s uncomfortable precisely because it’s honest — it measures whether the question got answered, not whether the system did what it was told.

It also changes who can ask. When the input is a description rather than a URL and a selector, the person who needs the data can specify the work. That has more organisational consequence than any technical shift on this list, because the bottleneck in most data teams was never extraction capability. It was the queue in front of it.

3. The cost curve inverted

For two decades the expensive part of scraping was access: proxies, residential IPs, browser farms, bandwidth, the arms race against bot detection. Parsing was nearly free. Whole businesses were built on making access cheaper, and they succeeded — that layer is now competitive and commoditized.

Now the expensive part is reasoning. Reading a page and understanding it costs orders of magnitude more than fetching it. Which inverts a lot of received wisdom:

Per-page pricing describes less of the real bill every year.
Caching and compilation matter more than raw throughput. If the reasoning happens once and subsequent runs execute compiled logic, the cost curve stays flat. If every run re-reasons, it doesn’t.
“Just scrape more” stopped being a strategy. At the old cost structure, over-collecting was rational insurance. At the new one, deciding what’s worth reading is itself an engineering problem.

The teams that get surprised by their bills are almost always running an architecture designed for the old curve.

4. Demand exploded on the other side

The commentary focuses on AI changing how data is collected. The larger change is that AI became the biggest new consumer of web data in decades.

Every agent needs live context. Every retrieval system needs current sources. Every model-backed feature that answers a question about the present is, underneath, a web data pipeline. The volume of programmatic web access is increasingly driven by machines reading on behalf of people rather than by people reading directly — and that demand is far less tolerant of staleness than the old batch-analytics demand was, because it’s answering in real time to someone waiting.

This is why freshness moved from a nice-to-have to a design constraint, and why “how old is this data” became a question buyers ask before “how much does it cost.”

5. The web is pushing back

All of the above is happening while the terms of access are being rewritten.

Bot detection escalated in sophistication and in willingness to block. Licensing deals between publishers and model developers established that large-scale access is a thing that gets negotiated rather than assumed. robots.txt acquired AI-specific directives; llms.txt appeared as a convention for sites that want to be read well by models. Some publishers began blocking AI crawlers specifically while continuing to welcome search crawlers — a distinction that didn’t exist five years ago and required new infrastructure on both sides to express.

The direction of travel: access is becoming explicit. Less of the web is a default-open commons and more of it is a set of stated terms. That’s a meaningful constraint on anyone building in this category, and pretending otherwise is how you end up with a pipeline that works right up until it’s an incident.

What building for this actually looks like

Five shifts is a diagnosis. The more useful question is what a web data system looks like if you take all five seriously at once — because most tools in this category were designed against one or two of them and retrofitted for the rest.

Riveter is what that argument looks like built out.

The description is the artifact, not the selector. Extractions are defined by what a field means rather than where it sits, so a redesign triggers regeneration instead of a ticket. That’s shift one taken literally: the thing you maintain is the thing that doesn’t change.

The unit of work is the question. Enrichments start from a description rather than a URL, and columns feed each other — so one column finds a source, the next reads it, a third pulls a field from what it read, and a call to your own or a third-party API can sit anywhere in that chain. Multi-step logic is configuration, not a codebase. And because the same definition runs interactively for one record and programmatically for a hundred thousand, the person who needs the field can add the field. That’s the organisational half of shift two, and it’s the half that changes how fast a team moves.

Reasoning happens once; execution happens many times. Extractions compile into fast programmatic runs rather than re-reading every page with a model on every pass. Against shift three’s inverted cost curve, that’s the difference between a bill that stays flat as you repeat and one that scales with every run.

Retrieval is live, and it reads the whole source. The web is searched at the moment of the request, so a value is as fresh as the source rather than as fresh as somebody’s crawl — and the source can be a JavaScript-heavy page, a large document, a PDF, or an image, not only clean HTML. That’s what shift four demands of anything feeding an agent or a customer-facing surface, and it’s the mechanism behind filling cells that index-based tools leave blank: finding the answer, not ranking pages that might contain it.

Monitoring closes the loop. Extractions and enrichments re-run on a schedule so a dataset stays current between live lookups, which is what makes a large number of external sources survivable without a proportional maintenance commitment.

None of that removes the last section’s problems. It’s aimed squarely at the first five.

What hasn’t changed

Four things, and they’re the reason this is a shift rather than a solution.

Sites still change without warning. Semantic extraction regenerates instead of breaking, which is a genuine improvement — but the change still happens, still has to be detected, and detection is still the hard half. A system that heals silently and a system that fails silently look identical from the outside until you check the data.

Completeness is still unverifiable. No model can tell you what it didn’t find. If a query returns 340 of 500 entities, it returns 340 rows that look entirely correct. This hasn’t improved at all, and there’s no reason to expect it to, because the missing rows leave no trace to detect.

A confident wrong answer is still the worst output in the category. Arguably it got worse. Traditional scrapers failed loudly — an empty result, a thrown exception, a nulled field. Model-based extraction fails fluently, producing a plausible value in the right format that is simply not true. That’s a much more expensive failure mode, and it’s the strongest argument for keeping source_url on every value and treating “not found” as a first-class output rather than a null.

Judgment is still required. Which sources to trust, what a field means when two sources disagree, whether a value is plausible for this entity. AI moved that work; it didn’t remove it.

The honest summary: AI has made the doing dramatically cheaper and the knowing whether it’s right barely cheaper at all. Most of the interesting engineering in this category over the next few years will be on the second problem, not the first.

FAQ

How is AI changing web scraping?

Five things at once: extraction moved from positional selectors to semantic description, so logic can be regenerated rather than repaired; the unit of work moved from “fetch this page” to “answer this question,” which makes fill rate the metric instead of request success; the cost curve inverted, so reasoning is now expensive and fetching is cheap; AI became the largest new consumer of web data, making freshness a design constraint; and the web began pushing back through bot detection, licensing, and AI-specific crawl directives.

Will AI replace traditional web scrapers?

Not entirely, and not soon. Deterministic selectors are cheaper and more predictable for high-volume extraction against stable, well-structured sources, and that’s a large share of real workloads. The shift is that description-based extraction handles the cases selectors handle badly — sites that change often, sources you have to find first, documents rather than pages — and most serious systems will run both.

Why do AI scrapers still get things wrong?

Because they fail fluently rather than loudly. A traditional scraper that breaks returns an error or a null; a model-based extraction that misreads a page returns a plausible, well-formatted value that happens to be false. That makes source capture and explicit “not found” outputs more important than they were, not less.

What does agentic web scraping mean?

Extraction where the system decides how to get the answer — which pages to open, which links to follow, when it has enough — rather than executing a fixed sequence you specified. The practical difference is the input: an agentic system can start from a description of what you want, where a traditional one starts from a URL and a rule for where the value sits. Riveter works this way — it searches the live web from a description, reads whole sources including PDFs and images, and returns named fields rather than pages.

Is web scraping still legal and viable in 2026?

The technical viability is fine; the terms of access are becoming explicit rather than assumed. Bot detection is more sophisticated, licensing arrangements between publishers and model developers are now normal, and site owners increasingly express AI-specific crawl preferences through robots.txt directives and conventions like llms.txt. The practical implication is that access is a thing to design for deliberately, not a default to rely on.

Next step. If your pipeline was designed for the old cost curve — selectors you maintain, an index you read from, reasoning re-paid on every run — bring the workload it half-finishes and compare fill rate rather than architecture.