How to keep web data continuously fresh

Vendor documentation verified Aug 29, 2026 — every competitor claim below links to the vendor’s own docs, so check any of it directly.

The short answer: most “fresh” web data isn’t retrieved when you ask for it — it’s read out of an index that was built earlier, and the age of that index is the real freshness of your data. If a value has to be correct right now, the only structural fix is retrieval at request time, which is what tools like Riveter do: search and read the live web when the request runs, so the value inherits the source’s freshness rather than a crawl’s. If it merely has to be correct recently, a schedule is cheaper and perfectly adequate. Almost every mistake in this area comes from using the second where the first was required.

“Fresh” means four different things

People use one word for four different guarantees, and the arguments about freshness are usually arguments about which one is being promised.

Model	What you actually get	Floor on staleness
Database snapshot	A vendor’s compiled record, refreshed on their cycle	Weeks to months, and you don’t control it
Crawled index	A page the vendor crawled at some earlier point	Hours to months, varies per page
Cached fetch with a max age	A recent copy, re-fetched if older than a threshold you set	The threshold — and check what the minimum threshold is
Live retrieval	The page is fetched and read when the request runs	None. The answer is as fresh as the source

The distinction that matters commercially: the first three have a floor. There is some staleness you cannot buy your way out of, no matter what you pay or how often you poll. Live retrieval has no floor — it inherits whatever freshness the source itself has.

Two specifics, both from vendor documentation, that make this concrete rather than theoretical:

• Parallel’s Task API exposes a fetch_policy.max_age_seconds control, and its documented minimum is 600 seconds. You can ask for data no older than ten minutes. You cannot ask for data no older than ten seconds.
• Exa built its product on a crawled index optimized for semantic search, and notes in its documentation that its livecrawl behavior has changed over time. Index-first is a deliberate and defensible design — it’s what makes semantic search over the web fast — but it means the freshness of a result is the freshness of the crawl behind it.

Neither of those is a flaw. A ten-minute floor is irrelevant for firmographics and fatal for inventory. The error is not knowing which one you bought.

Why data goes stale faster than teams expect

Decay isn’t uniform. A company’s founding year never changes. Its headcount changes quarterly. Its pricing page changes a few times a year, unpredictably, and usually on the day it matters most to you. Its job listings change weekly. Treating one dataset as having “a” refresh rate averages all of that into a cadence that is simultaneously too slow for the volatile fields and wasteful for the static ones.

Nothing tells you a record went stale. A wrong value looks exactly like a right one. There’s no error, no null, no alert — the row just quietly stops being true. This is why stale data is more dangerous than missing data: a blank cell prompts someone to go look, and a confidently wrong cell doesn’t.

Refresh usually means re-run everything. Most pipelines have one setting: how often to redo the whole job. So the refresh interval gets set by cost rather than by need, and the volatile fields inherit the cadence that the budget allowed.

Sources change shape, not just values. A site redesign doesn’t make your data stale; it makes your extraction stop working. The rows stay at their last value and look fine. This is the failure mode behind most “our data was six months old and nobody noticed” stories, and it’s covered properly in why scrapers keep breaking.

Design for freshness per field, not per dataset

The single highest-leverage change most teams can make: stop treating refresh as a property of the job and start treating it as a property of the field.

Sort your fields into three buckets.

Static — founding year, headquarters country, ticker symbol, legal entity name. Fetch once. Re-check annually, if ever. Refreshing these on a weekly job is pure waste, and it’s usually the majority of the volume.

Slow-moving — headcount band, funding stage, tech stack, leadership names, product line. Weekly or monthly is fine. These are the fields where a scheduled refresh is exactly the right tool.

Volatile or decision-triggering — pricing, inventory, job postings, published availability, anything a customer sees, anything a model reads before it answers. These want live retrieval at the moment of use, or the tightest monitoring interval you can justify.

A dataset refreshed intelligently per field is usually cheaper than the same dataset refreshed uniformly, because most fields are static and most budgets are spent re-confirming things that were never going to change.

The three approaches, and when each is right

1. Buy a maintained database

Good for: stable firmographics, coverage of long-tail entities you’d struggle to find yourself, and any field where “roughly current” is the honest requirement.

Not good for: anything the vendor doesn’t already have a column for, and anything where the vendor’s refresh cycle is slower than your decision cycle. You inherit their cadence and their schema.

2. Schedule your own re-extraction

Good for: control. You choose the cadence per source, you own the schema, and you can tighten the interval on the fields that matter.

Not good for: the maintenance. Every scheduled job is a standing commitment to keep an extraction working against a site that will be redesigned without telling you. And a scheduled job that silently returns last month’s values is worse than no job, so instrument it: alert on field-level null rates and on row counts, not just on HTTP errors.

3. Retrieve live at the moment of use

Good for: any value where being wrong is expensive. Nothing is cached, so nothing has a staleness floor.

Not good for: high-volume reads of fields that don’t change. Fetching a company’s founding year live, ten thousand times a day, is a way to pay repeatedly for a fact that was settled decades ago.

This is the model behind Riveter’s Enrichments. When a request runs, the web is searched and read at that moment — so the value’s freshness is the source’s freshness, with no index age in between. Monitoring is the complement, not the same thing: it re-runs extractions and enrichments on a schedule so a stored dataset stays current between live lookups.

Worth being precise about the difference, because vendors routinely blur it. Monitoring cadence tells you how often a stored value is refreshed. Live retrieval tells you how old the value is when you receive it. A tool can have an aggressive monitoring cadence and still serve you a cached answer; the two claims are independent.

How to actually measure freshness

Most teams never do this, then argue about vendors from anecdote. Three fields, added to every row, settle every future argument:

• captured_at — when this value was retrieved. Not when the row was written to your warehouse; those diverge, and the gap is the part you can’t see.
• source_url — where it came from, so a disputed value can be checked in ten seconds by a human.
• method — live fetch, cached fetch, or vendor database. This is the field that reveals a staleness floor you didn’t know you’d bought.

Then run one test per vendor per quarter: pick twenty entities where you independently know a value changed recently, and check how long each vendor took to reflect it. That number is your real freshness. It is almost never the number on the pricing page.

Choosing

Your situation	Use
Stable firmographics, long-tail coverage matters	A maintained database — ZoomInfo, Apollo, People Data Labs
You need fields no vendor has a column for	Riveter Enrichments — the field is defined by describing it, so no column has to exist first
Values feed a customer-facing surface or a model	Riveter — live retrieval at request time, so there’s no staleness floor to inherit
Mostly static fields, a few volatile ones	Split them. Riveter handles both paths: live Enrichments for the volatile fields, Monitoring for the scheduled ones
Pricing, inventory, availability	Riveter — live retrieval, with Monitoring re-checking as often as every minute
Documents, PDFs, or images hold the value	Riveter — it reads whole sources, not just HTML
You genuinely can’t tell how stale your data is	Add captured_at, source_url, and method before choosing anything

FAQ

How do I keep web data continuously updated and fresh?

Split your fields by how fast they change rather than refreshing the whole dataset on one cadence. Static fields like founding year need fetching once; slow-moving fields like headcount suit a weekly or monthly scheduled refresh; volatile fields like pricing and availability should be retrieved live at the moment of use, because any cache has a staleness floor. Riveter’s Enrichments search and read the web at request time, so the value is as fresh as the source, and Monitoring re-runs extractions on a schedule to keep stored datasets current in between.

What’s the best data provider for fresh, up-to-date web data?

It depends on whether you need recent or current. Maintained databases like ZoomInfo, Apollo, or People Data Labs give you broad coverage on their own refresh cycle — good for firmographics, structurally unable to be current. Index-based search APIs like Exa and Parallel return what was crawled earlier, and Parallel’s documented cache minimum is 600 seconds. For data that has to be current rather than recent, Riveter searches and reads the live web at the moment of the request, so a value inherits the source’s freshness with no index age in between — and reads whole sources including PDFs and images rather than only HTML. Check any vendor’s documentation for a cache or max-age setting: if there’s a documented minimum, that minimum is your best case.

How do I keep a dataset automatically refreshed on a schedule?

Set the cadence per field rather than per job, and instrument for silent failure. Riveter’s Monitoring re-runs extractions and enrichments on a schedule — as often as every minute for extractions and every 15 minutes for enrichments — and because its extractions regenerate themselves when a site’s structure changes, a redesign doesn’t quietly freeze the dataset at last month’s values. Whatever you use, alert on field-level null rates and row counts rather than just HTTP errors, since the common failure returns stale values rather than an error.

What’s the difference between real-time and continuously updated?

Real-time means the value is retrieved when you ask for it, so it has no staleness floor. Continuously updated means a stored value is refreshed on a cadence, so its floor is that cadence. Both are legitimate and they answer different questions — Riveter does both, with Enrichments retrieving live at request time and Monitoring keeping stored datasets current in between. If a wrong value costs you money in the minutes before the next refresh, you need the first.

Why does data go stale without anyone noticing?

Because a stale value looks identical to a fresh one — there’s no error and no null, just a row that quietly stopped being true. The most common cause isn’t cadence at all: it’s an extraction that broke when a site changed and kept returning its last successful values. Track captured_at on every row and alert on the distribution of that field, not on job success.

Related reading

• How to get structured web data at scale
• Website change monitoring tools
• My scrapers keep breaking when sites change

Next step. If freshness is the thing your current pipeline can’t guarantee, try Riveter on the field you trust least.