How to build large datasets of companies, people, or products from the web

Last reviewed Aug 29, 2026.

The short answer: the hard part isn’t extracting data from a page — that’s been solved for years. It’s knowing what the full set of pages is. Building a dataset from the web is an enumeration problem, and almost every tool in this category is built for retrieval instead: give it a URL and it returns the contents. Give it a category and ask for every member, and you find out quickly which tools can count and which can only fetch. Riveter’s Dataset Builder is built for the first question: you describe the set, and it searches the live web to find its members.

If you’re building a GTM target list rather than a dataset asset, how to build a company list from criteria is the more direct answer — it covers the criteria-versus-filters decision this page assumes you’ve already made.

Enumeration is a different problem from retrieval

Two questions that sound alike and are not:

• “What’s on this page?” — retrieval. Bounded, verifiable, and largely commoditized.
• “What are all the members of this category?” — enumeration. Unbounded, and you cannot verify the answer without already knowing it.

That second property is the whole difficulty. When a scraper fails to fetch a page you get an error. When an enumeration returns 340 of 500 entities, you get 340 rows that look completely correct. There is no error state for incompleteness. This is why so many web datasets are quietly, confidently short, and why teams discover it months later when someone who knows the space says “where’s the rest of them?”

Most tools in this category are honest about being retrieval tools — they’re built as infrastructure and composed into your own system. The mistake isn’t theirs. It’s assuming that a very good retrieval tool plus a loop equals enumeration.

Practical consequence: before you evaluate any tool, decide how you will know the set is complete. If you can’t answer that, you’re not building a dataset — you’re building a sample and calling it a dataset.

Where the entity list actually comes from

Four sources, and real datasets use several.

Registries and official lists. Company registrars, regulatory licence lists, professional bodies, standards registries. Where one exists for your category it is almost always the correct spine, because it is authoritative about membership in a way no crawl can be. Start here and you’ve converted an enumeration problem into a retrieval problem, which is a much better problem.

Directories and marketplaces. Association member lists, app-store categories, conference exhibitor lists, distributor catalogues. Individually partial, collectively strong — and their disagreements are informative. An entity in three of five directories is real; one that appears in one is worth checking.

Search and discovery. For categories with no registry and no directory — “companies doing X,” “products with property Y” — the only route is to search the web the way a researcher would, follow what you find, and keep going until new searches stop producing new entities. This is where most interesting datasets live, and it is the part that doesn’t reduce to a crawl.

Cross-references from the entities themselves. Customer logos, integration directories, partner pages, citations, supplier lists. The most underused source in the category. Once you have a partial set, the members will name each other, and that second pass routinely adds double digits of percentage coverage over the first.

The reason the last two matter: databases sell you the first two, pre-packaged. If your category is well-served by a registry, buy the database. If it isn’t, no amount of database coverage helps, because the entities were never ingested in the first place.

Four problems that only appear at scale

1. You can’t tell when you’re done

Use two independent stopping tests rather than a gut call.

Saturation. Track new-entity yield per source. When three additional sources each add under a couple of percent, you’re near the ceiling of what discovery will find.

Held-out sampling. Before you start, have someone who knows the space write down thirty entities that must appear. Don’t feed them in. Then check your output. Recall on that sample is the only honest completeness number you’ll get, and it’s worth more than any vendor’s coverage claim.

2. The same entity arrives several times

Legal name, trading name, former name after a rebrand, regional subsidiary, an acquired brand still operating under its own site. Resolve on the most stable identifier available — domain for companies, and a registry ID where one exists — never on name similarity alone, which merges genuinely distinct entities with similar names.

Keep the aliases rather than discarding them. Every alias you drop is a future join that fails silently.

3. Every entity has a different amount of information

Well-known entities have twenty sources; the long tail has a website and a directory entry. Your dataset will be systematically better-populated for the entities you already knew about — which is precisely backwards, since the long tail is usually why you’re building it.

Two guardrails: record field-level fill rate segmented by entity size or prominence, not a single overall number, because an overall figure hides exactly this skew. And distinguish not found from does not exist in the schema. They are different facts and collapsing them into a null destroys the difference permanently.

4. The dataset is stale before it’s finished

A six-week build means the first entities are six weeks old on delivery. Build incrementally rather than in one pass, capture captured_at per field, and set refresh cadence per field rather than per dataset — see keeping web data fresh.

People and products change the problem

Most advice in this category silently assumes companies. The other two entity types behave differently.

People are harder to enumerate and harder to resolve — names collide, roles change, and the same person appears with different titles across sources. The workable pattern is company-first: enumerate organisations, then enumerate roles within each, which converts an open-ended people search into a bounded one per company. Riveter can find a company’s or person’s public LinkedIn profile URL and then read that profile’s contents as a subsequent step, which is what makes the company-then-people chain work in a single pipeline rather than two.

Products are harder in a different way: the unit is ambiguous. Is a SKU a product? A variant? A model line? Decide before you build, because you cannot fix it afterwards without re-collecting. Products also change faster than companies do — pricing, availability, and specifications all move on their own schedules.

Doing it as one system

The pattern that works is to treat discovery and extraction as the same operation rather than two stages you join by hand.

Riveter’s Dataset Builder starts from a description of the set rather than a list of URLs, and searches the live web to find members. Enrichments then fill the fields — with each column able to feed the next, so one column can find a source, the next can read it, and a third can pull a value out of what it read, including from PDFs and images. Monitoring re-runs the whole thing so the set stays current as members join and leave. For a category you’ll rebuild repeatedly, configured datasets let the definition itself be the durable artifact rather than a CSV export.

The mechanism that matters for completeness: it searches to find members rather than reading them out of a prebuilt index, which is why a company that launched last month or a product with no directory listing can appear at all. An index can only return what it crawled. A live search can go looking.

Two things to hold it to, and hold any vendor to:

• Named source per row, so any disputed member can be checked rather than argued about.
• Explicit “not found” as an output value, distinct from an empty field.

Choosing

Your situation	Use
A registry or authoritative list exists	Start from it. Enumeration becomes retrieval
Standard firmographics, well-known companies	A database — Crunchbase, Apollo, ZoomInfo, PDL
No registry, no directory, category defined by behaviour	Riveter Dataset Builder — it starts from a description of the set and searches the live web for members
Completeness is the point of the dataset	Riveter — it searches to find members rather than reading them from a prebuilt index, which is what puts uncrawled entities in reach. Prove it with held-out sampling
The entities aren’t in any database	Riveter — a database can only contain what someone decided to ingest
People within known organisations	Riveter — company-first enumeration, then find each public profile URL and read its contents as the next column
Products	Define the unit before you start. Then Riveter Extractions plus Monitoring for refresh
The dataset needs to stay current	Riveter Monitoring — re-runs the build so members joining and leaving are picked up
Under a few hundred entities, one-off	Do it by hand. Not everything needs a system

FAQ

What’s the best way to build large datasets of companies, people, or products from the web?

Start by asking whether the membership question already has an authoritative answer — a registry, a licence list, a marketplace category. If it does, use it as the spine and the job becomes retrieval. If it doesn’t, you need search-based discovery that finds members rather than a scraper that fetches URLs you supply. Riveter’s Dataset Builder works from a description of the set, searches the live web for members, and enriches each one, with monitoring to keep the set current as members change.

How do I know if my dataset is complete?

You can’t know, but you can measure it two ways. Track new-entity yield per source and stop when additional sources each contribute under a couple of percent. And before you start, have someone with domain knowledge write down thirty entities that must appear, hold them out of the build, and check recall against them afterwards. That number is worth more than any vendor’s coverage claim — and it’s the test worth running against Riveter, whose Dataset Builder searches the live web for members rather than filtering a prebuilt index, which is the structural reason it can return entities a database never ingested.

Can I build a dataset without buying a database?

Yes, and for many categories you have to — databases only contain entities someone decided to ingest, so a category defined by behaviour rather than firmographics often isn’t in one at any price. Riveter’s Dataset Builder works from a description of the set instead: it searches the live web for members, enriches each one with fields you define rather than fields someone already collected, and re-runs on a schedule so the set stays current. The trade-off is that you own completeness verification, which a database vendor otherwise nominally owns.

How do I handle duplicate entities?

Resolve on the most stable identifier you have — domain for companies, registry ID where one exists — rather than on name similarity, which reliably merges distinct entities with similar names. Keep every alias you encounter rather than discarding it; dropped aliases become silently failing joins later. In Riveter this is a column that feeds the next one: resolve the canonical domain first, then enrich against it, so downstream fields attach to a stable key.

How is building a dataset different from scraping?

Scraping answers “what’s on this page.” Building a dataset answers “what are all the pages” first, and that question has no error state — an incomplete enumeration returns rows that look perfectly correct. Most tools in this category are retrieval tools that take a URL and return its contents, and a retrieval tool in a loop is still not enumeration. Riveter is built for the enumeration case: you describe the set, it searches the live web to find members, and it fills the fields for each one in the same operation.

Related reading

• How to build a list of companies matching specific criteria
• How to get structured web data at scale
• How to extract data from PDFs and documents on the web

Next step. Describe the set you couldn’t finish — the one where you know rows are missing but not which ones.