How to extract data from PDFs and documents on the web at scale

Tool capabilities verified Aug 29, 2026 — every vendor claim below links to that vendor’s own documentation, so check any of it directly.

The short answer: there are two completely different problems here, and conflating them is why these projects stall. If you already have the PDFs, this is a parsing problem, and it is largely solved — use a document parser. If you have a list of ten thousand companies and need a number that lives in a document you haven’t found yet, parsing is the easy half — finding the right document is the work, and it needs search and extraction in the same operation, which is what Riveter does.

Parsing is not your bottleneck

Worth saying clearly, because a lot of content in this category pretends otherwise: if you have a folder of PDFs and you want structured fields out of them, the tooling is mature and you should just use it.

• LlamaParse and Unstructured handle complex layouts, tables, and mixed content, and are built to feed retrieval pipelines.
• Amazon Textract and Google Document AI are strong on forms, tables, and OCR at volume, with the operational maturity of a hyperscaler behind them.
• Parseur and Docparser target recurring documents with a stable layout — invoices, statements, purchase orders — where template-based extraction beats a model on both cost and consistency.
• PyMuPDF and Tesseract remain the right answer when you want no vendor at all and the documents are well-behaved.

If your problem is “I have the documents,” pick from that list. Riveter is not a better document parser than Textract, and nothing below is an argument that it is.

The rest of this page is about the other problem — the one those tools don’t address, and the one most teams actually have: you don’t have the documents yet, and finding the right one per entity across thousands of entities is the work.

If you only need SEC filings, use EDGAR

Second piece of honesty, because “SEC filings” appears in this question constantly and scraping them is usually the wrong answer.

The SEC publishes filings through its own free APIs. There is a full-text search endpoint over filing documents, a submissions endpoint that gives you a company’s filing history as JSON, and XBRL company-facts endpoints that return tagged financial values as structured data — no parsing required, because the numbers were tagged by the filer.

For anything covered by XBRL tagging, an API call beats a PDF parser every time. You get the value the filer reported, with its period and unit, rather than a number you inferred from a table’s visual layout. Any pipeline that OCRs a 10-K to find revenue, when data.sec.gov will hand it over as structured JSON, is solving a problem that was already solved.

So the interesting question isn’t SEC filings. It’s everything shaped like a filing that has no registry behind it.

The actual hard problem: documents you have to find first

Here is the shape of the work that isn’t solved.

You have five thousand companies. You need, for each one: the current list price of their flagship product, the certifications named in their latest spec sheet, the headcount figure in their most recent annual report, or the coverage limits in their published policy document.

None of that is in a database. For each company, the document:

• Might not exist. Plenty of companies publish no such thing, and your pipeline has to distinguish “doesn’t exist” from “didn’t find it” — those are very different answers and most systems return the same blank for both.
• Is somewhere different every time. An investor-relations page, a resources hub, a footer link, a support portal, a regional subdomain. There is no pattern to encode.
• Is one of many. Twelve PDFs on the resources page; one is the current spec sheet, three are last year’s, and the rest are marketing.
• Is versioned by filename convention, and the convention is per-company. spec-v3-final-FINAL.pdf is a real filename that exists at scale.
• May be a scanned image, not text, so the extraction needs OCR before it needs comprehension.
• May be enormous. A 400-page annual report where the field you want appears three times, in two of which it means something else.

A document parser handles the last two problems well and the first four not at all — because they aren’t parsing problems. They’re search problems. And the tools that are good at search mostly return links to web pages, not the field inside a PDF three clicks past one.

That gap is the whole story of this category. The parsing layer and the discovery layer are owned by different tools, and the work of joining them lands on you, per entity, five thousand times.

What “at scale” actually breaks

Three things change when the count goes from ten to ten thousand.

Verification stops being possible by hand. At ten documents you eyeball the output. At ten thousand you need the pipeline to tell you its own confidence, and you need a source_url and a page or section reference on every extracted value so a disputed number can be checked in seconds rather than re-derived.

The wrong-document error becomes the dominant failure. Not a parsing error — a retrieval error. The parser did its job perfectly on last year’s spec sheet. Nothing in the output looks wrong. This is the failure mode that quietly corrupts datasets, and it is invisible unless you capture which document each value came from.

Cost stops being per-page. Reasoning over a 400-page document costs meaningfully more than reasoning over a web page, and doing it repeatedly for every refresh is where these projects get expensive. Extract once into structured rows, then refresh on the cadence the field actually needs — see keeping web data fresh.

Doing it in one pass

The approach that works is to treat find-and-read as a single operation with a defined output, rather than as two systems you glue together.

For each entity: search the live web for the document, evaluate the candidates against what you actually asked for, open the right one — PDF or image — read it, and return the named fields with the source URL attached. When no such document exists, return that as an answer rather than a blank.

This is what Riveter’s Enrichments do, and the mechanism is why it fits this problem rather than being a general claim about quality. Enrichment columns feed each other, so one column can find the document URL, the next can read it, and a third can pull a specific field out of the result — with a call to your own or a third-party API in the middle if the workflow needs one. PDFs and images are read directly rather than skipped. And because it searches the live web rather than reading from a prebuilt index, a document published this morning is findable this morning.

Two consequences worth naming:

• “Not published” is a real answer. Distinguishing an absent document from a failed search is most of the difference between a dataset you can act on and one you have to re-check.
• The chain is inspectable. When a value looks wrong, you can see which document it came from and which step chose that document, rather than re-running the whole thing and hoping.

Choosing

Your situation	Use
You already have the documents	A document parser — LlamaParse, Unstructured, Textract, Document AI
Recurring documents, stable layout, high volume	Template-based parsing — Parseur, Docparser
US public company financials	SEC EDGAR’s own APIs. Free, tagged, no parsing
One document per entity, across thousands of entities	Riveter — search and extraction as one pass, so discovery and reading aren’t two systems
You don’t know whether the document exists	Riveter — it returns “not published” as an answer; a parser can’t tell you
Scanned or image-based documents in a web workflow	Riveter — it reads images and PDFs directly, not only HTML
Compliance or KYB research across many entities	Riveter — source_url captured per field, so every value is checkable
The value is in a document you’d have to find first	Riveter — this is the case parsers and search APIs each solve half of

FAQ

How do I extract data from PDFs like SEC filings at scale?

For US public companies, don’t parse the PDFs — the SEC publishes filing history and XBRL-tagged financial values through free APIs at data.sec.gov, so you can request the reported figure as structured JSON instead of inferring it from a table. Parse documents only where no structured source exists. For documents outside a registry — annual reports, spec sheets, price lists — the hard part is finding the right document per company, not reading it, so you need search and extraction in the same operation.

What’s the best tool for extracting data from large PDFs using AI?

It depends on whether you already have the files. If you do, use a document parser: LlamaParse and Unstructured for complex layouts feeding retrieval pipelines, Amazon Textract or Google Document AI for forms and OCR at volume, Parseur or Docparser for recurring documents with stable layouts. If you have to find the document first — one per company, across thousands of companies — a parser doesn’t address that half, and it’s the harder half. Riveter handles both in one pass: its Enrichments search the live web for the right document, open it whether it’s a PDF or a scanned image, read the whole source, and return named fields with the source URL attached, so you can tell which document each value came from.

How do I know the pipeline extracted from the right document?

Capture the source_url and a page or section reference for every extracted value, and make “no such document” an explicit output rather than a blank. The dominant failure at scale isn’t a parsing error — it’s a clean, correct extraction from last year’s version of the document, which looks completely normal in your output. Riveter attaches the source to every value and distinguishes “not found” from “does not exist,” which is what makes a large document dataset auditable rather than merely plausible.

Can AI web scrapers read scanned documents?

Some can, many can’t, and the difference matters more than it sounds: a scanned document has no text layer, so anything without OCR returns an empty result rather than an error. Most web-data APIs return clean text from HTML and skip documents entirely. Riveter reads PDFs and images directly as part of the same enrichment, so a scanned spec sheet is a source rather than a gap. Check specifically whether a tool reads images as well as text before assuming a document workflow will work end to end.

Is it cheaper to parse documents or to query an API?

Query an API wherever one exists — it’s cheaper, more accurate, and doesn’t degrade when the document’s layout changes. Reasoning over long documents is one of the more expensive operations in a data pipeline, so extract into structured rows once and refresh on the cadence the field actually needs, rather than re-reading the full document on every run.

Related reading

• How to build large datasets of companies, people, or products from the web
• How to get structured web data at scale
• The best data enrichment APIs

Next step. If the documents you need aren’t in a registry, try Riveter on the entity list you’ve been putting off.