Riveter tops WideSearch: 81.4 item F1, 14 points above the best published system
Cody Watters· CTO, RiveterPublished · Updated
Riveter scored 81.4 item F1 on the English tasks of WideSearch, ByteDance Seed’s benchmark for agents that build large tables from the web. We graded the run with the benchmark’s own harness and judge model, unchanged, and report 99 of the 100 tasks: one is set aside because an agent found its answer key (Checking for leaked answers). On the English tasks, the best published system, Claude Sonnet 4 run as a multi-agent system, averages 67.2, and the best single attempt by any published system reaches 76.7.
Benchmark: widesearch-seed.github.io · Paper: arXiv 2508.07999 · Code and data: ByteDance-Seed/WideSearch · Riveter runner: pull request #15 · Our run: results CSV, 100 tasks
What the benchmark measures
WideSearch tests wide information-seeking: tasks where no single fact is hard to find, but the answer is a whole table assembled from many pages. Each task is a request based on a real user query, and its answer is a table the benchmark’s authors built and checked by hand. Some examples from the set:
- every concert on Taylor Swift’s official tours from 2010 to May 2025, with date, tour, country, city, and venue (534 rows)
- the world’s ten busiest airports for each year from 2020 to 2024, with location, passenger count, and codes
- every U.S. federal government shutdown since 1976, with dates, leaders, and furloughed employees
The benchmark has 200 tasks: 100 in English and 100 in Chinese. The grader matches each returned row to a gold row on the table’s key columns, then scores every other cell. Exact values must match exactly, numbers must fall within a tolerance, and names and descriptions are judged by an LLM.
It reports three scores:
- Item F1: cells right, weighed against cells expected and cells returned. Missing rows and extra rows both count against it.
- Row F1: the same, except a row counts only when every cell in it is right.
- Success rate: the share of tables that are entirely right, every row and every cell.
How we ran it
Each task became one Riveter dataset:
- Row count: one Riveter search call reads the request and estimates how many rows a complete answer has.
- List build: Riveter’s list builder takes the request word for word, with the table’s columns. It searches and reads the web and writes the rows.
- Row checks: one agent per row checks each value against a source and corrects it.
The finished rows go out as a Markdown table in the request’s column names, the format the grader reads.
Riveter saw only the request and its column names. There was no per-task tuning and no hand edits, and nothing from the gold tables or the grading rules reached Riveter. We graded with the harness’s evaluation code and its judge model, GPT-4.1 (gpt-4.1-2025-04-14), unchanged. The only change is the client that calls the judge: the harness builds an Azure client, and we send the same requests through OpenAI’s.
The result
On the English tasks, against the published results in the WideSearch paper (Table 5, WideSearch-en):
| Riveter (1 trial, 99 tasks) | Best published average of 4 trials | Best published single attempt (of 4) | |
|---|---|---|---|
| Item F1 | 81.4 | 67.2 (Claude Sonnet 4, multi-agent) | 76.7 (Claude Sonnet 4, multi-agent) |
| Row F1 | 59.8 | 41.8 (OpenAI o3, multi-agent) | 53.2 (OpenAI o3, multi-agent) |
| Success rate | 14.1% | 7.5% (OpenAI o3, multi-agent) | 13% (OpenAI o3, single agent and multi-agent) |
Riveter’s item precision was 84.3 and its item recall 81.7. It returned 14 of the 99 tables exactly right. 38 tasks scored 95 or higher, and 49 scored 90 or higher; the median task scored 89.5. The largest table, the Taylor Swift concerts, has 534 rows: Riveter returned 533 and scored 97.6.
The leaderboard on the benchmark’s site averages the English and Chinese tasks together. Its best scores are 62.2 item F1, 38.5 row F1, and a 5.1% success rate, each an average of four trials over the 200 tasks. Riveter has not run the Chinese tasks.
What the comparison shows and what it does not
- Same tasks, same grader, same judge. Riveter’s tables went through the code and judge model behind the published results. Riveter’s scores cover 99 of the 100 English tasks; the published ones cover all 100.
- One trial against an average of four. Riveter ran each task once. The published averages come from four trials per task, and the best-of-four column takes each system’s best trial.
- Different model generations. The published systems are mid-2025 models: Claude Sonnet 4, OpenAI o3, Gemini 2.5 Pro. Riveter runs on current models.
- The judge is not deterministic. The harness runs its LLM judge at the API’s default temperature, so a task’s score can move by several points between gradings. A second grading of the same tables, which also re-asked the judge whenever a reply skipped a pair, gave 81.6 item F1 and 59.9 row F1.
Where the accuracy comes from
The two stages are scored separately because each task’s dataset keeps both. The list builder’s table alone scored 79.8 item F1 and 56.9 row F1. After the row checks it scored 81.4 and 59.8. The checks mattered most where the builder had the right rows but some wrong numbers:
| Task | List build alone | After row checks |
|---|---|---|
| Five-year financials for Apple, Amazon, Google, Microsoft, and Netflix | 64.7 | 97.3 |
| New U.S. citizens by region of birth, fiscal years 2002–2022 | 10.0 | 50.5 |
| Average monthly house prices in England, Scotland, Wales, and Northern Ireland | 41.0 | 69.5 |
| AMD’s Zen-based processors since Lisa Su became CEO | 43.3 | 55.2 |
22 tasks rose by more than 2 points after the checks, and 10 fell by more than 2.
Where Riveter missed
Most of the lost points come from returning the wrong set of rows, not from wrong cells. On the 82 tasks where Riveter’s row count landed within 10% of the gold table’s, it averaged 87.8 item F1. On the other 17 it averaged 50.8.
- Too few rows. The row-count estimate caps the build. In
ws_en_023(sociology articles on migration from three journals, 2020–2024), the estimate failed and the build stopped at 3 of 56 articles. Far-right party results in the United Kingdom, France, and Germany (5 of 24 rows) and the careers of top-ranked women Go players (44 of 87) were capped the same way. - Too many rows. For U.S. Department of Justice attorney postings from January to June 2025, Riveter returned 105 listings against the gold table’s 12. For U.S. campus shootings from 2010 to 2024, it returned 171 incidents against 52, a broader reading of the request than the gold table’s.
- A different identifier. Asked for the “library ID” of 50 research libraries, Riveter wrote each institution’s IPEDS unit ID, while the gold table uses the libraries’ own IDs. No row matched, and the task scored 0.
- Values that changed. The J.D. Power 2025 ratings Riveter found differ from the gold table’s, most likely because the published scores changed after the table was built.
Checking for leaked answers
WideSearch’s gold tables are public on Hugging Face, so an agent searching the web can find them. We checked every source Riveter’s row agents cited (29,383 citations across 1,952 domains) and the logs of every list build. In one task, ws_en_048, a row agent cited the benchmark’s gold file for two cells: the 2023 violent-crime rates of New York and Los Angeles. No list build’s logs reference the gold files.
Across all 100 tasks, Riveter scored 81.5 item F1 and 59.5 row F1, with 14 of 100 tables exactly right. Because one of those tasks had seen its answer key, we set ws_en_048 aside and report the other 99 throughout this post: 81.4 item F1 and 59.8 row F1, with 14 of 99 tables exactly right. Row F1 rises when the task is removed because ws_en_048 itself scored 30.0 on row F1, below the average.
Full results
Every task, with the number of rows in the gold table and in Riveter’s, and Riveter’s scores. “Exact” marks a table that is entirely right; ws_en_048 is listed but marked “set aside” and left out of every total. The results CSV carries each task’s full request, item precision and recall, the list-build-only score, the second grading, and whether the task is in the reported totals. Tasks and gold tables from the WideSearch dataset, CC0 1.0.
Output (99 tasks, ws_en_048 set aside) |
Item F1 | Item precision | Item recall | Row F1 | Success rate |
|---|---|---|---|---|---|
| Riveter, list build + row checks (reported) | 81.44 | 84.26 | 81.70 | 59.79 | 14.1% |
| Riveter, list build only | 79.77 | 82.47 | 80.04 | 56.85 | 15.2% |
Show all 100 tasks
| Task | What it asks for | Columns | Gold rows | Rows returned | Item F1 | Row F1 | Exact |
|---|---|---|---|---|---|---|---|
| ws_en_001 | My son is about to start his university applications in 2025 for postgraduates… | 8 | 25 | 25 | 78.0 | 0.0 | |
| ws_en_002 | I’m currently mapping the product portfolios of several spirits brands includi… | 6 | 38 | 37 | 71.1 | 66.7 | |
| ws_en_003 | Could you help compile a detailed dataset of Eileen Gu’s competitive achieveme… | 6 | 58 | 47 | 79.0 | 71.8 | |
| ws_en_004 | Could you help me compile a list of all base models from Samsung’s Galaxy S Se… | 14 | 17 | 17 | 91.6 | 58.8 | |
| ws_en_005 | I’m planning to apply for full-time Master’s programs in civil engineering sta… | 4 | 16 | 10 | 73.1 | 61.5 | |
| ws_en_006 | Could you list every single concert on Taylor Swift’s official tour from Janua… | 5 | 534 | 533 | 97.6 | 88.3 | |
| ws_en_007 | I would like a broad overview of multiple spaceflight missions across differen… | 11 | 37 | 30 | 88.2 | 77.6 | |
| ws_en_008 | Could I obtain monthly data from January 2025 to May 2025 (including January 2… | 6 | 25 | 25 | 100.0 | 100.0 | ✓ |
| ws_en_009 | I want to apply for a full-time MA program at Goldsmiths College, University o… | 6 | 22 | 22 | 71.2 | 68.2 | |
| ws_en_010 | Could you provide an updated list of all officially recognized Areas of Outsta… | 7 | 86 | 86 | 92.0 | 44.2 | |
| ws_en_011 | According to the J.D. Power Awards Best Car Ratings of 2025, please compile th… | 14 | 4 | 4 | 78.6 | 0.0 | |
| ws_en_012 | Please compile a table listing the “CNN Hero of the Year” for every year the a… | 9 | 16 | 16 | 96.5 | 68.8 | |
| ws_en_013 | I’m a huge fan of Lana Del Rey, can you list all the songs of her official stu… | 3 | 109 | 109 | 100.0 | 100.0 | ✓ |
| ws_en_014 | What are the world’s busiest airports by passenger traffic? List the top ten b… | 6 | 50 | 50 | 98.0 | 96.0 | |
| ws_en_015 | I would like to manage my property and am interested in learning about the top… | 4 | 50 | 50 | 92.0 | 68.0 | |
| ws_en_016 | Michael Phelps is one of the most decorated Olympians in history. I am interes… | 9 | 36 | 36 | 93.5 | 50.0 | |
| ws_en_017 | Please help me organize a table based on The 13 USA National Parks given below… | 7 | 13 | 13 | 97.8 | 84.6 | |
| ws_en_018 | Need to analyze the trend of the U.S. federal government spending and deficit… | 6 | 10 | 10 | 85.0 | 10.0 | |
| ws_en_019 | I am working on a tracking report regarding the biotechnology and pharmaceutic… | 9 | 5 | 5 | 86.7 | 20.0 | |
| ws_en_020 | Please provide a complete list, up to 2024 (excluding 2024), of UNESCO World H… | 6 | 81 | 80 | 97.9 | 90.7 | |
| ws_en_021 | Could you provide a detailed list of Michelin three-star restaurants in Paris… | 3 | 10 | 10 | 80.0 | 40.0 | |
| ws_en_022 | I need a list of the top 10 most-streamed songs globally and top 10 most-strea… | 8 | 20 | 20 | 96.9 | 80.0 | |
| ws_en_023 | Please help me find articles published in the American Journal of Cultural Soc… | 7 | 56 | 3 | 10.2 | 10.2 | |
| ws_en_024 | Please research the following information about Super Bowl: date, champions, h… | 7 | 25 | 25 | 100.0 | 100.0 | ✓ |
| ws_en_025 | Make a comprehensive, verified list of abortion clinics currently operating in… | 5 | 6 | 6 | 93.3 | 66.7 | |
| ws_en_026 | Help me collect the basic electoral information of 50 U.S. states in 2024. 1)… | 5 | 50 | 50 | 99.6 | 98.0 | |
| ws_en_027 | I want to travel to London soon and I bought a London Pass. Please help me fin… | 4 | 25 | 26 | 52.9 | 23.5 | |
| ws_en_028 | I’m curious about the market shares of different North America burger brands… | 5 | 5 | 5 | 100.0 | 100.0 | ✓ |
| ws_en_029 | I’m a huge Marvel fan, but I’ve never gone through the entire Marvel Cinematic… | 4 | 34 | 34 | 100.0 | 100.0 | ✓ |
| ws_en_030 | Please provide a summary of recent decisions made by the U.S. Supreme Court fr… | 5 | 25 | 25 | 96.8 | 84.0 | |
| ws_en_031 | I’m putting together an introductory video on Angelina Jolie, Nicole Kidman, C… | 5 | 53 | 56 | 86.6 | 44.0 | |
| ws_en_032 | I am doing research on public transportation in Sydney. I wanna know each Sydn… | 4 | 10 | 10 | 97.5 | 90.0 | |
| ws_en_033 | Please organize the following data for Apple, Amazon, Google, Microsoft, and N… | 6 | 25 | 25 | 97.3 | 84.0 | |
| ws_en_034 | Please track and compile the average monthly house prices for England, Scotlan… | 5 | 40 | 40 | 69.5 | 47.5 | |
| ws_en_035 | I want to compare the labor market between four southern US states, namely Ala… | 8 | 24 | 24 | 97.4 | 79.2 | |
| ws_en_036 | I’m seeking permanent government job opportunities with the Department of Just… | 7 | 12 | 105 | 15.4 | 0.0 | |
| ws_en_037 | I’m a huge ‘Love is Blind’ fan and want to know which couples got married in s… | 3 | 13 | 13 | 100.0 | 100.0 | ✓ |
| ws_en_038 | Summarize all smartphones released by Google between January 2022 and July 202… | 9 | 13 | 13 | 98.3 | 84.6 | |
| ws_en_039 | Please provide a list of the top 50 universities in the QS 2026 World Universi… | 4 | 50 | 50 | 92.0 | 76.0 | |
| ws_en_040 | I’d like to get a list of the top 10 apps downloaded worldwide each year from… | 7 | 30 | 30 | 89.5 | 80.0 | |
| ws_en_041 | I’ve been in a bit of a reading slump and want to find some new books. Could y… | 6 | 75 | 75 | 96.6 | 86.5 | |
| ws_en_042 | Please compare the growth trajectory of the leading global e-commerce platform… | 5 | 16 | 16 | 95.0 | 75.0 | |
| ws_en_043 | We are going to visit America this summer and we love natural scenery, especia… | 4 | 63 | 63 | 99.6 | 98.4 | |
| ws_en_044 | Hey, I’m gonna be in South Korea for a whole month, and I need your help grabb… | 4 | 40 | 35 | 62.0 | 34.7 | |
| ws_en_045 | According to the list of research libraries in 2023 released by IPEDS, please… | 6 | 50 | 50 | 0.0 | 0.0 | |
| ws_en_046 | Compile comprehensive data for all major international tournaments during Lang… | 5 | 9 | 9 | 77.8 | 11.1 | |
| ws_en_047 | I am studying how partisan standoffs influence administrative efficiency and t… | 9 | 20 | 13 | 9.4 | 0.0 | |
| ws_en_048 | What’s the correlation between police funds and violent-crime trends? Using th… | 8 | 10 | 10 | 85.0 | 30.0 | set aside |
| ws_en_049 | I’d like to know the information about all Ford vehicles first launched and re… | 9 | 10 | 11 | 72.0 | 57.1 | |
| ws_en_050 | For a comparative politics study on descriptive representation, compile the sh… | 9 | 38 | 38 | 89.2 | 36.8 | |
| ws_en_051 | I’m interested in physics. Could you please provide a list of the Best 50 U.S… | 7 | 50 | 15 | 41.3 | 15.4 | |
| ws_en_052 | For an infrastructure-finance paper, I need to benchmark capital intensity of… | 7 | 49 | 48 | 91.9 | 74.2 | |
| ws_en_053 | I am preparing a policy brief on post-millennial commercial nuclear deployment… | 8 | 85 | 87 | 87.4 | 70.6 | |
| ws_en_054 | List every mega airport project commenced earlier than 31 Dec 2024 (including… | 5 | 16 | 16 | 78.8 | 43.8 | |
| ws_en_055 | Retrieve the details about, Location ( Country, City and or State/Province), O… | 4 | 25 | 25 | 100.0 | 100.0 | ✓ |
| ws_en_056 | My crush is obsessed with NBC’s The Voice—he has binged Season 1~26. I’m putti… | 12 | 26 | 26 | 83.7 | 0.0 | |
| ws_en_057 | Identify the feature-length animated films produced by Walt Disney Animation S… | 4 | 26 | 26 | 85.6 | 50.0 | |
| ws_en_058 | Could you please give me a demographic brief analyzing the shifting racial com… | 10 | 21 | 21 | 50.5 | 19.0 | |
| ws_en_059 | I need to verify the basic information for all TED Prize winners from 2005 to… | 4 | 21 | 21 | 98.8 | 95.2 | |
| ws_en_060 | List the annual global top 1-top 10 grossing games downloaded via iPhone & iPa… | 5 | 50 | 50 | 63.6 | 28.0 | |
| ws_en_061 | Please compile the names of every individual who has served as a chief executi… | 8 | 25 | 25 | 100.0 | 100.0 | ✓ |
| ws_en_062 | Please help me compile the founding year, governing authority, official websit… | 6 | 24 | 24 | 93.1 | 62.5 | |
| ws_en_063 | I would like to obtain information on the per capita health expenditure and li… | 6 | 20 | 20 | 85.8 | 50.0 | |
| ws_en_064 | I am currently representing a case and need to gather information from the cod… | 4 | 41 | 41 | 93.9 | 75.6 | |
| ws_en_065 | Curious about the birth rate, death rate, and the overall population of the Un… | 6 | 12 | 12 | 100.0 | 100.0 | ✓ |
| ws_en_066 | I am analyzing the regional adaptation of Dyson’s Supersonic hair dryers (cons… | 6 | 15 | 15 | 74.4 | 26.7 | |
| ws_en_067 | Please gather data on girls’ single-sex high schools operating in Seoul as of… | 4 | 66 | 77 | 65.4 | 47.6 | |
| ws_en_068 | Please provide a table of the books on the “Ten Best Books List” of The New Yo… | 5 | 30 | 30 | 95.3 | 83.3 | |
| ws_en_069 | I am conducting film research and need you to list the top five domestic films… | 5 | 25 | 25 | 99.2 | 96.0 | |
| ws_en_070 | My child is about to finish undergraduate studies and is planning to pursue a… | 4 | 32 | 32 | 75.8 | 40.6 | |
| ws_en_071 | Please list the top 10 individuals from the Forbes Global Billionaires Ranking… | 6 | 60 | 60 | 94.7 | 85.0 | |
| ws_en_072 | For a paper on U.S. politics, please list the U.S. presidents and vice preside… | 6 | 60 | 60 | 99.4 | 96.7 | |
| ws_en_073 | Please help me sort out a list of NVIDIA’s desktop graphics card products rele… | 10 | 510 | 375 | 71.5 | 49.9 | |
| ws_en_074 | Please track the performance of political parties labelled as far-right in the… | 7 | 24 | 5 | 26.6 | 20.7 | |
| ws_en_075 | It’s now June 2025, and I’ve recently become fascinated with rap music. I’d li… | 4 | 125 | 125 | 98.2 | 97.6 | |
| ws_en_076 | Please list the specific military expenditure (in billions of US dollars, such… | 7 | 30 | 30 | 96.7 | 80.0 | |
| ws_en_077 | How many changes occurred among cabinet secretaries and cabinet-level official… | 5 | 22 | 22 | 95.5 | 95.5 | |
| ws_en_078 | I am conducting a research project on the growth trajectory of female power an… | 4 | 54 | 57 | 73.9 | 73.9 | |
| ws_en_079 | I would like to know the operating performance in 2024 of the companies grante… | 9 | 8 | 9 | 81.0 | 23.5 | |
| ws_en_080 | I need you to help me count the number of gold medals won by delegations from… | 4 | 11 | 11 | 100.0 | 100.0 | ✓ |
| ws_en_081 | Give me some fun action games, please help retrieve the games on the 2024 Stea… | 3 | 12 | 12 | 100.0 | 100.0 | ✓ |
| ws_en_082 | Give me the resume backgrounds of the female Go players who have reached the t… | 7 | 87 | 44 | 53.0 | 29.2 | |
| ws_en_083 | Please organize a table showing the champion teams of each of the European top… | 6 | 25 | 25 | 95.3 | 80.0 | |
| ws_en_084 | Compile the campus shootings in the United States from 2010 to 2024 (including… | 8 | 52 | 171 | 31.8 | 12.6 | |
| ws_en_085 | Please help me compile information on the films that were in the line-up for t… | 5 | 21 | 21 | 100.0 | 100.0 | ✓ |
| ws_en_086 | Please compile a list of the girl groups from SM Entertainment and YG Entertai… | 6 | 19 | 17 | 82.4 | 77.8 | |
| ws_en_087 | Please sort out all African countries involved in the “Belt and Road” initiati… | 6 | 52 | 52 | 94.2 | 67.3 | |
| ws_en_088 | Billboard publishes its year-end singles chart each year. I would like informa… | 4 | 100 | 100 | 100.0 | 100.0 | ✓ |
| ws_en_089 | I want to research Apple Inc.’s product development history by compiling a lis… | 5 | 40 | 38 | 83.6 | 35.9 | |
| ws_en_090 | I want to check the recent performance of large models. Please list Gemini 2.5… | 7 | 17 | 17 | 84.0 | 47.1 | |
| ws_en_091 | It is said that “Mom Su” has saved AMD. Please help me sort out the specific p… | 12 | 331 | 438 | 55.2 | 0.0 | |
| ws_en_092 | I would like to know about the boy groups and girl groups (no sub-units) that… | 5 | 47 | 45 | 68.3 | 0.0 | |
| ws_en_093 | I want to catch up with the current large language model trends, so I need to… | 4 | 129 | 22 | 17.7 | 9.3 | |
| ws_en_094 | I’m compiling a list of the top three box office films released in South Korea… | 7 | 45 | 45 | 86.3 | 66.7 | |
| ws_en_095 | Please help me count the Korean dramas that aired on Netflix between January 2… | 5 | 38 | 38 | 70.0 | 55.3 | |
| ws_en_096 | I want to buy a camera to take landscape photos. I heard that Nikon is good at… | 9 | 14 | 14 | 95.2 | 64.3 | |
| ws_en_097 | Tony Leung is a well-known film actor. Please help me sort out all the movies… | 6 | 29 | 30 | 76.8 | 10.2 | |
| ws_en_098 | I’m wondering about the recent development of Africa. Can you list the GDP ann… | 4 | 48 | 48 | 71.9 | 31.2 | |
| ws_en_099 | Compile a comprehensive list of U.S. wildfires that caused no less than 10 dea… | 7 | 12 | 11 | 51.9 | 18.2 | |
| ws_en_100 | Help me to compile a complete list of U.S. Supreme Court justices who were suc… | 8 | 21 | 21 | 100.0 | 100.0 | ✓ |
Build your own table
The datasets behind this result are the same kind you build in Riveter for any list: describe the rows, name the columns, and Riveter finds the rows and fills every cell with a source. Start free or book a demo.
Related reading: Company Enrichment benchmark · Lookalike benchmark · Company Funding benchmark
Run on November 5–6, 2026 against the 100 English tasks of WideSearch, one trial, scores reported over 99 of them (ws_en_048 set aside for a leaked answer), graded with the WideSearch harness at commit 9825ba7 and its default judge model, gpt-4.1-2025-04-14. Published English-task results from the WideSearch paper, Table 5 (WideSearch-en), average and best of four trials. Leaderboard figures from widesearch-seed.github.io, read November 6, 2026. Runner code: pull request #15.
