Research

Riveter tops WideSearch: 81.4 item F1, 14 points above the best published system

Cody Watters· CTO, RiveterPublished · Updated

Riveter scored 81.4 item F1 on the English tasks of WideSearch, ByteDance Seed’s benchmark for agents that build large tables from the web. We graded the run with the benchmark’s own harness and judge model, unchanged, and report 99 of the 100 tasks: one is set aside because an agent found its answer key (Checking for leaked answers). On the English tasks, the best published system, Claude Sonnet 4 run as a multi-agent system, averages 67.2, and the best single attempt by any published system reaches 76.7.

Bar chart of item F1 on the English tasks of the WideSearch benchmark: Riveter 81.4 on this run, against the published average of four trials for each system: Claude Sonnet 4 (Thinking) multi-agent 67.2, Kimi K2 multi-agent 65.4, Gemini end-to-end system 65.2, OpenAI o3 multi-agent 64.3, Gemini 2.5 Pro multi-agent 63.1, Claude Sonnet 4 (Thinking) single agent 62.0, Doubao-Seed-1.6 (Thinking) multi-agent 60.4, OpenAI o3 single agent 60.0, Kimi K2 single agent 59.9, Doubao-Seed-1.6 (Non-Thinking) multi-agent 59.0, Gemini 2.5 Pro single agent 56.4, Doubao-Seed-1.6 (Non-Thinking) single agent 55.1, Claude end-to-end system 53.3, Doubao-Seed-1.6 (Thinking) single agent 52.8, DeepSeek-R1 multi-agent 50.9, DeepSeek-R1 single agent 48.6, OpenAI o3 end-to-end system 45.0
Item F1, WideSearch's most granular score, on the English tasks: Riveter's run against every system in the WideSearch paper's English results.

Benchmark: widesearch-seed.github.io · Paper: arXiv 2508.07999 · Code and data: ByteDance-Seed/WideSearch · Riveter runner: pull request #15 · Our run: results CSV, 100 tasks

What the benchmark measures

WideSearch tests wide information-seeking: tasks where no single fact is hard to find, but the answer is a whole table assembled from many pages. Each task is a request based on a real user query, and its answer is a table the benchmark’s authors built and checked by hand. Some examples from the set:

  • every concert on Taylor Swift’s official tours from 2010 to May 2025, with date, tour, country, city, and venue (534 rows)
  • the world’s ten busiest airports for each year from 2020 to 2024, with location, passenger count, and codes
  • every U.S. federal government shutdown since 1976, with dates, leaders, and furloughed employees

The benchmark has 200 tasks: 100 in English and 100 in Chinese. The grader matches each returned row to a gold row on the table’s key columns, then scores every other cell. Exact values must match exactly, numbers must fall within a tolerance, and names and descriptions are judged by an LLM.

It reports three scores:

  • Item F1: cells right, weighed against cells expected and cells returned. Missing rows and extra rows both count against it.
  • Row F1: the same, except a row counts only when every cell in it is right.
  • Success rate: the share of tables that are entirely right, every row and every cell.

How we ran it

Each task became one Riveter dataset:

  1. Row count: one Riveter search call reads the request and estimates how many rows a complete answer has.
  2. List build: Riveter’s list builder takes the request word for word, with the table’s columns. It searches and reads the web and writes the rows.
  3. Row checks: one agent per row checks each value against a source and corrects it.

The finished rows go out as a Markdown table in the request’s column names, the format the grader reads.

Riveter saw only the request and its column names. There was no per-task tuning and no hand edits, and nothing from the gold tables or the grading rules reached Riveter. We graded with the harness’s evaluation code and its judge model, GPT-4.1 (gpt-4.1-2025-04-14), unchanged. The only change is the client that calls the judge: the harness builds an Azure client, and we send the same requests through OpenAI’s.

The result

On the English tasks, against the published results in the WideSearch paper (Table 5, WideSearch-en):

Riveter (1 trial, 99 tasks) Best published average of 4 trials Best published single attempt (of 4)
Item F1 81.4 67.2 (Claude Sonnet 4, multi-agent) 76.7 (Claude Sonnet 4, multi-agent)
Row F1 59.8 41.8 (OpenAI o3, multi-agent) 53.2 (OpenAI o3, multi-agent)
Success rate 14.1% 7.5% (OpenAI o3, multi-agent) 13% (OpenAI o3, single agent and multi-agent)

Riveter’s item precision was 84.3 and its item recall 81.7. It returned 14 of the 99 tables exactly right. 38 tasks scored 95 or higher, and 49 scored 90 or higher; the median task scored 89.5. The largest table, the Taylor Swift concerts, has 534 rows: Riveter returned 533 and scored 97.6.

Bar chart of row F1 on the English tasks of the WideSearch benchmark: Riveter 59.8 on this run, against the published average of four trials for each system: OpenAI o3 multi-agent 41.8, Gemini end-to-end system 41.0, Claude Sonnet 4 (Thinking) multi-agent 40.1, OpenAI o3 single agent 38.7, Kimi K2 multi-agent 37.7, Gemini 2.5 Pro multi-agent 36.0, Doubao-Seed-1.6 (Thinking) multi-agent 35.1, Claude Sonnet 4 (Thinking) single agent 33.2, Gemini 2.5 Pro single agent 33.0, Doubao-Seed-1.6 (Non-Thinking) multi-agent 32.4, Kimi K2 single agent 31.5, Doubao-Seed-1.6 (Thinking) single agent 30.6, Doubao-Seed-1.6 (Non-Thinking) single agent 28.9, Claude end-to-end system 27.4, DeepSeek-R1 multi-agent 24.6, DeepSeek-R1 single agent 22.9, OpenAI o3 end-to-end system 20.4
Row F1, the share of rows with every cell right, on the English tasks: Riveter's run against every system in the WideSearch paper's English results.

The leaderboard on the benchmark’s site averages the English and Chinese tasks together. Its best scores are 62.2 item F1, 38.5 row F1, and a 5.1% success rate, each an average of four trials over the 200 tasks. Riveter has not run the Chinese tasks.

What the comparison shows and what it does not

  • Same tasks, same grader, same judge. Riveter’s tables went through the code and judge model behind the published results. Riveter’s scores cover 99 of the 100 English tasks; the published ones cover all 100.
  • One trial against an average of four. Riveter ran each task once. The published averages come from four trials per task, and the best-of-four column takes each system’s best trial.
  • Different model generations. The published systems are mid-2025 models: Claude Sonnet 4, OpenAI o3, Gemini 2.5 Pro. Riveter runs on current models.
  • The judge is not deterministic. The harness runs its LLM judge at the API’s default temperature, so a task’s score can move by several points between gradings. A second grading of the same tables, which also re-asked the judge whenever a reply skipped a pair, gave 81.6 item F1 and 59.9 row F1.

Where the accuracy comes from

The two stages are scored separately because each task’s dataset keeps both. The list builder’s table alone scored 79.8 item F1 and 56.9 row F1. After the row checks it scored 81.4 and 59.8. The checks mattered most where the builder had the right rows but some wrong numbers:

Task List build alone After row checks
Five-year financials for Apple, Amazon, Google, Microsoft, and Netflix 64.7 97.3
New U.S. citizens by region of birth, fiscal years 2002–2022 10.0 50.5
Average monthly house prices in England, Scotland, Wales, and Northern Ireland 41.0 69.5
AMD’s Zen-based processors since Lisa Su became CEO 43.3 55.2

22 tasks rose by more than 2 points after the checks, and 10 fell by more than 2.

Where Riveter missed

Most of the lost points come from returning the wrong set of rows, not from wrong cells. On the 82 tasks where Riveter’s row count landed within 10% of the gold table’s, it averaged 87.8 item F1. On the other 17 it averaged 50.8.

  • Too few rows. The row-count estimate caps the build. In ws_en_023 (sociology articles on migration from three journals, 2020–2024), the estimate failed and the build stopped at 3 of 56 articles. Far-right party results in the United Kingdom, France, and Germany (5 of 24 rows) and the careers of top-ranked women Go players (44 of 87) were capped the same way.
  • Too many rows. For U.S. Department of Justice attorney postings from January to June 2025, Riveter returned 105 listings against the gold table’s 12. For U.S. campus shootings from 2010 to 2024, it returned 171 incidents against 52, a broader reading of the request than the gold table’s.
  • A different identifier. Asked for the “library ID” of 50 research libraries, Riveter wrote each institution’s IPEDS unit ID, while the gold table uses the libraries’ own IDs. No row matched, and the task scored 0.
  • Values that changed. The J.D. Power 2025 ratings Riveter found differ from the gold table’s, most likely because the published scores changed after the table was built.

Checking for leaked answers

WideSearch’s gold tables are public on Hugging Face, so an agent searching the web can find them. We checked every source Riveter’s row agents cited (29,383 citations across 1,952 domains) and the logs of every list build. In one task, ws_en_048, a row agent cited the benchmark’s gold file for two cells: the 2023 violent-crime rates of New York and Los Angeles. No list build’s logs reference the gold files.

Across all 100 tasks, Riveter scored 81.5 item F1 and 59.5 row F1, with 14 of 100 tables exactly right. Because one of those tasks had seen its answer key, we set ws_en_048 aside and report the other 99 throughout this post: 81.4 item F1 and 59.8 row F1, with 14 of 99 tables exactly right. Row F1 rises when the task is removed because ws_en_048 itself scored 30.0 on row F1, below the average.

Full results

Every task, with the number of rows in the gold table and in Riveter’s, and Riveter’s scores. “Exact” marks a table that is entirely right; ws_en_048 is listed but marked “set aside” and left out of every total. The results CSV carries each task’s full request, item precision and recall, the list-build-only score, the second grading, and whether the task is in the reported totals. Tasks and gold tables from the WideSearch dataset, CC0 1.0.

Output (99 tasks, ws_en_048 set aside) Item F1 Item precision Item recall Row F1 Success rate
Riveter, list build + row checks (reported) 81.44 84.26 81.70 59.79 14.1%
Riveter, list build only 79.77 82.47 80.04 56.85 15.2%
Show all 100 tasks
Task What it asks for Columns Gold rows Rows returned Item F1 Row F1 Exact
ws_en_001 My son is about to start his university applications in 2025 for postgraduates… 8 25 25 78.0 0.0
ws_en_002 I’m currently mapping the product portfolios of several spirits brands includi… 6 38 37 71.1 66.7
ws_en_003 Could you help compile a detailed dataset of Eileen Gu’s competitive achieveme… 6 58 47 79.0 71.8
ws_en_004 Could you help me compile a list of all base models from Samsung’s Galaxy S Se… 14 17 17 91.6 58.8
ws_en_005 I’m planning to apply for full-time Master’s programs in civil engineering sta… 4 16 10 73.1 61.5
ws_en_006 Could you list every single concert on Taylor Swift’s official tour from Janua… 5 534 533 97.6 88.3
ws_en_007 I would like a broad overview of multiple spaceflight missions across differen… 11 37 30 88.2 77.6
ws_en_008 Could I obtain monthly data from January 2025 to May 2025 (including January 2… 6 25 25 100.0 100.0 ✓
ws_en_009 I want to apply for a full-time MA program at Goldsmiths College, University o… 6 22 22 71.2 68.2
ws_en_010 Could you provide an updated list of all officially recognized Areas of Outsta… 7 86 86 92.0 44.2
ws_en_011 According to the J.D. Power Awards Best Car Ratings of 2025, please compile th… 14 4 4 78.6 0.0
ws_en_012 Please compile a table listing the “CNN Hero of the Year” for every year the a… 9 16 16 96.5 68.8
ws_en_013 I’m a huge fan of Lana Del Rey, can you list all the songs of her official stu… 3 109 109 100.0 100.0 ✓
ws_en_014 What are the world’s busiest airports by passenger traffic? List the top ten b… 6 50 50 98.0 96.0
ws_en_015 I would like to manage my property and am interested in learning about the top… 4 50 50 92.0 68.0
ws_en_016 Michael Phelps is one of the most decorated Olympians in history. I am interes… 9 36 36 93.5 50.0
ws_en_017 Please help me organize a table based on The 13 USA National Parks given below… 7 13 13 97.8 84.6
ws_en_018 Need to analyze the trend of the U.S. federal government spending and deficit… 6 10 10 85.0 10.0
ws_en_019 I am working on a tracking report regarding the biotechnology and pharmaceutic… 9 5 5 86.7 20.0
ws_en_020 Please provide a complete list, up to 2024 (excluding 2024), of UNESCO World H… 6 81 80 97.9 90.7
ws_en_021 Could you provide a detailed list of Michelin three-star restaurants in Paris… 3 10 10 80.0 40.0
ws_en_022 I need a list of the top 10 most-streamed songs globally and top 10 most-strea… 8 20 20 96.9 80.0
ws_en_023 Please help me find articles published in the American Journal of Cultural Soc… 7 56 3 10.2 10.2
ws_en_024 Please research the following information about Super Bowl: date, champions, h… 7 25 25 100.0 100.0 ✓
ws_en_025 Make a comprehensive, verified list of abortion clinics currently operating in… 5 6 6 93.3 66.7
ws_en_026 Help me collect the basic electoral information of 50 U.S. states in 2024. 1)… 5 50 50 99.6 98.0
ws_en_027 I want to travel to London soon and I bought a London Pass. Please help me fin… 4 25 26 52.9 23.5
ws_en_028 I’m curious about the market shares of different North America burger brands… 5 5 5 100.0 100.0 ✓
ws_en_029 I’m a huge Marvel fan, but I’ve never gone through the entire Marvel Cinematic… 4 34 34 100.0 100.0 ✓
ws_en_030 Please provide a summary of recent decisions made by the U.S. Supreme Court fr… 5 25 25 96.8 84.0
ws_en_031 I’m putting together an introductory video on Angelina Jolie, Nicole Kidman, C… 5 53 56 86.6 44.0
ws_en_032 I am doing research on public transportation in Sydney. I wanna know each Sydn… 4 10 10 97.5 90.0
ws_en_033 Please organize the following data for Apple, Amazon, Google, Microsoft, and N… 6 25 25 97.3 84.0
ws_en_034 Please track and compile the average monthly house prices for England, Scotlan… 5 40 40 69.5 47.5
ws_en_035 I want to compare the labor market between four southern US states, namely Ala… 8 24 24 97.4 79.2
ws_en_036 I’m seeking permanent government job opportunities with the Department of Just… 7 12 105 15.4 0.0
ws_en_037 I’m a huge ‘Love is Blind’ fan and want to know which couples got married in s… 3 13 13 100.0 100.0 ✓
ws_en_038 Summarize all smartphones released by Google between January 2022 and July 202… 9 13 13 98.3 84.6
ws_en_039 Please provide a list of the top 50 universities in the QS 2026 World Universi… 4 50 50 92.0 76.0
ws_en_040 I’d like to get a list of the top 10 apps downloaded worldwide each year from… 7 30 30 89.5 80.0
ws_en_041 I’ve been in a bit of a reading slump and want to find some new books. Could y… 6 75 75 96.6 86.5
ws_en_042 Please compare the growth trajectory of the leading global e-commerce platform… 5 16 16 95.0 75.0
ws_en_043 We are going to visit America this summer and we love natural scenery, especia… 4 63 63 99.6 98.4
ws_en_044 Hey, I’m gonna be in South Korea for a whole month, and I need your help grabb… 4 40 35 62.0 34.7
ws_en_045 According to the list of research libraries in 2023 released by IPEDS, please… 6 50 50 0.0 0.0
ws_en_046 Compile comprehensive data for all major international tournaments during Lang… 5 9 9 77.8 11.1
ws_en_047 I am studying how partisan standoffs influence administrative efficiency and t… 9 20 13 9.4 0.0
ws_en_048 What’s the correlation between police funds and violent-crime trends? Using th… 8 10 10 85.0 30.0 set aside
ws_en_049 I’d like to know the information about all Ford vehicles first launched and re… 9 10 11 72.0 57.1
ws_en_050 For a comparative politics study on descriptive representation, compile the sh… 9 38 38 89.2 36.8
ws_en_051 I’m interested in physics. Could you please provide a list of the Best 50 U.S… 7 50 15 41.3 15.4
ws_en_052 For an infrastructure-finance paper, I need to benchmark capital intensity of… 7 49 48 91.9 74.2
ws_en_053 I am preparing a policy brief on post-millennial commercial nuclear deployment… 8 85 87 87.4 70.6
ws_en_054 List every mega airport project commenced earlier than 31 Dec 2024 (including… 5 16 16 78.8 43.8
ws_en_055 Retrieve the details about, Location ( Country, City and or State/Province), O… 4 25 25 100.0 100.0 ✓
ws_en_056 My crush is obsessed with NBC’s The Voice—he has binged Season 1~26. I’m putti… 12 26 26 83.7 0.0
ws_en_057 Identify the feature-length animated films produced by Walt Disney Animation S… 4 26 26 85.6 50.0
ws_en_058 Could you please give me a demographic brief analyzing the shifting racial com… 10 21 21 50.5 19.0
ws_en_059 I need to verify the basic information for all TED Prize winners from 2005 to… 4 21 21 98.8 95.2
ws_en_060 List the annual global top 1-top 10 grossing games downloaded via iPhone & iPa… 5 50 50 63.6 28.0
ws_en_061 Please compile the names of every individual who has served as a chief executi… 8 25 25 100.0 100.0 ✓
ws_en_062 Please help me compile the founding year, governing authority, official websit… 6 24 24 93.1 62.5
ws_en_063 I would like to obtain information on the per capita health expenditure and li… 6 20 20 85.8 50.0
ws_en_064 I am currently representing a case and need to gather information from the cod… 4 41 41 93.9 75.6
ws_en_065 Curious about the birth rate, death rate, and the overall population of the Un… 6 12 12 100.0 100.0 ✓
ws_en_066 I am analyzing the regional adaptation of Dyson’s Supersonic hair dryers (cons… 6 15 15 74.4 26.7
ws_en_067 Please gather data on girls’ single-sex high schools operating in Seoul as of… 4 66 77 65.4 47.6
ws_en_068 Please provide a table of the books on the “Ten Best Books List” of The New Yo… 5 30 30 95.3 83.3
ws_en_069 I am conducting film research and need you to list the top five domestic films… 5 25 25 99.2 96.0
ws_en_070 My child is about to finish undergraduate studies and is planning to pursue a… 4 32 32 75.8 40.6
ws_en_071 Please list the top 10 individuals from the Forbes Global Billionaires Ranking… 6 60 60 94.7 85.0
ws_en_072 For a paper on U.S. politics, please list the U.S. presidents and vice preside… 6 60 60 99.4 96.7
ws_en_073 Please help me sort out a list of NVIDIA’s desktop graphics card products rele… 10 510 375 71.5 49.9
ws_en_074 Please track the performance of political parties labelled as far-right in the… 7 24 5 26.6 20.7
ws_en_075 It’s now June 2025, and I’ve recently become fascinated with rap music. I’d li… 4 125 125 98.2 97.6
ws_en_076 Please list the specific military expenditure (in billions of US dollars, such… 7 30 30 96.7 80.0
ws_en_077 How many changes occurred among cabinet secretaries and cabinet-level official… 5 22 22 95.5 95.5
ws_en_078 I am conducting a research project on the growth trajectory of female power an… 4 54 57 73.9 73.9
ws_en_079 I would like to know the operating performance in 2024 of the companies grante… 9 8 9 81.0 23.5
ws_en_080 I need you to help me count the number of gold medals won by delegations from… 4 11 11 100.0 100.0 ✓
ws_en_081 Give me some fun action games, please help retrieve the games on the 2024 Stea… 3 12 12 100.0 100.0 ✓
ws_en_082 Give me the resume backgrounds of the female Go players who have reached the t… 7 87 44 53.0 29.2
ws_en_083 Please organize a table showing the champion teams of each of the European top… 6 25 25 95.3 80.0
ws_en_084 Compile the campus shootings in the United States from 2010 to 2024 (including… 8 52 171 31.8 12.6
ws_en_085 Please help me compile information on the films that were in the line-up for t… 5 21 21 100.0 100.0 ✓
ws_en_086 Please compile a list of the girl groups from SM Entertainment and YG Entertai… 6 19 17 82.4 77.8
ws_en_087 Please sort out all African countries involved in the “Belt and Road” initiati… 6 52 52 94.2 67.3
ws_en_088 Billboard publishes its year-end singles chart each year. I would like informa… 4 100 100 100.0 100.0 ✓
ws_en_089 I want to research Apple Inc.’s product development history by compiling a lis… 5 40 38 83.6 35.9
ws_en_090 I want to check the recent performance of large models. Please list Gemini 2.5… 7 17 17 84.0 47.1
ws_en_091 It is said that “Mom Su” has saved AMD. Please help me sort out the specific p… 12 331 438 55.2 0.0
ws_en_092 I would like to know about the boy groups and girl groups (no sub-units) that… 5 47 45 68.3 0.0
ws_en_093 I want to catch up with the current large language model trends, so I need to… 4 129 22 17.7 9.3
ws_en_094 I’m compiling a list of the top three box office films released in South Korea… 7 45 45 86.3 66.7
ws_en_095 Please help me count the Korean dramas that aired on Netflix between January 2… 5 38 38 70.0 55.3
ws_en_096 I want to buy a camera to take landscape photos. I heard that Nikon is good at… 9 14 14 95.2 64.3
ws_en_097 Tony Leung is a well-known film actor. Please help me sort out all the movies… 6 29 30 76.8 10.2
ws_en_098 I’m wondering about the recent development of Africa. Can you list the GDP ann… 4 48 48 71.9 31.2
ws_en_099 Compile a comprehensive list of U.S. wildfires that caused no less than 10 dea… 7 12 11 51.9 18.2
ws_en_100 Help me to compile a complete list of U.S. Supreme Court justices who were suc… 8 21 21 100.0 100.0 ✓

Build your own table

The datasets behind this result are the same kind you build in Riveter for any list: describe the rows, name the columns, and Riveter finds the rows and fills every cell with a source. Start free or book a demo.

Related reading: Company Enrichment benchmark · Lookalike benchmark · Company Funding benchmark


Run on November 5–6, 2026 against the 100 English tasks of WideSearch, one trial, scores reported over 99 of them (ws_en_048 set aside for a leaked answer), graded with the WideSearch harness at commit 9825ba7 and its default judge model, gpt-4.1-2025-04-14. Published English-task results from the WideSearch paper, Table 5 (WideSearch-en), average and best of four trials. Leaderboard figures from widesearch-seed.github.io, read November 6, 2026. Runner code: pull request #15.

See Riveter in action

Build your first dataset in minutes.