Evaluating Elicit’s Paper Search

On BioASQ, a widely used biomedical search benchmark, Elicit found more of the papers experts used to answer their questions than any of the five other search systems we tested. Across questions, Elicit found 60.3% of each question’s gold-standard papers on average in the first 50 results, compared with 47.4% for the next-best system. Elicit also had the highest recall at every result count tested, from 10 to 200.

Motivation

We recently launched the Elicit API and MCP, bringing Elicit’s building blocks to you. Our ambition is to be the most comprehensive scientific search engine. When called by an agent, Elicit needs to find and return the best evidence so that the agent can investigate the mechanism behind a drug target or a compound’s safety signal. So, we set out to measure our search performance using two sources: BioASQ, a widely used benchmark for biomedical question answering and search, and a sample of anonymized queries from Elicit users.

Elicit achieved the highest recall on BioASQ, finding more of the relevant papers than any other search system we tested. On a separate sample of 399 anonymized Elicit searches, papers from the updated search were also preferred over those from the version it replaced.

What we evaluated

BioASQ is an established benchmark for biomedical question answering and search. Its dataset and construction were described in a peer-reviewed paper in Scientific Data, a Nature Portfolio journal. A 500-question version of BioASQ is also included in BEIR, a widely used benchmark suite for comparing information retrieval systems. We use Phase A of Task B where experts write questions which reflect their information needs. For each question, the expert identifies a set of papers sufficient to support an answer, which becomes our gold standard retrieval set.

We used BioASQ because the questions are drawn from the real world needs of practicing biomedical experts who also identify papers they consider relevant and sufficient to answer each question. We used BioASQ to compare Elicit’s semantic search API with other commercial and academic search APIs.

We also wanted to ensure that improvements to Elicit’s search engine held up across the broader range of questions our users ask. Alongside BioASQ, we evaluated 399 anonymized searches: 300 from Elicit’s public API and 99 from the Find Papers workflow.

How we evaluated

The BioASQ benchmark

We combined all questions from 12 years of BioASQ’s official Golden Enriched test sets, which pair each question with papers biomedical experts considered sufficient to answer it. To recreate the literature available when each question was evaluated, we excluded papers published in later years when making API calls to search engines. Together, the test set contained 5,486 questions, with an average of 10.2 papers in the gold standard per question and a median of 6. The section “Benchmark construction and deduplication” in the appendix has more details.

One BioASQ question asks, “Are astronauts in higher risk [sic] for developing cancer?”

The gold standard for this question contains two papers:

• Chancellor et al. (2018), Limitations in predicting the space radiation health risk for exploration astronauts (PMID 29644336)

• Elgart et al. (2018), Radiation Exposure and Mortality from Cardiovascular Disease and Cancer in Early NASA Astronauts (PMID 29855508)

How we queried each system

For semantic search providers, we submitted the original BioASQ question. For keyword search providers, we used Claude Opus 4.8 to generate a separate query tailored to each provider’s syntax. For example, the astronaut question above became astronauts cancer risk space radiation for Google Scholar. When Claude returned a safety refusal, GPT 5.6 Sol generated the query using the same prompt. We also used GPT 5.6 Sol to make minimal repairs to queries containing unsupported wildcard syntax. The appendix gives the full counts and describes an additional formatting fix for Semantic Scholar.

We asked each provider for up to 200 results. Consensus returned at most 20, and OpenAlex’s semantic search at most 50.

How we scored the results

We wanted to measure whether each search engine returned the papers BioASQ experts identified as sufficient to answer each question. We used recall, which measures the percentage of those papers found by the search engine. We matched results to BioASQ’s gold standard using PMIDs where available, then DOIs, and finally titles. The appendix gives the full breakdown by search system.

To see how performance changed as we considered more results, we measured recall at several cutoffs. At a cutoff of 20, for example, only papers among the first 20 results in the API’s ranking count. We calculated recall for each question, then averaged across questions so each question contributed equally to the final score.

We chose recall rather than precision or mean average precision because BioASQ’s gold standard is not exhaustive. Experts identified enough papers to answer each question, not every relevant paper. A search engine may return a useful paper that BioASQ’s experts did not include. Precision and mean average precision would count that paper as irrelevant, penalizing the search engine because the label is missing rather than because the result is poor. Recall does not directly count unlabeled papers as errors. It measures whether the search engine found the known gold-standard papers.

Each provider’s average includes only requests recorded as successful. Elicit and Consensus completed all 5,486 requests, while other providers failed on some questions. We excluded those failures rather than assigning them zero recall and report the number included for each provider. As a result, the averages are not always based on exactly the same questions.

Results

Elicit’s search API has the highest recall of any search API we tested. Below is a table showing the recall of each search API at increasing cutoffs (10, 20, 50, 100, and 200 results). We compute the recall of every question and report the average of averages (macro-recall).

Elicit achieved the highest recall of all six search systems we tested at every result depth, from 10 to 200 results.

Higher recall means that, for the same biomedical question and number of search results, an agent using Elicit would retrieve more of BioASQ’s gold-standard papers on average. The agent could then build its answer from a more complete evidence base. Elicit’s lead persisted at every cutoff. The largest gap was at 50 results, where Elicit achieved 60.3% average recall, compared with 47.4% for the next-best workflow, OpenAlex keyword search. That is a 13 percentage-point advantage, or 27% higher recall.

Beyond BioASQ

The BioASQ results above are for an updated version of Elicit search. We wanted to make sure that its stronger performance on BioASQ did not come at the expense of relevance on the searches people actually run in Elicit.

We compared the updated search with the version it replaced on 399 searches from Elicit users. Before evaluating them, we removed all personally identifiable information. The sample included 300 searches from Elicit’s public API and 99 from the Find Papers workflow.

The searches covered a broader range of fields and formats than BioASQ. API searches included short topic descriptions, such as an overview of CAR-T therapy or clean-energy industrial policy and supply chains. Find Papers searches were more often complete research questions, including how PFAS behaves during plasma water treatment.

For each search, we selected six pairs of papers from the two versions’ top 10 results, with one paper from each version in every pair. GPT 5.5 judged which paper was more relevant without knowing which version returned it or where it ranked. We wrote the judging instructions and manually reviewed a sample of its judgments as a quality check.

The updated search received a 54.1% preference score, where 50% indicates no overall preference between the versions and ties count as half. The 95% confidence interval was 51.7% to 56.4%. The result was above 50% for both sources: 53.2% for public API searches and 56.9% for Find Papers searches.

The updated search performed better not only on BioASQ, but also across this separate sample of real Elicit searches.

Limitations

BioASQ’s sufficient papers are not an exhaustive list of every relevant paper. Recall therefore measures how often each system retrieved the papers BioASQ’s experts identified as sufficient, rather than every paper that could help answer the question. BioASQ is also biomedical and draws its papers from PubMed, so this evaluation does not establish that the same ordering holds across every scientific field.

The comparison is a snapshot of each API when we ran it. Other providers failed on some questions, and we calculated their averages only over successful requests rather than assigning those failures zero recall. This may make their reported averages higher than if failures counted as misses. Our evaluation on Elicit searches used GPT 5.5 judgments of sampled paper pairs rather than judgments from researchers or a measure of downstream task performance. A public, cross-domain benchmark would make it easier to test scientific search across more fields and research tasks.

Conclusion

Across 5,486 BioASQ questions, Elicit achieved the highest recall of all six systems at every result depth tested. The updated search was also preferred over the version it replaced on a separate sample of 399 real Elicit searches. For researchers and agents, higher recall puts more of the evidence needed to answer a question within the same number of search results.

Reproduce this evaluation

The evaluation repository contains the code and data needed to reproduce the Elicit results, including BioASQ inputs, keyword-generation prompts and queries, configuration, Elicit outputs, and data attribution. It also includes fetchers for running the evaluation against the other search APIs. We do not include competitor result payloads.

Appendix

Benchmark construction and deduplication

BioASQ identifies each question with an ID. All 5,486 records in this evaluation had distinct IDs, so none were removed as duplicates. We kept records with repeated wording when BioASQ gave them different IDs. Among the 5,486 records, 115 fell into 57 groups with the same question type and wording, ignoring capitalization and extra spaces.

Ten of the 51 groups with different sufficient papers also appeared in different challenge years. Collapsing the six remaining pairs into single questions changes macro recall by at most 0.0003 and does not change which system performs best at any result depth.

Keyword query generation and repair

We generated three keyword queries per BioASQ question, one each for Semantic Scholar, OpenAlex, and Google Scholar, for 16,458 provider-question queries in total.

• Opus 4.8 generated 16,353 queries.

• GPT 5.6 Sol generated 105 replacements for Opus refusals: 40 for Semantic Scholar, 34 for OpenAlex, and 31 for Google Scholar. GPT received the same provider-specific prompt as Opus.

• GPT 5.6 Sol later repaired unsupported wildcard syntax in 83 of the Opus-generated queries: 75 for OpenAlex and 8 for Google Scholar. Each repair prompt included the original question, rejected query, validation error, and provider constraints. The prompt instructed GPT to preserve the query and change only what was necessary. All 83 repairs produced valid queries on the first attempt.

• Before calling Semantic Scholar, we replaced unsupported hyphens with spaces in 164 generated queries.

The evaluation repository contains the exact prompts, generated queries, request strings, models, and query-version identifiers.

Request coverage

The table below shows how many questions each search engine was able to answer.

Paper matching

This table below shows the method by which each provider’s correct answers were matched to BioASQ’s gold standard papers

Statistical significance

At each result depth, we compared Elicit with the non-Elicit system with the highest macro recall at that depth. The confidence intervals use 50,000 paired bootstrap resamples of whole questions and are corrected across all 20 provider-depth comparisons.

All five intervals are above zero. Counting OpenAlex’s nine failed keyword requests as zero recall gives the same conclusion at 50, 100, and 200 results.

Evaluation on Elicit searches

The analysis included 399 searches: 300 from public API usage and 99 from Find Papers. For each search, we used a fixed random seed to select six pairs from the two systems’ top 10 results. Each pair contained one paper from the improved search and one from the version it replaced.

GPT 5.5 judged which paper was more relevant to the search without seeing which system returned it or where it ranked. A win for the improved search scored 1, a tie or comparison of the same paper scored 0.5, and a win for the previous version scored 0. We averaged the six scores for each search, then averaged across searches so that every search had equal weight.

The 95% confidence interval comes from 10,000 bootstrap samples taken at the search level, with the public API and Find Papers groups kept in the same proportions.