We evaluate Elicit against web search engines on BioASQ, a widely used biomedical search benchmark that we previously used to evaluate paper search engines. Elicit achieves the highest recall scoring 60.3% at 50 papers, versus 40.3% for the next-best system.

Introduction

We recently launched Elicit’s API and MCP, giving developers programmatic access to search on Elicit. While we’d previously compared Elicit’s search to academic search engines, this post compares our scientific search with general-purpose web search APIs configured to return scientific papers. BioASQ is an established benchmark for biomedical question answering and search. We describe BioASQ in more detail in our original post.

Evaluating web search APIs

We evaluated four web search engines: Exa, Brave, Parallel and Perplexity. Elicit searches academic sources only, while web search engines search the open web. BioASQ’s gold standards are academic papers, so an unrestricted web search engine on it would return non-academic sources that cannot count as hits, depressing their scores. To be fair to them, we restricted results to academic sources, using Exa’s publication filter and limiting the other APIs to PubMed and PubMed Central.

How we queried each system

We sent each BioASQ question to every API exactly as written and configured the APIs as below:

• Exa: we used the publication category,

• Brave: the two domains pubmed.ncbi.nlm.nih.gov and pmc.ncbi.nlm.nih.gov allow-listed

• Parallel: advanced mode, same two domains allow-listed

• Perplexity: the raw Search API, same two domains allow-listed.

We requested the maximum number of results that each API allows: 20 from Perplexity and Parallel, and 100 from Exa. Brave accepts up to 200, but results on its later pages tend to duplicate earlier results, so we requested 100 there as well.

Scoring the APIs

BioASQ questions were created in the past, and their gold standard papers reflect the papers available to researchers at that point of time. As in our original evaluation, we used a time filter to remove results that are after the cutoff. Of the four APIs, only Exa supports filtering by publication date. Footnote in framer: [start] Despite this, about .5% of Exa search results still had a date after the cutoff date, which we removed. Perplexity also accepts a date filter, but it applies to the date of the web page rather than the publication date of the paper, so we were unable to rely on it. [end footnote]

For the APIs where we couldn’t filter the date as part of the API request, we used PMIDs (directly, or resolved from their PMC identifiers) to get the publication year from PubMed. We then dropped results published after the question’s year.

Dropping results leaves gaps in a provider’s ranking, so we fill the gaps by moving lower-ranked results up without changing their relative order. Finally, we collapse results that resolve to the same paper, keeping the highest-ranked copy. See the table in the limitations section for how many papers each engine returns.

Results

Below is a table showing the recall of each search API at increasing cutoffs (10, 20, and 50 results). We compute the recall of every question and report the average of averages (macro-recall). Elicit’s search API has the highest recall of any search API we tested.

A dash means we do not report that system at that depth, because its results ran out before the cutoff on too many questions for the number to mean anything. We report a system at a cutoff only when at least 95% of questions returned that many results; the Limitations section gives the exact figures. We matched results to BioASQ’s gold standard using PMIDs where available, then DOIs, and finally titles, the same way we scored the academic search engines.

At every depth, the confidence interval on Elicit’s advantage over the next-best system excludes zero.

Limitations

Along with the limitations mentioned in our original post, we have limitations specific to web search engines. One limitation is that they do not have reliable filtering for the publication year, so we remove results published after the question’s year ourselves, and then fill the gaps that leaves by promoting later results without changing their order. Removing results shortens a provider’s ranked list, so at a deep cutoff a provider often has fewer papers available than the cutoff implies.

The table below shows the share of the 5,486 questions for which each system returned at least 10, 20, and 50 results after this filtering. We report a system’s recall at a cutoff only where that share reaches 95%.

Only Elicit and Exa clear that bar beyond 10 results.

Appendix: Paper matching

The table below shows the method by which each provider’s correct answers were matched to BioASQ’s gold standard papers. The three PubMed-restricted APIs return PubMed and PMC pages, whose URLs give a PMID directly, so nearly all of their matches come from PMIDs. Exa returns publisher pages instead, which is why more of its matches rest on DOIs and titles. Titles are matched exactly after normalization: we apply Unicode NFKC normalization, lowercase, strip punctuation, and collapse whitespace, then require the two strings to be identical. There is no fuzzy or substring matching, so a result whose title differs from the gold record by more than punctuation and case counts as a miss.

Appendix: Statistical significance

At each result depth, we compared Elicit with whichever web API had the highest recall at that depth. The confidence intervals use 50,000 paired bootstrap resamples of whole questions, and the interval level is widened to account for all comparisons we ran.