How VDI/VDE Uses Elicit to Scale Systematic Reviews 11x Faster
Challenge
Manual screening limited reviews to 50-100 papers, forcing outsourcing and sacrificing quality control.
Challenge
Manual screening limited reviews to 50-100 papers, forcing outsourcing and sacrificing quality control.
Solution
Integrated Elicit after validating 99.4% extraction accuracy with initial testing.
Results
VDI/VDE now evaluates 11x more papers per systematic literature review, expanding their capacity from 50 to 550 papers.
About
VDI/VDE Innovation + Technik GmbH is a German consulting firm that advises governments and the private sector on innovation and technology. In 2023, the company employed more than 1,000 subject-matter experts and managed ~3 billion euros for clients.
VDI/VDE-IT uses Elicit to evaluate 11x more papers per project, enabling them to offer clients more ambitious and rigorous systematic reviews.
About
VDI/VDE Innovation + Technik GmbH is a German consulting firm that advises governments and the private sector on innovation and technology. In 2023, the company employed more than 1,000 subject-matter experts and managed ~3 billion euros for clients.
VDI/VDE-IT uses Elicit to evaluate 11x more papers per project, enabling them to offer clients more ambitious and rigorous systematic reviews.
Before Elicit: Constrained by resources
VDI/VDE-IT had limited capacity to conduct and market in-house systematic reviews.
Veit Wasserfuhr, Consultant, said, “In a normal industry systematic literature review, it’s typically impossible to closely analyze more than 50-100 papers manually. It’s painstaking and inefficient. There’s not enough time to go deep on each paper. You have to scan papers quickly and report with limited information.”
They dealt with these constraints in part by outsourcing systematic reviews, but this made it difficult for them to control quality. Wasserfuhr adds, “When we outsource systematic reviews, we pay the company and get a report back. We don’t get to be involved in drafting it.”
Before Elicit: Constrained by resources
VDI/VDE-IT had limited capacity to conduct and market in-house systematic reviews.
Veit Wasserfuhr, Consultant, said, “In a normal industry systematic literature review, it’s typically impossible to closely analyze more than 50-100 papers manually. It’s painstaking and inefficient. There’s not enough time to go deep on each paper. You have to scan papers quickly and report with limited information.”
They dealt with these constraints in part by outsourcing systematic reviews, but this made it difficult for them to control quality. Wasserfuhr adds, “When we outsource systematic reviews, we pay the company and get a report back. We don’t get to be involved in drafting it.”
Choosing Elicit: A formal evaluation
Wasserfuhr took the initiative to try and solve these longstanding issues with a tool he trusted: “I started using Elicit 4 years ago before the ChatGPT hype and was already really impressed. I followed the newsletters and saw continuous improvement on the fundamentals.”
To validate this first impression, VDI/VDE-IT rigorously compared Elicit’s extraction to their own manual extraction of 50 papers. The results were, in Wasserfuhr’s words, "very impressive": Elicit was 99.4% accurate. “In some extractions, it provided more comprehensive answers than manual reviewers.”
Choosing Elicit: A formal evaluation
Wasserfuhr took the initiative to try and solve these longstanding issues with a tool he trusted: “I started using Elicit 4 years ago before the ChatGPT hype and was already really impressed. I followed the newsletters and saw continuous improvement on the fundamentals.”
To validate this first impression, VDI/VDE-IT rigorously compared Elicit’s extraction to their own manual extraction of 50 papers. The results were, in Wasserfuhr’s words, "very impressive": Elicit was 99.4% accurate. “In some extractions, it provided more comprehensive answers than manual reviewers.”
Today: Using Elicit to evaluate 11x more papers per systematic literature review
The firm recently used Elicit to expand the scope of a systematic review on educational interventions from 50 papers to 550 papers. Commenting on this, Wasserfuhr said, “With Elicit, we can produce a very high quality review for clients in a couple of months. That’s a valuable selling point.”
Dr. Ernst Andreas Hartmann, Head of VDI/VDE-IT’s Education consulting division, added “What I really liked about Elicit is that you can check the references behind a result within the context of the original paper. Elicit’s claim verification is really, really good and gives me confidence in using it.”
Today: Using Elicit to evaluate 11x more papers per systematic literature review
The firm recently used Elicit to expand the scope of a systematic review on educational interventions from 50 papers to 550 papers. Commenting on this, Wasserfuhr said, “With Elicit, we can produce a very high quality review for clients in a couple of months. That’s a valuable selling point.”
Dr. Ernst Andreas Hartmann, Head of VDI/VDE-IT’s Education consulting division, added “What I really liked about Elicit is that you can check the references behind a result within the context of the original paper. Elicit’s claim verification is really, really good and gives me confidence in using it.”
Learn more about these features
Learn more about
these features
Tomorrow: Scaling Elicit across the company
In March, Hartmann and Wasserfuhr plan to showcase Elicit to other departments.
“There is quite a bit of interest.” Wasserfuhr said “Our researchers want to use Elicit not only for systematic literature reviews, but also for scoping reviews, rapid reviews, and trend analysis.” He detailed a project where VDI/VDE-IT plans to analyze trends in the abstracts of more than 50,000 papers.
When discussing his experiences with competitor tools, Wasserfuhr summed it up with: “For data extraction in rigorous systematic literature reviews, I trust Elicit.”
Tomorrow: Scaling Elicit across the company
In March, Hartmann and Wasserfuhr plan to showcase Elicit to other departments.
“There is quite a bit of interest.” Wasserfuhr said “Our researchers want to use Elicit not only for systematic literature reviews, but also for scoping reviews, rapid reviews, and trend analysis.” He detailed a project where VDI/VDE-IT plans to analyze trends in the abstracts of more than 50,000 papers.
When discussing his experiences with competitor tools, Wasserfuhr summed it up with: “For data extraction in rigorous systematic literature reviews, I trust Elicit.”
If you’re interested in Elicit for enterprise or institutional use, reach out to us at [email protected] for more information regarding our security, or to request a copy of our SOC 2 Type II report under NDA.
If you’re interested in Elicit for enterprise or institutional use, contact us here.
About
VDI/VDE Innovation + Technik GmbH is a consulting firm that advises governments and the private sector on innovation and technology.
Industry
Policy
Company size
200–800
Founded
1978
Headquarted
Berlin, Germany
