Overview
Covidence provides automatic extraction in Extraction 1 (DE1) for fields where reliable suggestions are available.
Extraction suggestions will show for all users in DE1, requiring them to either accept or reject each suggestion individually. If we aren’t able to find a suggestion for the field, it will appear as a normal input (no suggestion).
Extraction suggestions are available for the following data points:
Author’s name (first author)
Address (first author)
Country
Email
Institution (first author)
Sponsorship source (funding sources)
Model information
The study characteristic extraction model uses a combination of OpenAlex and Large Language Models (LLM) to automatically extract a set of study characteristics.
Evaluated on 136,081 data points, the model achieved a mean user acceptance rate of 97.44% (95% CI: 96.37% – 98.50%) across Author's name, Institution, Country, Email, and Address fields. Sponsorship source was evaluated separately on a curated sample of 107 studies, achieving a precision of 92.2% (95% CI: 88.6% – 95.7%) and recall of 100.0% for extracted values.
Known limitations to be aware of:
Only available for studies with a DOI linked in Covidence.
Access to the full-text PDF is required to retrieve the Address, Country, Email, and Sponsorship Source fields, either through an accessible and readable open access link or user-uploaded content.
Author's name and Institution suggestion coverage depends on data availability in OpenAlex.
Suggestion coverage varies by field: approximately 85% of studies for Author's name, Institution and Sponsorship source; and 35% for Address, Country and Email.
Acceptance rates may be influenced by automation bias. Sponsorship source evaluations are based on a curated sample, limited to English-language papers, and may not cover all journal styles or layouts.
For more detailed information on the model design, evaluation methodology and performance, see the full technical documentation.
Enabling the feature
This feature is enabled by default.
In review settings, enable the feature by selecting "Provide suggestions during data extraction (available for Extraction 1 only)"
Reporting feature usage
For your Manuscript (Methods Section) use the following text to transparently report use of this feature in line with RAISE standards:
We will use the “study characteristic extraction suggestions” feature (no version number available; accessed on [date accessed]) developed by Covidence to suggest values for the following extraction study characteristic fields: Sponsorship source, Country, Author name, Institution, Email and Address.
Supporting quotes are also provided for sponsorship source suggestions.
The tool will be used according to the Covidence user guide with no customisation, training or parameter changes applied.
Outputs from the tool are justified for use in our synthesis because:
Humans make a decision on every suggestion: Reviewers must assess each suggested value and explicitly accept or reject it, defaulting to manual extraction when a suggestion is unavailable. This process maintains quality through human judgment remaining critical while extracting data.
Higher accuracy than typical human performance: The extraction suggestions and supporting quotes (applicable to sponsorship source suggestions only) were evaluated against relevant datasets, with all fields performing better than typical human extraction rates (80-85% precision). The suggested fields carry a minor consequence of error, given the limited impact from any suggestion mistakes.
Limitations of the tool include:
Feature limitations:
The feature only provides suggestions for studies with a DOI attached in Covidence.
Suggestions for the author's name and institution are sourced from OpenAlex, requiring relevant data to be available through their database.
Suggestions for country, email, address and sponsorship source require access to the full-text PDF, either through accessible open access or a user-uploaded PDF.
Evaluation limitations:
The evaluation for author's name, institution, country, email and address are based on interaction with suggestions, which may have influenced human judgments.
The evaluation for sponsorship source extractions is based on a curated sample and may not cover every journal style. Performance could dip on unusual layouts or formats.
Sponsorship source extractions are evaluated on English-language papers only. Performance may differ for papers written in other languages.
Risk of automation bias: While all suggestions are still assessed by human reviewers, the presence of incorrect suggestions may influence their independent judgment in ways the tool cannot fully safeguard against.
A detailed description of the methodology, including parameters and validation procedures, is available in the Covidence support documentation and related supplementary materials.