Overview
Covidence provides automatic extraction suggestions for fields where reliable suggestions are available. Suggestions are currently supported in both Extraction 1 and Extraction 2.
Reviewers must accept or reject each suggestion individually. If no suggestion is available for a field, it will appear as a normal input.
In Extraction 1, suggestions are available for the following fields:
Author's name (first author)
Address (first author)
Country (in which study was conducted)
Email (corresponding author)
Institution (first author)
Sponsorship source (funding sources)
In Extraction 2, suggestions are currently available for:
Study funding sources
Model information
The study characteristic extraction model uses a combination of OpenAlex and Large Language Models (LLM) to automatically extract a set of study characteristics.
The model has been evaluated on a curated sample of studies including a range of different study types:
Field | n | Precision | Recall |
Sponsorship source | 107 | 92.2% (95% CI: 88.6% – 95.7%) | 100.0% |
Country | 142 | 96.3% (95% CI: 93.1% – 99.3%) | 99.2% (95% CI: 97.6 – 100.0) |
Author’s name | 142 | 93.7% (95% CI: 89.4% – 97.2%) | 100.0% |
Institution | 142 | 94.3% (95% CI: 90.0% – 97.9%) | 99.3% (95% CI: 97.7% – 100.0%) |
142 | 99.3% (95% CI: 97.9% – 100.0%) | 100.0% | |
Address | 142 | 95.7% (95% CI: 92.2% – 98.6%) | 99.3% (95% CI: 97.8% – 100.0%) |
Known limitations to be aware of:
Access to the full-text PDF is required to retrieve field suggestions, either through an accessible and readable open access link or user-uploaded content.
Evaluations are based on a curated sample, limited to English-language papers, and may not cover all journal styles or layouts.
For more detailed information on the model design, evaluation methodology and performance, see the full technical documentation.
Enabling the feature
This feature is enabled by default.
In review settings, enable the feature by selecting "Provide suggestions during data extraction":
Reporting feature usage
For your Manuscript (Methods Section) use the following text to transparently report use of this feature in line with RAISE standards:
We will use the "study characteristic extraction suggestions" feature (no version number available; accessed on [date accessed]) developed by Covidence to suggest values for the following extraction study characteristic fields: [DE2: Study funding sources or DE1: Sponsorship source, Country, Author name, Institution, Email and Address].
The tool will be used according to the Covidence user guide with no customisation, training or parameter changes applied.
Outputs from the tool are justified for use in our synthesis because:
Humans make a decision on every suggestion: Reviewers must assess each suggested value and explicitly accept or reject it, defaulting to manual extraction when a suggestion is unavailable. This process maintains quality through human judgment remaining critical while extracting data.
Higher accuracy than typical human performance: The extraction suggestions and supporting quotes (applicable to sponsorship source suggestions only) were evaluated against relevant datasets, with all fields performing better than typical human extraction rates (80-85% precision). The suggested fields carry a minor consequence of error, given the limited impact from any suggestion mistakes.
Limitations of the tool include:
Feature limitations:
Suggestions require access to the full-text PDF, either through accessible open access or a user-uploaded PDF.
Older, locked and scanned PDFs may not be readable, resulting in limited performance for these studies.
Evaluation limitations:
The evaluations are based on a curated sample and may not cover every journal style. Performance could dip on unusual layouts or formats.
Extractions are evaluated on English-language papers only. Performance may differ for papers written in other languages.
Risk of automation bias: While all suggestions are still assessed by human reviewers, the presence of incorrect suggestions may influence their independent judgment in ways the tool cannot fully safeguard against.
A detailed description of the methodology, including parameters and validation procedures, is available in the Covidence support documentation and related supplementary materials.