Overview
Covidence provides automatic extraction suggestions for fields where reliable suggestions are available. Suggestions are currently supported in both Extraction 1 and Extraction 2.
Reviewers must accept or reject each suggestion individually. If no suggestion is available for a field, it will appear as a normal input.
Suggestions are available for the following fields:
Author's name (first author)
Address (first author)
Country (in which study was conducted)
Email (corresponding author)
Institution (first author)
Sponsorship source (study funding sources)
Start date
End date
Setting
Inclusion criteria
Exclusion criteria
Year of publication
Aim of study
Method of recruitment
Conflicts of interest (for study authors)
Total sample size
Group differences (Extraction 1 only)
Number of withdrawals (Extraction 1 only)
Reason for withdrawals (Extraction 1 only)
Type of evidence source
Trial registration number
Method of data collection
Model information
The study characteristic extraction model uses Large Language Models (LLM) to automatically extract a set of study characteristics.
The model has been evaluated on a curated sample of studies including a range of different study types:
Field | Precision | Recall |
Sponsorship source | 92.2% (95% CI: 88.6% – 95.7%) | 100.0% |
Country | 96.3% (95% CI: 93.1% – 99.3%) | 99.2% (95% CI: 97.6 – 100.0) |
Author’s name | 93.7% (95% CI: 89.4% – 97.2%) | 100.0% |
Institution | 94.3% (95% CI: 90.0% – 97.9%) | 99.3% (95% CI: 97.7% – 100.0%) |
99.3% (95% CI: 97.9% – 100.0%) | 100.0% | |
Address | 95.7% (95% CI: 92.2% – 98.6%) | 99.3% (95% CI: 97.8% – 100.0%) |
Start date | 98.9% (95% CI: 96.2 - 100.0) | 100.0% |
End date | 97.7% (95% CI: 94.0 - 100.0) | 100.0% |
Setting | 98.0% (95% CI: 95.0 - 100.0) | 99.0% (95% CI: 96.2 - 100.0) |
Publication year | 99.0% (95% CI: 97.0 - 100.0) | 99.0% (95% CI: 96.9 - 100.0) |
Inclusion criteria | 99.0% (95% CI: 97.1 - 100.0) | 100.0% |
Exclusion criteria | 96.6% (95% CI: 92.2 - 100.0) | 98.8% (95% CI: 96.2 - 100.0) |
Aim of study | 98.0% (95% CI: 94.9 - 100.0) | 100.0% |
Method of recruitment | 98.9% (95% CI: 96.5 - 100.0) | 98.9% (95% CI: 96.7 - 100.0) |
Conflicts of interest | 98.9% (95% CI: 96.7 - 100.0) | 98.9% (95% CI: 96.7 - 100.0) |
Total sample size | 93.0% (95% CI: 87.0 - 97.0) | 100.0% |
Group differences | 87.8% (95% CI: 80.5 - 94.2) | 97.5% (95% CI: 93.9 - 100.0) |
Number of withdrawals | 88.5% (95% CI: 81.2 - 94.9) | 100.0% |
Reason for withdrawals | 90.4% (95% CI: 82.3 - 98.0) | 88.7% (95% CI: 79.2 - 96.2) |
Type of evidence source | 97.0% (95% CI: 93.0 - 100.0) | 100.0% |
Trial registration number | 100.0% | 100.0% |
Method of data collection | 91.7% (95% CI: 85.4 - 96.6) | 100.0% |
Known limitations to be aware of:
A full-text PDF is needed. Suggestions can only be generated when the study's full text is available. That means either a working open-access link or a PDF you've uploaded yourself.
Accuracy has only been tested on a curated set of papers. We measured how well suggestions perform using a hand-picked sample of English-language papers. Accuracy may differ for papers in other languages or with unusual journal styles or layouts.
Suggestions aren't generated for fields added later. If suggestions were already generated for a study, new fields added to the template afterwards won't get suggestions for that study. Suggestions for all fields are only available for studies:
newly included in the review, or
in a review where the extraction template is being published for the first time.
Suggestions may take a little while to appear. After you publish the template, AI suggestions can take some time to generate, so they may not show up straight away.
For more detailed information on the model design, evaluation methodology and performance, see the full technical documentation.
Enabling the feature
This feature is enabled by default.
In review settings, enable the feature by selecting "Provide suggestions during data extraction":
Reporting feature usage
For your Manuscript (Methods Section) use the following text to transparently report use of this feature in line with RAISE standards:
We will use the "study characteristic extraction suggestions" feature (no version number available; accessed on [date accessed]) developed by Covidence to suggest values for the following extraction study characteristic fields: [DE2: Author name, Institution, Email, Address, Start date, End date, Study funding sources, Country, Study setting, Year of publication, Inclusion criteria, Exclusion criteria, and Total sample size; or DE1: Sponsorship source, Country, Author name, Institution, Email, Address, Setting, Start date, End date, Inclusion criteria, Exclusion criteria, and Year of publication].
The tool will be used according to the Covidence user guide with no customisation, training or parameter changes applied.
Outputs from the tool are justified for use in our synthesis because:
Humans make a decision on every suggestion: Reviewers must assess each suggested value and explicitly accept or reject it, defaulting to manual extraction when a suggestion is unavailable. This process maintains quality through human judgment remaining critical while extracting data.
Higher accuracy than typical human performance: The extraction suggestions and supporting quotes were evaluated against relevant datasets, with all fields performing better than typical human extraction rates (80-85% precision). The suggested fields carry a minor consequence of error, given the limited impact from any suggestion mistakes.
Limitations of the tool include:
Feature limitations:
Suggestions require access to the full-text PDF, either through accessible open access or a user-uploaded PDF.
Older, locked and scanned PDFs may not be readable, resulting in limited performance for these studies.
Evaluation limitations:
The evaluations are based on a curated sample and may not cover every journal style. Performance could dip on unusual layouts or formats.
Extractions are evaluated on English-language papers only. Performance may differ for papers written in other languages.
Risk of automation bias: While all suggestions are still assessed by human reviewers, the presence of incorrect suggestions may influence their independent judgment in ways the tool cannot fully safeguard against.
A detailed description of the methodology, including parameters and validation procedures, is available in the Covidence support documentation and related supplementary materials.