Overview
If you’re completing a review that looks only at the effects of health interventions through Covidence and want to only consider research papers reporting on Randomised Controlled Trials (RCTs), consider using this feature.
All eligible references imported to the review that don’t have any manual screening votes applied will be run through the Cochrane RCT classifier to identify and remove the references not reporting on RCTs.
The references automatically removed can be reviewed via the “Auto-marked as ineligible” removed before screening tab, under the import references section:
By clicking here, you can also review the list of auto-marked ineligible references and, if you believe any should be included, simply click “Move to screening” to return them to the screening list. This gives you full control over what’s reviewed.
The Cochrane RCT Classifier
We’ve integrated with the leading RCT classifier algorithm, developed by EPPI-Centre, to provide you with a prediction on whether studies in your review potentially report on an RCT. The Classifier has been endorsed by Cochrane and is also being used in the Cochrane Screen4Me workflows, which also includes Cochrane Crowd.
Evaluation
The classifier has been shown in testing to successfully identify over 99.5% of health-related references that potentially report on RCTs, only incorrectly classifying ~0.5% of RCTs as not being an RCT (Thomas et al, 2021).
Known limitations
Given the RCT classifier has been trained, calibrated and validated using health-related research papers, we only allow the features to be used on Cochrane reviews or reviews in the “medical and health science” research area.
To ensure the classifier has enough context about each reference to make informed predictions, we'll only input references with titles of 14 characters or more and abstracts of 400 characters or more. This mirrors the criteria used for validating the classifier's performance.
Enabling the feature
When creating a review, you will be shown RCT-related settings under the “Automation options” section when the review’s research area is set to “Medical and health sciences” or it is a Cochrane review:
Disabling the feature
You can disable the feature at any point via the automation options section on review settings (shown above).
Disabling the feature will cause all references automatically removed by the classifier to be moved back to the title and abstract screening step for manual screening. You will be able to view the actions performed on each reference in their history:
Reporting feature usage
For your Manuscript (Methods Section) use the following text to transparently report use of this feature in line with RAISE standards:
We will use the “Remove references reporting on non-RCTs before screening” feature (version number not supplied, accessed on dd/mm/yyyy) developed by Covidence for removing references that are not likely to describe randomized controlled trials before the Title & Abstract screening stage. The tool will be used with no customisation, training or parameter changes applied.
Outputs from the tool are justified for use in our synthesis because:
High-sensitivity model performance: The classifier, developed by the EPPI-Centre and endorsed by Cochrane, identifies more than 99.5% of health-related references that potentially report RCTs and misclassifies only ~0.5% of true RCTs as non-RCTs (Thomas et al., 2021), prioritising high sensitivity to minimise false exclusions.
Human reviewers can oversee the references removed: All references removed by this feature are reviewable by human reviewers at any stage - via both the PRISMA flow diagram and a separate listing page - ensuring human judgment can always override any automatically removed reference before screening is finalised.
Mitigation for short or uninformative abstracts: A known limitation of the classifier is reduced performance on records with very short or poorly reported titles/abstracts. Covidence mitigates this risk by only attempting to remove records that meet minimum information thresholds (title >14 characters and abstract >400 characters), consistent with the filters used in the original evaluation. Records below these thresholds left to be screened entirely by human reviewers, ensuring that low-information records receive full manual assessment.
Limitations of the tool include:
Evaluation limitations: The evaluation by Thomas et al. (2021) had limitations that users of this feature should be aware of. The classifier was trained and evaluated primarily on English-language, biomedical records (Embase, Cochrane, Clinical Hedges), so performance in other languages, non-health domains, or niche subfields is less certain. Its best performance (99.5% recall) is observed only for records with sufficiently long and informative titles/abstracts; older studies and very short or poorly reported abstracts are more likely to be misclassified and require manual handling. Because the classifier analyses only the title and abstract, and does not use full text or additional metadata, any signals present only in the full text will not be detected.
Risk of automation bias: While all removed references can be reviewed by humans, the feature cannot fully eliminate the possibility that reviewers may rely too heavily on automated decisions.