Overview
Covidence uses advanced large language models to accurately identify and extract intervention groups (arms) from a study's full text. The 'Link to suggestions' field enables you to connect these automatically extracted intervention groups (arms) with the template-defined interventions in the extraction form.
Select the suggestion that is most relevant to the intervention you have selected from your template e.g. For the template intervention “Intervention” select the suggestion “Vitamin C”.
If there is no relevant suggestion, select 'No relevant suggestion'.
What will happen if I don’t use it?
Using the “Link to suggestions” field is optional. By using the “Link to suggestions” field, you are helping Covidence improve its automation capabilities. Teams that use the “Link to suggestions” field will also be able to leverage any future automation offerings released by Covidence, including characteristics and result data suggestions.
Why would I need to create multiple groups linked to template-defined interventions?
Many studies have one intervention group and one comparison group. However, it can get more complex. As an example, imagine a study that reports different variations of an intervention. In that case you will have multiple intervention groups (arms) which are given different variations of the intervention e.g. different doses, different locations.
In this case, you will want to extract all the relevant intervention groups (arms) and this is why it’s useful to be able to add multiple groups (arms) linked to your template-defined interventions and add a group name for accurate reporting
What is the group name field?
The “group name” field helps distinguish between multiple groups (arms) receiving the same intervention category e.g. Vitamin C 250 mg vs. Vitamin C 500 mg
It also helps you extract the group name as it’s reported in the study whilst mapping it to your template defined interventions.
Model information
This model automatically extracts intervention groups reported in a study using a large language model (LLM). We evaluated the model output against a benchmark dataset of 56 open-access studies. The model achieved a precision of 98.15% and a recall of 95.16%, outperforming typical human performance.
Known limitations to be aware of:
The model is only available for studies with a DOI linked in Covidence.
Full-text PDF access is required (via an open-access link or user-uploaded PDF).
Suggestion coverage is approximately 30% of studies, dependent on DOI availability and full-text PDF access.
Evaluations were conducted on a small sample of open-access studies in the medical and health sciences. Performance may differ across other disciplines or study types.
For more detail on the model design, evaluation methodology and performance see the full technical documentation.
Reporting feature usage
For your Manuscript (Methods Section) use the following text to transparently report use of this feature in line with RAISE standards:
We will use the “intervention group suggestions” feature (no version number available; accessed on [date accessed]) developed by Covidence to suggest intervention groups reported in a study’s full-text.
The tool will be used according to the Covidence user guide with no customisation, training or parameter changes applied.
Outputs from the tool are justified for use in our synthesis because:
Humans make a decision on every suggestion: Reviewers must assess each suggested value and make the selection themselves, defaulting to manual processes when a suggestion is unavailable. This process maintains quality through human judgment remaining critical while extracting data.
Higher accuracy than typical human performance: The extraction suggestions were evaluated against 59 open access studies, with all fields performing significantly better than typical human extraction accuracy (55-60%). The suggested fields carry a major consequence of error, given the potential impact from any suggestion mistakes on the research outcome.
Limitations of the tool include:
Feature limitations:
The feature only provides suggestions for studies with a DOI attached in Covidence.
The feature requires access to the full-text PDF, either through accessible open access or a user-uploaded PDF.
Evaluation limitations: Limited studies were used to evaluate performance, and were exclusively in the medical and health science domain.
Risk of automation bias: While all suggestions are still assessed by human reviewers, the presence of incorrect suggestions may influence their independent judgment in ways the tool cannot fully safeguard against.
A detailed description of the methodology, including parameters and validation procedures, is available in the Covidence support documentation and related supplementary materials.