When you've completed screening at Title & abstract or Full text review stage, you can export inter-rater reliability (IRR) data.
To do this, go to the Export page for your review; you'll see the section where you can choose to export a CSV file containing IRR data for Title & abstract or Full text review.
When you've selected your stage and clicked 'Export CSV', you'll shortly see an export file appear which you can download:
The export file includes not only the reviewer comparison data; you'll also get the calculations you need, including Cohen's Kappa coefficient:
About the calculation
- Resolving conflicts does not affect the inter-rater reliability scores.
- In inter-rater reliability calculations from the Title and abstract screening stage, 'maybe' votes are counted as 'yes' votes.
- Please note that only the final set of votes cast at each round are included for inter-rater reliability calculations. If votes have been undone and re-done at either stage, only the votes cast during the re-voting process will be included.