When you've completed screening at Title & abstract or Full text review stage, you can export inter-rater reliability (IRR) data.

To do this, go to the Export page for your review; you'll see the section where you can choose to export a CSV file containing IRR data for Title & abstract or Full text review.

 

When you've selected your stage and clicked 'Prepare file', you'll shortly see an export file appear which you can download:

 

The export file includes not only the reviewer comparison data; you'll also get the calculations you need, including Cohen's Kappa coefficient:

 

About the calculation

  • Resolving conflicts does not affect the inter-rater reliability scores.
  • In inter-rater reliability calculations from the Title and abstract screening stage, 'maybe' votes are counted as 'yes' votes.
  • Please note that only the final set of votes cast at each round are included for inter-rater reliability calculations. If votes have been undone and re-done at either stage, only the votes cast during the re-voting process will be included.