Agreement and Cohen's kappa
How the app measures how well screeners agree beyond chance, which pairs it reports, and why discussions and resolutions do not change it.
Cohen's kappa (κ) measures how much two screeners agree beyond what chance alone would produce. Journals expect it for dual screening.
The formula
κ = (pₒ − pₑ) / (1 − pₑ)
- pₒ is the share of records on which both screeners gave the same vote.
- pₑ is the agreement expected by chance, from each screener's own mix of include, exclude and maybe votes.
| κ | Reads as |
|---|---|
| 1 | Perfect agreement |
| 0 | No better than chance |
| Below 0 | Worse than chance |
A worked example
Two screeners judge 100 records. Both include 20, both exclude 70, and they disagree on 10. Screener A included 25 in total, screener B 25.
- pₒ = (20 + 70) / 100 = 0.90
- pₑ = (0.25 × 0.25) + (0.75 × 0.75) = 0.0625 + 0.5625 = 0.625
- κ = (0.90 − 0.625) / (1 − 0.625) = 0.275 / 0.375 ≈ 0.73
Next to κ in the app, How is this calculated? shows the same steps with your own numbers.
Which pairs are reported
κ is computed for every pair of screeners on the records both screened, from their independent votes:
- Humans: the average over all human pairs, weighted by how many records each pair screened in common. This is the inter-rater reliability the methods report.
- AI reviewers: the average over pairs of AI reviewers, from their votes before any discussion. When only AI reviewers screened, this is the figure the paper reports.
- AI and humans: pairs of one AI rater and one person. An AI "maybe" against a human "exclude" counts as a disagreement here, but it never changes the human κ.
What does not change it
A vote changed in a discussion still counts with its pre-discussion value, and resolving a conflict does not change κ. Agreement describes how independent judgements compare; what the team decided afterwards is reported separately.