Two psychiatrists rate 50 case vignettes as "psychosis present" or "absent" and agree on 46 of them — 92% agreement. Is that good? Not necessarily. If 44 of the 50 cases were positive, two raters could hit 88% agreement by simply guessing "present" every time. That is the problem Cohen's kappa solves, and it is why knowing how to calculate and interpret Cohen's kappa separates candidates who memorise reliability from candidates who understand it.
This guide walks through a full worked example — observed agreement, chance agreement, the formula, the interpretation bands — plus the prevalence paradox that examiners love to test.
What kappa measures and why percent agreement fails
Reliability statistics answer one question: how much of this agreement would we expect if the raters were guessing independently? Percent agreement (Po) ignores chance. Cohen's kappa (κ) subtracts the chance-expected agreement (Pe) from what we observed, then rescales — so 0 means exactly chance-level agreement and 1 means perfect agreement:
κ = (Po − Pe) / (1 − Pe)
Kappa is the right tool when exactly two raters (or one rater at two occasions) score the same subjects on a nominal scale — for example, two clinicians rating symptom presence on the SCID, or the same rater rescoring six months later. For more than two raters, or ordinal ratings where near-misses should count partly, use Fleiss' kappa or weighted kappa respectively.
A worked example, step by step
Two raters classify 20 patients as cases or non-cases:
| Rater B: case | Rater B: non-case | Row total | |
|---|---|---|---|
| Rater A: case | 8 (a) | 4 (b) | 12 |
| Rater A: non-case | 2 (c) | 6 (d) | 8 |
| Column total | 10 | 10 | 20 |
- Observed agreement: Po = (a + d) / N = (8 + 6) / 20 = 0.70
- Chance-expected agreement: Pe = (rowA₁/N × colB₁/N) + (rowA₂/N × colB₂/N) = (12/20 × 10/20) + (8/20 × 10/20) = 0.30 + 0.20 = 0.50
- Kappa: κ = (0.70 − 0.50) / (1 − 0.50) = 0.20 / 0.50 = 0.40
The interpretation: what looked like "70% agreement" is a moderate κ of 0.40 — roughly half as impressive once chance is removed.
Interpreting the number: Landis and Koch
The conventional bands (Landis & Koch, 1977):
- < 0.00 — worse than chance (usually a coding or direction error)
- 0.00–0.20 — slight
- 0.21–0.40 — fair
- 0.41–0.60 — moderate
- 0.61–0.80 — substantial
- 0.81–1.00 — almost perfect
Treat these as conventions, not laws: in high-stakes diagnostics most journals and instruments want κ ≥ 0.70, and some fields expect ≥ 0.80. Always report the confidence interval alongside the point estimate — with N = 20, κ = 0.40 has a wide interval that may include values you'd find unacceptable.
The prevalence problem (kappa paradox)
When a condition is rare or near-universal, expected agreement Pe balloons and kappa collapses even when percent agreement is high. Two raters agreeing on 98 of 100 cases (all "absent") can produce a kappa near 0.10. This is the kappa paradox, and it matters in psychiatry: inter-rater reliability for low-prevalence findings (catatonia, clozapine-induced agranulocytosis) often looks terrible on kappa despite excellent raw agreement.
Practical responses: report percent agreement and Pe alongside kappa, consider prevalence-adjusted alternatives (PABAK), or use weighted kappa when adjacent categories should count partially.
Where this shows up in practice and exams
Reliability of structured interviews (SCID, CIDI), rating scales (PANSS, YMRS) and diagnostic agreement between clinicians are all standard research-methods material — in Preppsy's stats notes it sits alongside validity and reliability concepts and confidence intervals.
Exam trap
The setup: "Two raters agree on 90% of cases; prevalence of the condition is 95%. Kappa = 0.15. Explain." The examiner wants the paradox: high Po, high Pe (both raters saying "yes" most of the time), so little agreement remains above chance. Second trap: kappa vs ICC — kappa is for categorical ratings; intraclass correlation coefficients are for continuous/ordinal ratings like total scale scores. And remember κ = 0 = chance, not "no agreement".
Reporting kappa like a methodologist
A kappa coefficient on its own is half a result. Complete reporting includes: the contingency table (so readers can verify), Po, Pe, the kappa estimate with its 95% confidence interval, the prevalence and bias distribution, and a statement of what agreement threshold the study required a priori. Reviewers increasingly reject "κ = 0.73, p < 0.01" as insufficient — the p-value tests against zero, which was never the interesting hypothesis.
Interpreting confidence intervals changes conclusions routinely: with 30 subjects, κ = 0.65 might carry an interval from 0.31 to 0.99 — compatible with both inadequate and excellent reliability. When the interval straddles your decision threshold, the correct reading is "underpowered reliability study," not "acceptable reliability."
Choosing between coefficients is itself examinable. Cohen's κ fits two raters on nominal categories. Weighted κ extends it to ordered categories with disagreement weights. Fleiss' κ generalises to any number of raters on nominal data. The intraclass correlation coefficient (ICC) is the family for continuous ratings — mean ratings, total scores — and its two-way mixed versus two-way random distinction (consistency vs absolute agreement) decides which variant is correct. Reporting guidelines (GRRAS) specify exactly which of these belong in your methods section.
Finally, kappa measures agreement, not accuracy: two raters can agree perfectly and both be wrong. Validity — whether ratings match the underlying truth — is a separate axis, assessed against diagnostic gold standards rather than against each other.
Frequently asked questions
How do you calculate Cohen's kappa?
Compute observed agreement P_o = (both-rater agreements / total), compute chance agreement P_e from each rater's marginal totals (summing the products of their category proportions), then apply κ = (P_o − P_e) / (1 − P_e). In the worked example: P_o = 0.70, P_e = 0.50, so κ = 0.40.
What does a kappa of 0.4 mean?
By the Landis and Koch convention, 0.41–0.60 is moderate agreement; 0.40 sits at the fair/moderate boundary. In practice, kappa below 0.60 is usually considered too low for diagnostic instruments, where ≥ 0.70 is the common floor — and the confidence interval should always be reported with it.
Why can kappa be low when percent agreement is high?
When a category dominates (high or low prevalence), the chance-expected agreement P_e is already high, leaving little room above chance — the kappa paradox. Report percent agreement and P_e alongside kappa, or use prevalence-adjusted alternatives when this occurs.
When should you use weighted kappa instead?
When ratings are ordinal and disagreements differ in seriousness — for example severity bands or stage ratings where adjacent-category disagreement is less serious than extreme disagreement. Weighted kappa discounts near-misses using quadratic or linear weights, typically raising the coefficient.
Key takeaways
- Cohen's κ = (Po − Pe) / (1 − Pe) — agreement above chance, rescaled.
- Worked example: 70% observed, 50% expected → κ = 0.40 (fair/moderate).
- Landis & Koch bands: <0.20 slight, 0.41–0.60 moderate, 0.81–1.00 almost perfect; most journals want ≥ 0.70 for diagnostics.
- High agreement + low kappa = prevalence paradox; always report Po, Pe and the confidence interval.
- κ for two raters on nominal data; weighted κ for ordinal; Fleiss for >2 raters; ICC for continuous scores.
Statistics questions recur across every psychiatry paper — drill them in Preppsy's MCQ mock exams with full worked explanations.
Cover image: “Cohen's Kappa Explained (cover)” by Preppsy Editorial Team (Original artwork) via Preppsy.
