Cohen's kappa

Cohen’s kappa compares observed agreement with a baseline built from how often each rater uses each label. It is useful alongside raw agreement, but the baseline needs interpretation.

Reference rows / predicted columns

Class	A	B
A	8	2
B	1	9

Intuition: frequent labels create easy agreements

If both sets of labels overwhelmingly use one class, high agreement can occur without good handling of the other classes. Kappa compares the observed agreement with what independent label draws would produce while preserving each side's class proportions.

Here, one label set can be a classifier and the other a reference. This does not make the reference infallible, or establish that the classifier is reliable in a new setting.

Compute the baseline

From a confusion matrix, let $r_k$ and $c_k$ be row and column totals and $N$ the total number of examples. Observed and expected agreements are

$$p_o=\frac{\sum_kC_{kk}}N,\qquad p_e=\sum_k\frac{r_k}{N}\frac{c_k}{N}.$$

Unweighted kappa is

$$\kappa=\frac{p_o-p_e}{1-p_e}.$$

The subtracted term is a model-based comparison, not an experimentally measured probability that someone was guessing. Kappa is one for perfect agreement when the denominator is defined, zero when observed agreement equals this expectation, and can be negative.

Work the example

The illustrative table gives $p_o=17/20=0.85$. Row proportions are 0.5 and 0.5, while column proportions are 0.45 and 0.55. Thus $p_e=0.5(0.45)+0.5(0.55)=0.5$ and $\kappa=(0.85-0.5)/(1-0.5)=0.7$.

If both raters assign every example to the same single class, then $p_o=p_e=1$. The expression is $0/0$, so kappa is undefined despite perfect raw agreement. The implementation used for lesson checks returns null in that case.

Use with care

Class prevalence and different marginal label frequencies affect the result. Universal labels such as “good” or “excellent” hide task-dependent consequences. Weighted kappa is a different statistic that assigns unequal costs to disagreements; it needs a stated weighting scheme and a defensible class ordering.

Where you will see this

Kappa can accompany a sleep-staging evaluation, together with the matrix, class support and split protocol. No project kappa is released on this site.

References

• Cohen (1960), A Coefficient of Agreement for Nominal Scales.
• scikit-learn: cohen_kappa_score.