6.5.10. Kappa Test for Inter-Observer Variation
This version will calculate a test statistic to measure the degree of agreement between two raters. The present implementation is the original form of Kappa test as introduced by Cohen, J. A. (1960) and Cohen, J. A. (1968), and also takes into account the correction in the calculation of large sample variances by Fleiss, J. L., Cohen, J., Everitt, B. S., (1969). A weighted Kappa is computed if weights are given and an unweighted Kappa otherwise.
The data for this test must be in the form of a square matrix. Columns of the matrix can be selected from the Variables Available list by clicking on [Variable]. If the number of rows is not equal to the number of columns the test cannot be performed. Missing values are not allowed.
If a weighted analysis is to be performed weights should also be arranged in the form of a square matrix with the same dimensions. Columns of the weights matrix can be selected from the Variables Available list by clicking on [Weight].
Example
The data is taken from Fleiss, J. L., Cohen, J., Everitt, B. S., (1969).
Open NONPARM2, select Statistics 1 → Nonparametric Tests (Multisample) → Kappa Test Inter-Observer Variation and select Diagnosis 1, Diagnosis 2 and Diagnosis 3 (C31 to C33) as [Variable]s to obtain the following results:
Kappa Test (Inter-Observer Variation)
Variables Selected: Diagnosis1, Diagnosis2, Diagnosis3
|
Expected Proportion = |
0.7000 |
|
Observed Proportion = |
0.4750 |
|
|
Value |
Standard Error |
Z-Statistic |
1-Tail Probability |
|
Kappa |
0.4286 |
0.0537 |
7.9792 |
0.0000 |
|
|
2-Tail Probability |
Lower 95% |
Upper 95% |
|
Kappa |
0.0000 |
0.3233 |
0.5338 |
Keeping the Diagnosis columns as [Variable]s, select Weight 1, Weight 2 and Weight 3 (C34 to C36) as [Weight]s:
Kappa Test (Inter-Observer Variation)
Variables Selected: Diagnosis1, Diagnosis2, Diagnosis3
Weights: Weight1, Weight2, Weight3
|
Expected Proportion = |
0.5672 |
|
Observed Proportion = |
0.7867 |
|
|
Value |
Standard Error |
Z-Statistic |
1-Tail Probability |
|
Weighted Kappa |
0.5071 |
0.0570 |
8.8967 |
0.0000 |
|
|
2-Tail Probability |
Lower 95% |
Upper 95% |
|
Weighted Kappa |
0.0000 |
0.3954 |
0.6188 |