To deal with the problem of indexing diagnostic agreement when clinicians make multiple diagnoses on the same case, an algorithm for calculating a specific level of disagreement for an individual case was developed, and a summary statistic for measuring overall agreement is defined. The output of a computer program for quantifying agreement on multiple diagnoses is described, and an application is given.
Fleiss JL, Spitzer RL, Endicott J, Cohen J. Quantification of Agreement in Multiple Psychiatric Diagnosis. Arch Gen Psychiatry. 1972;26(2):168–171. doi:10.1001/archpsyc.1972.01750200072015