Statistical tools to improve assessing agreement between several observers
I. Ruddat,
B. Scholz,
S. Bergmann,
A.-L. Buehring,
S. Fischer,
A. Manton,
D. Prengel,
E. Rauch,
S. Steiner,
S. Wiedmann,
L. Kreienbrock,
A. Campe
Affiliations
I. Ruddat
Department of Biometry, Epidemiology and Information Processing, WHO Collaborating Centre for Research and Training in Veterinary Public Health, University of Veterinary Medicine, Hannover, Germany
B. Scholz
Friedrich-Loeffler-Institut, Institute of Animal Welfare and Animal Husbandry, Celle, Germany
S. Bergmann
Department of Veterinary Science, Faculty of Veterinary Medicine, Chair of Animal Welfare, Ethology, Animal Hygiene and Animal Housing, Ludwig-Maximilians-University, Munich, Germany
A.-L. Buehring
Friedrich-Loeffler-Institut, Institute of Animal Welfare and Animal Husbandry, Celle, Germany
S. Fischer
Institute for Animal Breeding and Genetics, University of Veterinary Medicine, Hannover, Germany
A. Manton
Department of Farm Animal Ethology and Poultry Science, University of Hohenheim, Stuttgart, Germany
D. Prengel
Department of Veterinary Science, Faculty of Veterinary Medicine, Chair of Animal Welfare, Ethology, Animal Hygiene and Animal Housing, Ludwig-Maximilians-University, Munich, Germany
E. Rauch
Department of Veterinary Science, Faculty of Veterinary Medicine, Chair of Animal Welfare, Ethology, Animal Hygiene and Animal Housing, Ludwig-Maximilians-University, Munich, Germany
S. Steiner
Department of Veterinary Science, Faculty of Veterinary Medicine, Chair of Animal Welfare, Ethology, Animal Hygiene and Animal Housing, Ludwig-Maximilians-University, Munich, Germany
S. Wiedmann
Bavarian State Research Center for Agriculture, Kitzingen, Germany
L. Kreienbrock
Department of Biometry, Epidemiology and Information Processing, WHO Collaborating Centre for Research and Training in Veterinary Public Health, University of Veterinary Medicine, Hannover, Germany
A. Campe
Department of Biometry, Epidemiology and Information Processing, WHO Collaborating Centre for Research and Training in Veterinary Public Health, University of Veterinary Medicine, Hannover, Germany
In the context of assessing the impact of management and environmental factors on animal health, behaviour or performance it has become increasingly important to conduct (epidemiological) studies in the field. Hence, the number of investigated farms per study is considerably high so that numerous observers are needed for investigation. In order to maintain the quality and validity of study results calibration meetings where observers are trained and the current level of agreement is assessed have to be conducted to minimise the observer effect. When study animals were rated independently by the same observers by a categorical variable the exclusion test can be performed to identify disagreeing observers. This statistical test compares for each variable and each observer the observer-specific agreement with the overall agreement among all observers based on kappa coefficients. It accounts for two major challenges, namely the absence of a gold-standard observer and different data type comprising ordinal, nominal and binary data. The presented methods are applied on a reliability study to assess the agreement among eight observers rating welfare parameters of laying hens. The degree to which the observers agreed depended on the investigated item (global weighted kappa coefficients: 0.37 to 0.94). The proposed method and graphical description served to assess the direction and degree to which an observer deviates from the others. It is suggested to further improve studies with numerous observers by conducting calibration meetings and accounting for observer bias.
