Using simulation and data from a real-world trial we show that this effect is not trivial and complicates the interpretation of κ changes in the interpretation of reader performance, which, admittedly can change because of a multitude of factors.

Inter-rater kappa can change during trial even when raters do not