One point that confuses me a bit: The study initially used experts (no word on the experts?) and later only selects four experts for the subsample because thei

One point that confuses me a bit: The study initially used experts (no word on the experts?) and later only selects four experts for the subsample because their kappa is moderate. Is that a common approach? If there are two raters in a sample of six that disagree with the four others, I would argue that the IRR is an issue. Otherwise one could always run studies with n experts and only select a subsample of those raters that have a high interrater agreement, or not? The Fleiss kappas are already relatively low..