I think he means the organizers in addition to the raters. So instead of hand-selecting samples like Scott & I do, the organizers would just feed random samples

I think he means the organizers in addition to the raters. So instead of hand-selecting samples like Scott & I do, the organizers would just feed random samples of unknown quality & origin straight to the raters without any chance to bias selection in favor of one side or another.