At our university we do the clarification ourselves, usually with 3-5 classifiers per item. And it is surprising the high rate of disconformity between labelers

At our university we do the clarification ourselves, usually with 3-5 classifiers per item. And it is surprising the high rate of disconformity between labelers (even in binary or 4-class classification). People in internet don't always understand sarcasm, so maybe we need to benchmark humans at this task. But yeah, this and similar datasets (in Spanish, for example) have tons of misclassified texts.