2 ms·
It seems like writing a naive Bayesean classifier for "Was this written by a Turker" should be like taking candy from a baby who hates candy, has very slippery
by patio11 15y ago
It seems like writing a naive Bayesean classifier for "Was this written by a Turker" should be like taking candy from a baby who hates candy, has very slippery fingers, and is unconscious.
- crdoconnor 15y agoIf it was that easy, why did their human judges fail at it?
- socratic 15y agoDoes the experimental setup for the human judges sound fair to you? For example, the naive Bayes classifier knows the a priori distribution of review spam (which appears to be held to 50%), but do the undergraduate human judges? It would appear not, given that one judge only labeled 12% deceptive. Likewise, were the human judges able to see examples of truthful and deceptive reviews before beginning the task? (In other words, are the human judges solving a different problem, e.g., "deception detection", than the classifier e.g., "similarity to prior deceptive reviews from Turkers"). If these are differences between the human and computer annotator setups, are they major differences? Can you spot any other big differences between the two experimental setups?
- joe_the_user 15y agoTill the person who hires the Turkers includes said filter along with the work order. "Write a good that passes this, this and this filter by a fair margin" You might try seeing who's hiring Turkers for what. It might give you an idea how much filtering is needed.