4 ms·

I wonder how they address this problem, which affects all attempts to spot fake reviews: How do they know how accurate their results are?[0] What are the false

I wonder how they address this problem, which affects all attempts to spot fake reviews: How do they know how accurate their results are?[0] What are the false negative and false positive rates? To measure your accuracy objectively, you'd need an independent method to verify, with high reliability, the legitimacy of the reviews. (You may think you can spot them manually, and maybe you can spot some obvious ones, but false negatives are perilous: You don't know if the reviewer is simply better at hiding their trick than you are in discovering it - and they likely have far more expertise and experience than you do. Consider: Could you write a fake review that would trick someone like yourself?) Fakespot's FAQ says the following. I think the use of machine learning is a good example of the problem: The machine needs a good data set from which to learn, which means they'd have to flag reviews in the training data as legitimate or not - but they have no way to determine that. Basically, they are training the machine to make the same judgments, of unknown, accuracy, as the humans (in fairness, I'm making some presumptions about their use of machine learning). What criteria are used by Fakespot when analyzing reviews? Fakespot utilizes numerous technologies to validate the authenticity of reviews. The primary criteria is the language utilized by the reviewer, the profile of the reviewer,correlation with other reviewers data and machine learning algorithm that focuses on improving itself by detecting fraudulent reviews. The technologies include: profile clusters, sentiment analysis, cluster correlation and artificial intelligence intertwined with these functionalities. https://www.fakespot.com/faq https://www.fakespot.com/faq [0] As Richard Feynman said, "The first principle is that you must not fool yourself -- and you are the easiest person to fool."

Rapzid 9y ago
My knee-jerk reaction is that they are detecting anomalies and low-quality reviews for some definition of quality. As you pointed out, without a consistent ground truth target how could it really separate "fake" from "legit"? Perhaps they partnered with a fake review business and used their records?
netheril96 9y ago
I had the same reaction as yours. I clicked many links on the website, only to find such fluffy statements about machine learning algorithms, but no information whatsoever about the metrics and ground truth. They basically dodge the most important questions and therefore I conclude that they are not in any way trustworthy.
dfraser992 9y ago
Statistics.... No, absolute ground truth is impossible for this problem. There has been some research that estimated the percentage of fake / spam reviews at 2 - 6% but that was from a few years ago. I'm sure the percentage has increased greatly since then [great, another dissertation... My MSc dissertation was on this topic] Text-only based features are not fabulous, just useful (like 75% correct, using a dataset where ground truth is known). So there is / can be linguistic differences in the writing that indicate the writer is not 'truthful'. Sentiment analysis however, by itself, is abysmal - and the behavior of spammers has changed over the years so they are more knowledgeable how to craft reviews. But other signals, like relationships to other reviewers, IP addresses, submission time of review... they have been shown to be more accurate - but not in the 90+%. Establishing who is in a spamming group seems to be reliable though, so once that is known, you can more confidently label their reviews as spam [but not 100%, of course, to be fair and objective] I guess Fakespot takes a stab at estimating correctness and hopes the false negative rate is acceptably low. Yelp OTOH cranks things up so the false positive rate is high....
- forapurpose 9y ago
  Thanks; that kind of contribution is why I read HN. > using a dataset where ground truth is known How is this dataset created?

Rapzid 9y ago

My knee-jerk reaction is that they are detecting anomalies and low-quality reviews for some definition of quality. As you pointed out, without a consistent ground truth target how could it really separate "fake" from "legit"? Perhaps they partnered with a fake review business and used their records?

netheril96 9y ago

I had the same reaction as yours. I clicked many links on the website, only to find such fluffy statements about machine learning algorithms, but no information whatsoever about the metrics and ground truth. They basically dodge the most important questions and therefore I conclude that they are not in any way trustworthy.

dfraser992 9y ago

Statistics.... No, absolute ground truth is impossible for this problem. There has been some research that estimated the percentage of fake / spam reviews at 2 - 6% but that was from a few years ago. I'm sure the percentage has increased greatly since then [great, another dissertation... My MSc dissertation was on this topic] Text-only based features are not fabulous, just useful (like 75% correct, using a dataset where ground truth is known). So there is / can be linguistic differences in the writing that indicate the writer is not 'truthful'. Sentiment analysis however, by itself, is abysmal - and the behavior of spammers has changed over the years so they are more knowledgeable how to craft reviews. But other signals, like relationships to other reviewers, IP addresses, submission time of review... they have been shown to be more accurate - but not in the 90+%. Establishing who is in a spamming group seems to be reliable though, so once that is known, you can more confidently label their reviews as spam [but not 100%, of course, to be fair and objective] I guess Fakespot takes a stab at estimating correctness and hopes the false negative rate is acceptably low. Yelp OTOH cranks things up so the false positive rate is high....

forapurpose 9y ago
Thanks; that kind of contribution is why I read HN. > using a dataset where ground truth is known How is this dataset created?