18 ms·
Is That Review a Fake?
- tldrplz0 15y agonice :)
- JonnieCache 15y agoDirect link to the paper: http://aclweb.org/anthology/P/P11/P11-1032.pdf http://aclweb.org/anthology/P/P11/P11-1032.pdf
- tansey 15y agoInteresting. To summarize their methodology: - Take the top 20 hotels from trip advisor - Filter out all non-five star reviews, plus any non-english, excessively short, or first-time reviews. Then sample (via a log normal distribution on review length) 20 reviews per hotel. This is their "real" dataset. - Use Mechanical Turk to collect 400 reviews for these hotels. Turkers are instructed to pretend they work for the marketing dept of the hotel and are to write deceptively fake reviews. Again, quality filters on length, user approval rating, and deduplication are applied. Turkers are paid $1 per review. This creates their "fake" dataset. I suppose one could still argue that there are selection bias issues here. The sample size is also moderate. Nevertheless, it's a novel approach and you have to start somewhere. Interesting work.
- patio11 15y agoIt seems like writing a naive Bayesean classifier for "Was this written by a Turker" should be like taking candy from a baby who hates candy, has very slippery fingers, and is unconscious.
- crdoconnor 15y agoIf it was that easy, why did their human judges fail at it?
- socratic 15y agoDoes the experimental setup for the human judges sound fair to you? For example, the naive Bayes classifier knows the a priori distribution of review spam (which appears to be held to 50%), but do the undergraduate human judges? It would appear not, given that one judge only labeled 12% deceptive. Likewise, were the human judges able to see examples of truthful and deceptive reviews before beginning the task? (In other words, are the human judges solving a different problem, e.g., "deception detection", than the classifier e.g., "similarity to prior deceptive reviews from Turkers"). If these are differences between the human and computer annotator setups, are they major differences? Can you spot any other big differences between the two experimental setups?
- joe_the_user 15y agoTill the person who hires the Turkers includes said filter along with the work order. "Write a good that passes this, this and this filter by a fair margin" You might try seeing who's hiring Turkers for what. It might give you an idea how much filtering is needed.
- kubrickslair 15y agoYeah, there are some selection bias issues here. From what I have seen, anomaly detection especially graph based ones can be pretty robust, but they break when the person has written only one or so reviews- which is what I guess the majority of reviews are from.
- jpr 15y agoWhat's the point of filtering out all non-five star reviews? I for one would be very interested in fake reviews by my competitors.
- chalst 15y agoThey mention negative deceptive review detection as further work.
- coldarchon 15y agoI wrote a program to detect this. Without markov chains to analyze the structure of sentences this is pretty useless. If the structure is too clean with too many common expressions the review is fake ..
- nazgulnarsil 15y agoFake reputation is an interesting problem space for libertarians to grapple with since reputation is their solution to so many problems. One of my ideas in this space is a bitcoin like system that maintains a public ledger of contracts you've entered into and completed as verified by a variety of sources. People could supply their public key allowing you to check their reputation history and sign contracts with their private key.
- spottiness 15y agoWe are experimenting with reputation derived from complete anonymity since we believe that anonymity is required to maximize sincerity. In our current stage, we're manually moderating all the content generated, rejecting what looks fake based on common sense and intuition. If we ever generate traffic above a certain threshold then we have a mechanism where the content is validated by the creators themselves. Basically, the writers of opinions ("spots" for us) will have to evaluate other opinions before they can post their own, and a decision is made based on the number of coincidences among all the reviewers of a specific posting. We got the idea from previous work by CMU's Luis Von Ahn, in particular his (now Google's) ESP game. These are examples of anonymous reviews about hotels that we have in our site: A positive one => http://www.spottiness.com/spots/BHBZ8QJT http://www.spottiness.com/spots/BHBZ8QJT A negative one => http://www.spottiness.com/spots/RKTPXLJJ http://www.spottiness.com/spots/RKTPXLJJ Interestingly, they don't have the strong deceptive indicators...
- watmough 15y ago"We are experimenting with reputation derived from complete anonymity since we believe that anonymity is required to maximize sincerity. " Interesting. Where do "real name" Amazon.com reviews fit in? These make a selling point of being attributable to real people, and to me, imply sincerity, since often these reviewers seem to write reviews almost as a hobby, and often make a point of covering both good and bad aspects of a product. There's a interesting dynamic here, since Amazon is vanishingly unlikely to harass you on the web, unlike say an ebay seller, who might well come after you if you leave anything other than a perfect review. In this case you are likely to be anonymous as far as everyone but the seller is concerned, yet being sincere may carry some risk to your own ebay account.
- donnyg107 15y agoAs if the bots wont adjust their syntax when the software becomes more comprehensive. Silly Cornell, you're up against spammers, not actual robots.
- Sapient 15y agoLying well is actually really difficult and doesn't come easy to many people. The differenced between a review written from experience and a review written from someone's imagination may be slight, but I think there will always be markers.
- Vadoff 15y agoIf you read the article, it's a system to detect fake reviews in general - there's no mention of bots or spammers at all. By looking at their methodology it's clear that it targets people who write fake reviews, not bots. Also, it's not as if every spammer in the world is aware of this software and will adjust to it accordingly. In fact, I'd say the vast majority wouldn't even be aware.
- deleted 15y ago[deleted]
- donnyg107 15y agoSo after any sort of sensible application of this work is employed, these people would continue to write those same fake reviews and wonder "I wonder why my reviews are being taken down and my ratings not counted!" And if it is in fact spammers spending their time designing software which writes fake reviews, not business owners trying to hype up their own companies, the spammers' wellbeing is dependent upon how well their reviews pass as real ones. If some kind of software were developed and employed by the company whose pages will host the reviews, I think the spammers would be hard pressed not to make sure their software works and discover some kind of issue quickly. Then again, this study could have been meant not to be applied practically, but intended as an academic analysis of fake reviews, posted by random individuals who love to dishonestly talk up items or companies with which they are not well acquainted. But then this is no more than a psychological analysis of a minor issue, which is interesting, but not the article I thought it was.
- bmac27 15y agoAs these kinds of issues continue to bring into question the reputation of (mostly) anonymous review sites, the reliance on reviews from your social graph (I.E. folks you presumably know & trust) could continue to rise, particularly if the search engines remain unable to differentiate between the spammers and legit reviewers. No wonder Google proactively sought out that kid who worked on the paper.
- skimbrel 15y agoYelp has had this for a while: when you visit a business page, the first reviews they show you are ones from your friends, if available. It's a good way to make sure you're not seeing spam reviews right off the bat, but it does rely on you actually engaging with Yelp's social features.
- radog 15y agoShrewd (though likely just very lucky and timely) choice of topic. Looks like they can either (i) take their talents to any one of 10-15 companies that desperately need improvement in this area OR (ii) create a very promising startup and perhaps get taken out in a talent acquisition.
- chaz 15y agoThe related parent article is worth a read as well: "In a Race to Out-Rave, 5-Star Web Reviews Go for $5." (http://www.nytimes.com/2011/08/20/technology/finding-fake-reviews-online.html http://www.nytimes.com/2011/08/20/technology/finding-fake-re...)
- cpeterso 15y agoI usually find the low-star reviews more informative. What can the 5-star reviews do other than affirm the quality claims of the reviewed book/movie/hotel/restaurant? I read 1-star reviews for a laugh and 2- and 3-star reviews to be informed by reviewers who care enough to write a review but whose opinions are not extreme.
- afhof 15y agoI usually do the same thing with Newegg reviews. I find the low reviews more telling than the high reviews. The tend to list issues like driver problems or it didn't work with Linux or the like.
- hugh3 15y agoIndeed. While there's always some one-star reviews no matter how good the thing being reviewed might be, the best guide is just to read them and see whether they sound like rational complaints from sensible people. If the one-star reviews say things like "the checkin girl rolled her eyes at us the third time we asked for our room to be changed to one with a better view" then it's probably an alright hotel. If they say "filthy, noisy, unsanitary" then I'm probably not gonna stay there.
- prawn 15y agoHave to agree with you over ditching one-star reviews. There was a classic I discovered on TripAdvisor on a recent trip and which quickly became a catch-cry for my brother and I. The reviewer (female) complained that she had broken up with her boyfriend that day. There was no issue listed with the hotel itself, but the review finished with something like, "I cried all night. 1 star."
- tathagatadg 15y agoIt might be interesting that at UIC, we do a data mining course research project on this topic - http://www.cs.uic.edu/~liub/#projects http://www.cs.uic.edu/~liub/#projects. We used resellerratings.com and apart from a handpicked, there is no way of determining a pattern. There is no luxury of mechanical turks for a course project, and we are strictly forbidden from writing any reviews. I had to use the #of reviews, variance in similarity of the review text, time interval between posts, user since date, helpful review count, average rating(user/review/store),etc. and calculated the Mahalonobis distance[http://en.wikipedia.org/wiki/Mahalanobis_distance http://en.wikipedia.org/wiki/Mahalanobis_distance] to separate outliers as the spammers. Then with these as labeled spammers, used graph based semi-supervised learning to classify the reviewers as spammers. Its a wild open problem - and very easy to argue on both sides of any method :D
- amcintyre 15y agoIf I was writing a real review, it might look a lot like the one marked up with all the "deceptive" indicators, with the exception that I'm not a big exclamation mark user. The grumpy old man in me wants to suggest that anyone who actually learned how to write would be marked as a fake. Is well-written English so hard to come by nowadays that it makes people suspicious when they see it online? (Edited to add: apologies if I'm saying something that's covered in the article--couldn't get past the paywall.)
- vaksel 15y agopersonally I focus on negative reviews. But then you have to grapple with whether or not the bad review is true or not(i.e a competitor posting it). In those cases it's best to focus on places where the reviewer's reputation can be checked. i.e. all those review sites that let anyone review are more or less worthless. But a blog post by someone with hundreds of posts is a lot more likely to be a real story.
- marze 15y agoAs businesses get more adept at gaming the online review process, the rewards for a system capable of a high "honest review" fraction will increase.
- kleiba 15y agoThe interesting point lies in how they construct their training data. In addition to producing fake reviews via mechanical turk themselves, they use a complex procedure to retrieve actual positive reviews from tripadvisor. ...which is kind of funny if you think about it: to develop a classifier that can identify real reviews, the first thing they do is create a classifier that can identify real reviews to produce a training corpus. Then they try to approximate the output of their first (rule-based) classifier with a machine learning classifier.
- kahawe 15y agoSometimes I still cannot believe how prevalent fake reviews are. Not too long ago a friend of mine contributed a chapter to a programming book for one of the major publishers and they made sure to tell him to ask his friends to write reviews for the book on amazon, down to detailed instructions how to fake it "properly" and in accordance with how they were trying to position it on the market - like saying that "it is great for beginners to learn X" but one should also add a negative point about how this-and-that chapter needed more clarification or more examples or images.