4 ms·

Disclaimer: Not speaking for my employer, and my experience was several years ago and may be out of date. My first project at Google was working on fake review

Disclaimer: Not speaking for my employer, and my experience was several years ago and may be out of date. My first project at Google was working on fake reviews on Maps. ML was certainly useful for some things, but at the end of the day sometimes you literally have no practical way of knowing if a review is fake or not. Was this written by a customer or by the business owner's brother in law? Or by his competitor down the street? Who knows? Which means it's pretty hard to get good, complete training data to train your model on. Of course, there are classes of fake reviews that are easy enough to detect. But as the fake review-writing AIs get better, I don't know how the anti-fake review AIs can win.

sametmax 8y ago
Well, if a human can't tell, at some point, the AI can't tell either. Since humans are good at creating lies, and that we all get fooled at some moment in our life, it means the problem hasn't been solved for millennial and that it's not an AI problem. Sure you can throw at the AI way more data than we used to have, but even totalitarian state didn't manage to silenced the opposition so I doubt a business bound by law can do anything about it. Google protection measures are already becoming super invasive and instead of helping me, has locked me out of my account several times because it detected I was a fraud. There is a limit on what you can do properly here.
dmortin 8y ago
> at the end of the day sometimes you literally have no practical way of knowing if a review is fake or not Can't you check the account itself? If the account's behavior over time shows signs it's an actual human user instead of a bot. This can't filter the owner's brother in law, but if bot accounts can be filtered then they cannot leave lots of reviews either, because the owner does not have hundreds of relatives.
- Operyl 8y ago
  Then the problem becomes: are you stifling reviews from someone who just created an account and has no behavior to be spoken of?
  - dmortin 8y ago
    Since the problem of fake reviews is pretty serious I think it would be an acceptable compromise if fresh accounts could not write reviews. The account would need a bit of history, e.g. one or two weeks, before allowing it to write reviews.
    - plankers 8y ago
      That would also risk turning off new users who create an account explicitly to leave a review.
      - dmortin 8y ago
        Then allow them to write reviews like normal, but the review would actually be visible only for them in their trial period. The user would see the review, others wouldn't, and after 1 or 2 weeks when the user activity confirms the user is not a bot, make the review visible for everyone.
        nitrogen 8y ago
        Have you thought about what that looks like from the user's perspective? You'd just as likely turn them away for life with such an unpleasant experience.
        dmortin 8y ago
        The user would see his own review like today. The user wouldn't notice anything unusual. What would be the unpleasent experience? Surely, you don't think that average users check their reviews from private browsing tabs to make sure it's there and stuff.
        nitrogen 8y ago
        They intended to post a review. Not posting it but saying you did is straight up lying. Shadowbanning is morally dubious as a rarely used tool. It's pure evil as a default. Do you not care about lying to them just because they are "average"?
- splonk 8y ago
  (I probably worked with one of GP's predecessors on his job.) Account level signals are nice but insufficient. There's a very long tail of low activity accounts that are difficult to distinguish from a bot. You can't really classify them as bots without generating a ton of false positives, and in general you'd rather let a possibly fake review through than piss off a genuine first time reviewer.
  - telchar 8y ago
    I imagine Google has big advantage here, at least for brick and mortar places since they can use location data to determine whether the account connected to a device has physically been in a place. It would also explain why they always ask me to review/rate businesses 5 minutes after I walk in.
    - splonk 8y ago
      All the real-time location based stuff is after my time, but it definitely looks like an attempt to generate more known-good reviews. Possibly for better training data for modeling. I'm not sure that the historical location data is as useful as you might think - there's often a large temporal gap between the experience and the review writing. Google does have a big advantage in verifying good accounts through other activity, but that advantage shrinks dramatically for new accounts. There's still some things you can do, but at the end of the day you're only willing to assign so much risk to a brand new account based on metadata.
  - dmortin 8y ago
    Allowing reviews only from active accounts could at least alleviate the problem and in case of Google it can even check the account's activity via its various services. A google account cannot really be low activity if its owner browses the web, watches youtube, etc. The tiny minority who logs out of google when browsing, etc. could not leave a review, but it may be a small price for more valid reviews.
  - otterley 8y ago
    Isn't that what shadowbanning is for? If the initial conclusion of the evaluation is that the user is a bot, then you can make the submitting user think the post was successful; but then you can hide it from everyone else until the result changes to one that the user is real.
- tmpz22 8y ago
  Reminds me of player automation used to cheat in online video games. You'd write code to automate player movement and activity in order to progress in the game 24/7 rather then the couple hours a day an average player could do. The strategy at many studios was to use a barrage of imperfect solutions which raised the effective per/unit cost of botting to something most players weren't willing to accept, then hunt the remaining "big-time" offenders using a separate set of tactics, like identity "fingerprinting", correlating billing information etc.

sametmax 8y ago

Well, if a human can't tell, at some point, the AI can't tell either. Since humans are good at creating lies, and that we all get fooled at some moment in our life, it means the problem hasn't been solved for millennial and that it's not an AI problem. Sure you can throw at the AI way more data than we used to have, but even totalitarian state didn't manage to silenced the opposition so I doubt a business bound by law can do anything about it. Google protection measures are already becoming super invasive and instead of helping me, has locked me out of my account several times because it detected I was a fraud. There is a limit on what you can do properly here.

dmortin 8y ago

> at the end of the day sometimes you literally have no practical way of knowing if a review is fake or not Can't you check the account itself? If the account's behavior over time shows signs it's an actual human user instead of a bot. This can't filter the owner's brother in law, but if bot accounts can be filtered then they cannot leave lots of reviews either, because the owner does not have hundreds of relatives.

Operyl 8y ago
Then the problem becomes: are you stifling reviews from someone who just created an account and has no behavior to be spoken of?
- dmortin 8y ago
  Since the problem of fake reviews is pretty serious I think it would be an acceptable compromise if fresh accounts could not write reviews. The account would need a bit of history, e.g. one or two weeks, before allowing it to write reviews.
  - plankers 8y ago
    That would also risk turning off new users who create an account explicitly to leave a review.
    - dmortin 8y ago
      Then allow them to write reviews like normal, but the review would actually be visible only for them in their trial period. The user would see the review, others wouldn't, and after 1 or 2 weeks when the user activity confirms the user is not a bot, make the review visible for everyone.
      - nitrogen 8y ago
        Have you thought about what that looks like from the user's perspective? You'd just as likely turn them away for life with such an unpleasant experience.
        dmortin 8y ago
        The user would see his own review like today. The user wouldn't notice anything unusual. What would be the unpleasent experience? Surely, you don't think that average users check their reviews from private browsing tabs to make sure it's there and stuff.
        nitrogen 8y ago
        They intended to post a review. Not posting it but saying you did is straight up lying. Shadowbanning is morally dubious as a rarely used tool. It's pure evil as a default. Do you not care about lying to them just because they are "average"?
splonk 8y ago
(I probably worked with one of GP's predecessors on his job.) Account level signals are nice but insufficient. There's a very long tail of low activity accounts that are difficult to distinguish from a bot. You can't really classify them as bots without generating a ton of false positives, and in general you'd rather let a possibly fake review through than piss off a genuine first time reviewer.
- telchar 8y ago
  I imagine Google has big advantage here, at least for brick and mortar places since they can use location data to determine whether the account connected to a device has physically been in a place. It would also explain why they always ask me to review/rate businesses 5 minutes after I walk in.
  - splonk 8y ago
    All the real-time location based stuff is after my time, but it definitely looks like an attempt to generate more known-good reviews. Possibly for better training data for modeling. I'm not sure that the historical location data is as useful as you might think - there's often a large temporal gap between the experience and the review writing. Google does have a big advantage in verifying good accounts through other activity, but that advantage shrinks dramatically for new accounts. There's still some things you can do, but at the end of the day you're only willing to assign so much risk to a brand new account based on metadata.
- dmortin 8y ago
  Allowing reviews only from active accounts could at least alleviate the problem and in case of Google it can even check the account's activity via its various services. A google account cannot really be low activity if its owner browses the web, watches youtube, etc. The tiny minority who logs out of google when browsing, etc. could not leave a review, but it may be a small price for more valid reviews.
- otterley 8y ago
  Isn't that what shadowbanning is for? If the initial conclusion of the evaluation is that the user is a bot, then you can make the submitting user think the post was successful; but then you can hide it from everyone else until the result changes to one that the user is real.
tmpz22 8y ago
Reminds me of player automation used to cheat in online video games. You'd write code to automate player movement and activity in order to progress in the game 24/7 rather then the couple hours a day an average player could do. The strategy at many studios was to use a barrage of imperfect solutions which raised the effective per/unit cost of botting to something most players weren't willing to accept, then hunt the remaining "big-time" offenders using a separate set of tactics, like identity "fingerprinting", correlating billing information etc.