3 ms·

IP masking was applied very early but it did cut down a large number of requests, I am not sure how many though. Next were filters that would block devices send

IP masking was applied very early but it did cut down a large number of requests, I am not sure how many though. Next were filters that would block devices sending more than a few hundred requests a minute or so on. These would be usually bots or compromised devices. Then we would look at cuts of traffic where the characteristics were outside of acceptable parameters and block those. E.g., say a particular OS, OS version, device model, request country etc. This could be made much more granular. If you see too high a CTR or too low a conversion rate, or abnormal time between ad being displayed and being clicked on (fraud clicks had a tighter distribution between request and click as compared to genuine clicks) or any other weird stuff, you could block that cut of traffic. Also, we refrained from calling it fraud, the term was invalid traffic. This also accounted for things like someone double tapping on an ad instead of once, and so on. There is a good report by Alexander Tuzhilin [1] commissioned by Google when they were sued for charging for invalid traffic that might give you some more ideas. [1] https://googleblog.blogspot.com/pdf/Tuzhilin_Report.pdf https://googleblog.blogspot.com/pdf/Tuzhilin_Report.pdf

bredren 6y ago
Thank you for this. I’m more interested in it from the perspective of anti-web scraping algorithms. I was curious, when you say block traffic, would you send 400s or silently just not record the crucial events? It seems like maybe tipping off automation would cause them to readjust. Perhaps they are not that determined?
- bongoman37 6y ago
  Oh they were pretty determined. Sometimes when we would block traffic on particular cuts, we could see changes in traffic happening in as little as 30 mins which meant that fraudsters had automated systems responding to changes on our end. Things such as a single device ID sending hundreds of requests in a minute or blacklisted IPs would be just dropped silently, no point in even wasting bandwidth to show an ad. For cuts of traffic where the clicks etc were outside of predetermined ranges, it would again be blocked and dropped. In other cases, where fraud happened before we could block it, there was a flag called 'billable' which was set to False so the advertiser wasn't charged for it.
  - bredren 6y ago
    I see. Who were the fraud operators affiliated with / what was their incentive? Isn’t it the publishers that would see value, or perhaps a competitor trying to disrupt marketing efforts? Do you see opportunity for innovation in traffic validation? Have you seen much in the way of training ML for this purpose?

bredren 6y ago

Thank you for this. I’m more interested in it from the perspective of anti-web scraping algorithms. I was curious, when you say block traffic, would you send 400s or silently just not record the crucial events? It seems like maybe tipping off automation would cause them to readjust. Perhaps they are not that determined?

bongoman37 6y ago
Oh they were pretty determined. Sometimes when we would block traffic on particular cuts, we could see changes in traffic happening in as little as 30 mins which meant that fraudsters had automated systems responding to changes on our end. Things such as a single device ID sending hundreds of requests in a minute or blacklisted IPs would be just dropped silently, no point in even wasting bandwidth to show an ad. For cuts of traffic where the clicks etc were outside of predetermined ranges, it would again be blocked and dropped. In other cases, where fraud happened before we could block it, there was a flag called 'billable' which was set to False so the advertiser wasn't charged for it.
- bredren 6y ago
  I see. Who were the fraud operators affiliated with / what was their incentive? Isn’t it the publishers that would see value, or perhaps a competitor trying to disrupt marketing efforts? Do you see opportunity for innovation in traffic validation? Have you seen much in the way of training ML for this purpose?