4 ms·

GA used to be good in separating the bots from the humans, but now I don't think so. I see a huge number of visitors from Dublin that seem to be AWS sourced bot

by jerven 6y ago

GA used to be good in separating the bots from the humans, but now I don't think so. I see a huge number of visitors from Dublin that seem to be AWS sourced bots (and Asburn, both have bounce rates like no other city). Millions of visits, from a place where we would expect thousands.

cosmie 6y ago
GA's "Filter bots" setting simply applies the IAB bot filtering list[1]. The resources required to operate a page rendering bot[2] used to be a high enough bar that the ones doing so were large actors and would make their way onto that list pretty quickly. The IAB list is still a helpful baseline (if overpriced if you want to lease the list itself[3]), but all it's doing is applying suppression based on things like known bot IP ranges and user agents. It's far less effective than it used to be since it's so incredibly easy and cheap nowadays for anyone with an interest to spin up a rendering bot. Now you've got to supplement it actively with your own set of filters and heuristics if you really want to get rid of bot traffic polluting your data. [1] https://iabtechlab.com/software/iababc-international-spiders-and-bots-list/ https://iabtechlab.com/software/iababc-international-spiders... [2] Scraping bots were cheap and easy to run before, but didn't render the GA javascript code so never showed up in analytics. It's only bots that use something like a headless browser to render the page that show up in GA, and those have only become commoditized and cheap/easy relatively recently. [3] Google applies the IAB list to your traffic for free if you check the setting for it, but if you want to use the IAB list yourself you have to pay $4k - $14k annually to lease it from IAB.
eli 6y ago
The bots got better and harder to detect. You can actually set analytics to exclude traffic from a certain ISP or set of IPs on the View's Filters page. There is some legit traffic that comes from data centers, but only a tiny fraction.

by jerven 6y ago

cosmie 6y ago

GA's "Filter bots" setting simply applies the IAB bot filtering list[1]. The resources required to operate a page rendering bot[2] used to be a high enough bar that the ones doing so were large actors and would make their way onto that list pretty quickly. The IAB list is still a helpful baseline (if overpriced if you want to lease the list itself[3]), but all it's doing is applying suppression based on things like known bot IP ranges and user agents. It's far less effective than it used to be since it's so incredibly easy and cheap nowadays for anyone with an interest to spin up a rendering bot. Now you've got to supplement it actively with your own set of filters and heuristics if you really want to get rid of bot traffic polluting your data. [1] https://iabtechlab.com/software/iababc-international-spiders-and-bots-list/ https://iabtechlab.com/software/iababc-international-spiders... [2] Scraping bots were cheap and easy to run before, but didn't render the GA javascript code so never showed up in analytics. It's only bots that use something like a headless browser to render the page that show up in GA, and those have only become commoditized and cheap/easy relatively recently. [3] Google applies the IAB list to your traffic for free if you check the setting for it, but if you want to use the IAB list yourself you have to pay $4k - $14k annually to lease it from IAB.

eli 6y ago

The bots got better and harder to detect. You can actually set analytics to exclude traffic from a certain ISP or set of IPs on the View's Filters page. There is some legit traffic that comes from data centers, but only a tiny fraction.