10 ms·
80% of my traffic is excluded from Google Analytics
- adrianN 5y agoI wish we could just go back to "analytics" being an moderately complicated awk invocation on whatever logs your webserver produces.
- frenkel 5y agoWhy can’t we?
- gentleman11 5y agoWhat percent of visits are bots and how do you detect them serverside? That’s the only part I’m unsure of
- yjftsjthsd-h 5y agoCan you detect bots client side? I sorta assumed they used headless Chrome and looked like real users nowadays, although I guess with effort you can probably fingerprint?
- thayne 5y agoYou can if you track all of the user's online activity and use machine learning to determine if the behavior seems genuine. That's how invisible captcha works, I'd be surprised if google analytics didn't do something similar. Of course, that does mean you have to track all the user's online activity....
- NmAmDa 5y agoWhat about GDPR potential violation?
- adrianN 5y agoGDPR has exceptions for fraud detection.
- Nextgrid 5y agoA bot does not immediately imply fraud. If you're processing personal data to detect bots to prevent let's say card fraud, fair enough. If you're processing personal data to detect bots for analytics purposes, this might not fly. Of course, this assumes someone actually gives a shit about enforcing the GDPR, which is currently not the case.
- adrianN 5y agoI have literally worked on bot detection and the high powered company lawyers told me that it's okay for my code to collect and analyze PII for these purposes because GDPR has these exceptions. I'm not a lawyer myself though, so that might have been wrong.
- jerven 5y agoFor one of the sites I work with. Headless chrome defenitly inflates the GA user numbers. We can spot this because they tend to come from big cloud data centre locations when doing ip to geo mapping.
- yjftsjthsd-h 5y ago> We can spot this because they tend to come from big cloud data centre locations when doing ip to geo mapping. Of course, you can do that with server side logs, so in this case client side is less accurate:)
- NavinF 5y agoThe default build of chromedriver is pretty easy to detect client side because it injects js variables with known names. Most bots don't really care if they're detected and excluded from analytics. Unfortunately a few sites unfairly throttled me to 1 request/hour despite the cost of my requests being fuck-all. So a couple of years ago I had to randomize those variable names, distribute requests over 64 IPs, and screw up all their analytics numbers in the process. I hope it was worth saving $0.01 for them.
- adrianN 5y agoMost bots are pretty stupid, so it's easy to detect them. Bot detection need not be 100% accurate. It's an arms race with the goal to make writing bots financially unattractive.
- digitalengineer 5y agoThis can help: https://www.darwin.so/features/bots https://www.darwin.so/features/bots
- inetknght 5y ago> What percent of visits are bots It doesn't matter. > how do you detect them serverside? Instead of trying to detect whether something isn't human you should be trying to detect whether something is malicious.
- darig 5y agobecause developers don't know how to build anything anymore. they paint themselves into smaller and smaller focuses relying on someone else to provide them with everything they need. no more web developers... now we have front-end and back-end all fighting over who needs to do the architecture and build the necessary tooling.
- rognjen 5y agoIf I need accurate traffic numbers (for performance or server sizing for instance) I use GoAccess (https://goaccess.io/ https://goaccess.io/)
- deleted 5y ago[deleted]
- foobar33333 5y agoAnd then a user sends a support request with a screenshot showing an in browser error and demands you fix it but your server logs show nothing because this was a JS issue. While people who use frontend logging/tracking just search the user ID and see what went wrong.
- tinus_hn 5y agoIn Google Analytics?
- matsemann 5y agoThat has nothing to do with external visitor analytics, though. I've implemented log sending from frontend to backend multiple times. First party, same endpoints as other requests. Not getting blocked.
- lucb1e 5y agoTo me, one doesn't have to pre-emptively record every user session on the off chance that they run into a problem large enough to bother contacting support for. Having a 'copy error' button and/or logging the errors to a server seems sufficient here, no need to hire the biggest brother out there to track your users for that.
- Tenoke 5y agoThis doesn't work too well nowadays since a lot of visits don't hit my server as Cloudflare itself is serving them the page.
- Nextgrid 5y agoIn this case you'd get your logs from CF instead of your origin server.
- maple3142 5y agoIt doesn't work if the website is hosted on GitHub Pages or Netlify. Having a CDN such as CloudFlare will make log files on your server useless too.
- bruce343434 5y agoHow so? Isn't that just for pictures and other media?
- maple3142 5y agoHosting a (static) website on GitHub pages means you put all your resources including html files into a repo, then the whole website will be served from GitHub's server. You don't even have a server to view your access log. Netlify is pretty similar. As for CloudFlare, all the requests will go through CloudFlare's server for caching, so some request won't reach your server.
- xuki 5y ago> As for CloudFlare, all the requests will go through CloudFlare's server for caching, so some request won't reach your server. Only for static files, they don't cache HTML and dynamic contents. So you can still make sense of unique visitors to your site.
- viraptor 5y agoThat really depends on your setup. The cache whatever you tell them to cache. HTML and dynamic content is also cacheable with good results. For example you can vary the cache by the user ID cookie. Or for dynamic content you can in most cases do caching for 5min or so.
- realityking 5y agoNetlify offers an add-on for server-side analytics: https://www.netlify.com/products/analytics/ https://www.netlify.com/products/analytics/
- wdr1 5y agoTurning impression logs into people counts turns out to be a pretty difficult problem. Even with cookies it's still very complicated. Facebook has publicly acknowledged screwing it up several times. We could go back to the days of talking about "hits" on a website, but for most things where you'd care about these types of metrics, it's a pretty crap metric.
- rognjen 5y agoOr is that 80% of HN uses adblock? > Is GA still relevant today? Of course it is. While a large part of HN audience might use adblock, very few regular users do and almost no mobile users. Looking at my logs, however, I get a similar proportion. ~25k in GA, ~100k in logs.
- gorgonzolachz 5y agoThis is the right answer - HN is absolutely not representative of web users at large. If we were, I'm sure there would be much stricter anti-adblock/ad protection systems on large sites.
- sjtindell 5y agoTwitch and other services have already started down the road of embedding ads in the video stream rather than loading separately, etc. I use an ad blocker and am torn sometimes because I know my favorite content creator isn’t getting my view and I’m contributing to the coming wave of even more invasive ad tech.
- toxik 5y agoI think paying to get rid of ads is fair. I pay for YT Premium solely for that reason, it comes out to about $5/month/person if you use the family subscription. It’s expensive but I find it’s worth it to not have my partner and our parents watching ads. I watched the Euro 2020 games, and even the 5-second soundbite of “the games are sponsored by so-and-so” rubs me the wrong way. Why should I be told the name of some company when I’m watching football? I digress.
- riffraff 5y agoI get the hate for increasingly invasive ads (screw the little car that carries the ball), but I chuckled at the last sentence: we've had sponsors on football jersey for 50 years :)
- 5y ago
- deleted 5y ago[deleted]
- sMarsIntruder 5y agoIt definitely depends on the audience you have and the adblocker adoption for each country. "mainstream" websites still show a 5-10% gap. That for marketing usages it's still ok.
- speedcoder 5y agoWhat creeps me out even more is the ads and "automatic bing searches" that are cropping up in the Windows start menu and weather (location) tracker with ads in the taskbar. All this enabled by default after a Windows update and active while the user is not explicitly browsing something on the Internet. What metrics Microsoft is collecting?
- oefrha 5y ago> <s>80% of my traffic is excluded from Google Analytics</s> Unique visitor count on Google Analytics is 80% lower than the one on Cloudflare. FTFY. Cloudflare’s unique visitor count always felt inflated to me, compared to both GA and non-GA analytics solutions.
- sknzl 5y agoI came across the same issue recently and found the answer in the Cloudflare help. Unique visitors on CloudFlare are different from unique visitors of Google Analytics. Google Analytics expects the client to execute JS (most bots don’t do that) and Google Analytics excludes known bots from the unique visitor count. The relevant Cloudflare article: https://support.cloudflare.com/hc/en-us/articles/360037684111#4lt2VoRUorCudxN1xzxpOt https://support.cloudflare.com/hc/en-us/articles/36003768411...
- imalerba 5y agoIf the difference between GA and Cloudfare is only bots, then the problem is pretty terrible.. "80% of my traffic is bots"
- Tenoke 5y agoThe main difference is the execution of JS, less so the bots.
- lilyball 5y agoVery few non-bot visitors refuse to execute JS.
- th3typh00n 5y agoJavascript in general? Sure. Telemetry-specific Javascript on the other hand is prevented from executing by many ad blockers, for good reasons.
- lilyball 5y agoThey don't block JavaScript, they block the loading of resources deemed to be for advertising. These are typically ads, but they can also be images, CSS, or other resources hosted by advertisers. Such blocking will also block any other kind of tracking done by the external domain. On the other hand, JavaScript inline in the page is almost never blocked by users.
- randomperson_24 5y agoIt's not just Google Analytics, but any other alternative be it plausible, Matomo, or any other JS analytics solution. I would also say cloudflare doesn't count uniqu visitors accurately. Also recently there is big uptick in bots which server logs show, but are not real users.
- ipaddr 5y agoAre bots not unique visitors?
- throwaway3699 5y agoNo, of course not. On a small website, the bots are such a huge part of the logs that the whole exercise becomes an attempt at extracting understanding from random noise. It's why those visitor count widgets died out years ago.
- randomperson_24 5y agoTrue
- hamhamed 5y agoWe see around the same thing and our audience is totally NOT from HN or hacker like. Mixpanel and Cloudflare both report around 886k uniques (mixpanel being 5% less) for yesterday's Friday traffic... while GA reports 129k... that's a solid 80%+ too.
- deleted 5y ago[deleted]
- exciteabletom 5y agoAlso worth noting that recent versions of Firefox block GA by default
- SquareWheel 5y agoIt seems to allow it here. Fully up-to-date, default settings.
- exciteabletom 5y agoI should clarify it isn't blocked in the traditional sense. It loads a benign script that pretends to be GA but doesn't touch Google servers at all, to avoid breaking sites that expect GA to be available.
- Ozzie_osman 5y agoDealt with this recently. Something like half of Americans are using some sort of privacy blocker on desktop [1]. It's not as high on mobile but still sizable. My guess is it's higher with the HN crowd. I dealt with something similar so I had my back-end fire server-side events to compare to Google Analytics and other client-side reported data. Sure enough, 50% loss on desktop and 30% on mobile. 1. https://www.forbes.com/sites/tjmccue/2019/03/19/47-percent-of-consumers-are-blocking-ads/ https://www.forbes.com/sites/tjmccue/2019/03/19/47-percent-o...
- stevekemp 5y agoAgreed. I used to support my hosting costs via google adsense. Over recent years the impressions and clicks dropped off a cliff. In the end I realized I was just as guilty, after all I run adblock too - so I removed them all.
- duckmysick 5y agoHow do you support your hosting costs now?
- stevekemp 5y agoOut of pocket expenses, mostly. Though I did shuffle things around so that they cost me less these days.
- mattlondon 5y agoI think that GA samples the data anyway (unless you pay?), so it might not be a reliable "hit counter" these days. I guess the value comes from seeing "trends" in your site content - I.e. how people move around within the site, which sites are sending traffic etc rtlather than seeing absolute 100% accurate counts Log analysis will always be much more accurate counter if you just need a hit counter.
- timdaub 5y agoI made a similar observation with Plausible and Cloudflare and wrote about it: https://rugpullindex.com/blog#HowPlausibleareOurWebsiteAnalytics https://rugpullindex.com/blog#HowPlausibleareOurWebsiteAnaly...
- saimiam 5y agoThis link doesn’t seem to work correctly on mobile. It showed the right post for about a second, then scrolled to something from June 2nd and May 27th.
- jonplackett 5y agoI had the same thing when my game https//termsandconditions.game got to #1 spot on HN. I was using https://plausible.io https://plausible.io which I presume has a higher success rate than GA, but I still saw actual usage of resources (requests to the CDN) about 4 times higher than reported. I guess you couldn't pick more ad-block-heavy audience than HN though right?
- Saris 5y agoI use plausible on my site (and tested out a few other similar options), and ublock origin and brave browser both blocked all of them. So I guess it's not too surprising that it's still the same ratio.
- sgaduuw 5y agoSo we might be optimising for the wrong things due to survivor bias?
- feistypharit 5y agoYou can actually use cloudflare workers to mitigate this. You use a worker to act as a proxy so that most blockers won't block the tracker. More details: https://blog.garble.org/using-cloudflare-to-increase-analytics-accuracy/ https://blog.garble.org/using-cloudflare-to-increase-analyti... I don't believe it would scale well to a big site though ($$)
- asadkn 5y agoFirefox has built-in tracking protection since a while and would block Google Analytics. Similarly, Adblock extensions would also block GA (usually behind a strict setting). Considering we're talking HN visitors, this wouldn't be representative of visitors in general.
- jefftk 5y agoFirefox doesn't block analytics in its default configuration; see https://news.ycombinator.com/item?id=27792881 https://news.ycombinator.com/item?id=27792881 above
- nokya 5y agoI don't know the answer but with my smartphone I'd rather connect to a website through a VPN running on my own server (with pihole and dns sinkhole), than actually browsing any website directly.
- herbst 5y agoIf you want real numbers just parse your server log. I stopped using analytics years ago because in average it would only show half of my actual traffic. Goaccess works great.
- sleavey 5y agoIronically this is how it was done before Google Analytics.
- herbst 5y agoI even remember many refusing to use Google analytics for privacy reasons for a few years until everyone got lazy I guess
- rchaud 5y agoHow do you filter bot traffic from your server logs? If all you are looking for is a 'hit counter', then GA is overkill. It's value is not in providing page traffic data, but in tracking things like click events, ecommerce funnels, marketing campaigns, A/B testing etc.
- herbst 5y ago> How do you filter bot traffic from your server logs? That's a option for goaccess --ignore-crawlers Honestly I am not doing a lot of tracking on my users, the few campaigns I run I simply track within my application. That way I also don't miss any relevant Events. Also a lot of my traffic is from tech people, so using anything like GA would also only show a small part of the audience I actually mostly care about.
- arpowers 5y agoHey guys, I’m the founder of Darwin (free analytics) https://www.darwin.so https://www.darwin.so There are a lot of ways to measure traffic depending if you care about all visits, https requests or just real users. For example, in our research we've found that around 56% of internet traffic is actually headless bots (bots pretending to be human). If you not only count headless bots but also count http requests as users then you'll have 5/1 ratio as Eric mentioned here on twitter. - Cloudflare uses cache hits and so on to measure traffic and therefore use http requests. Most of which are coming from data centers. This means their numbers are highly inflated. - GA allows HTTP requests as well but this traffic can be easily filtered. They allows most/all bots and provide no tools to fight this. - Darwin we exclude everything we can to try and ensure the best measure of "real" users. This we believe is more helpful to marketers. tl;dr your real traffic is 10% of what you see in cloudflare, 30% of what you see in unfiltered GA. With Darwin around 85% of traffic is real.
- MiddleEndian 5y agoGood.
- idoxer 5y agoSafari also blocks GA by default, also plausible have some versus articles about Cloudflare analytics (both server and client side) https://plausible.io/vs-cloudflare-web-analytics https://plausible.io/vs-cloudflare-web-analytics https://plausible.io/blog/server-log-analysis https://plausible.io/blog/server-log-analysis
- whatever_dude 5y agoSame, and the strongest reason why I stopped using GA altogether. People don't talk much about that, I suppose, but to me it's clear GA is completely dead as a product.
- kbelder 5y agoThis isn't true at all. If you get outside the niche tech field, you'll see that GA is blocked by 6% - 10% of unique visitors... less on mobile, which is growing.