10 ms·

Analytics Without Google

203 points by luxurytent 6y ago

joe5150 6y ago
GoAccess is really cool. Matomo is free and self-hosted if you want more of the features of Google Analytics like tracking time on page.
- XCSme 6y ago
  I am also working on a similar product to Matomo, it's not free but it also provides some of the Matomo's premium features for a much cheaper price: https://usertrack.net https://usertrack.net
gpanders 6y ago
I noticed this post mentions GoatCounter but says that it doesn't meet his needs because it isn't self-hosted. It is not by default, but you certainly can self host it, and the author has been quite open about that being a viable path for people interested in doing so. I self host GoatCounter myself and it works very well.
- Carpetsmoker 6y ago
  It looks like I have some work to do on clarifying the homepage here, hah.
- luxurytent 6y ago
  Hey, yeah. This is my bad! I didn’t dig deep into GoatCounter but on second glance I can see the link to source. I amended my article to clarify what I know (not much)
- galfarragem 6y ago
  +1 to Goatcounter. Not affiliated with it, just an happy (free) user on a low traffic/hobby project. Simple and no BS. More that enough if you just want to know the big picture and care about privacy. The data I get is this (my data is public but it’s up to you): https://slowernews.goatcounter.com/ https://slowernews.goatcounter.com/
- owenshen24 6y ago
  Also a happy donating user for the free plan. (No affiliation, I just like the project.)
johnghanks 6y ago
So... more work, slower, and fewer features than GA.
- llukas 6y ago
  Reduced surveillance sounds like a welcome feature.
  - johnghanks 6y ago
    Here's a tip: you and the users of your site are being tracked anyways.
    - nicbou 6y ago
      Yes, but the data is used by you and only you. It's not shared with Google. It's not merged with data from other sites and used against your users. The collected data is under your control, and it's used solely for its intended purpose.
- b20000 6y ago
  google is only interested in selling you ads. They don't care whether you actually get anything out of their analytics product. If it's free, you ARE the product.
  - johnghanks 6y ago
    And? GA is a good product.
    - b20000 6y ago
      it is not, it only gives you the data in a way that is convenient for them to sell you ads. it does not give you the data in a way that you want or is best for your business.
  - tjbiddle 6y ago
    Correction: If it's free, your users are the product. And an addition: If you're enterprise and need Google 360, which is very much not free ($100k+/year I believe), not only are you paying - but your customers are still the product!
girzel 6y ago
I feel like all I need is a good script to filter out bots and spiders. I could take the rest from there.
- luxurytent 6y ago
  This is an interesting thought. I wonder if I could pre process my log files to get rid of this noise ahead of piping to a log analyzer.
  - girzel 6y ago
    Or even a logging module in the web server itself...
  - Carpetsmoker 6y ago
    I worked on a library to do this for my own analytics thing[1]; adding a CLI so you can filter bots from logfiles isn't too hard: just need to parse the log lines to a http.Request{} with the correct fields filled in (mostly User-Agent, but also looks at IP address). One of the signals is from various JS properties though (navigator.webdriver, window._phantom), so you'll miss out on that. I have some other ideas as well. Either way, it's better than nothing? https://github.com/zgoat/isbot https://github.com/zgoat/isbot
    - luxurytent 6y ago
      This is cool! I've bookmarked it for future review. Thanks for sharing
  - nicbou 6y ago
    You could do this with nginx for sure. I don't know what the impact on performance would be.
esperent 6y ago
One possible benefit that I don't see discussed much in this context is bypassing ad blockers. I run a tech focused website and Google analytics registers about 10k visits/month. I figure that a good chunk of my visitors have ublock and so they don't show up. Presumably, alternative analytics or self hosted analytics are not blocked, so I'd get more accurate stats. Is this a correct assumption?
- dmkii 6y ago
  It’s true, but GA also does a pretty good job at filtering out bot traffic for example, so it depends on what your definition of more accurate is. There are also ways to send GA hits through a sub domain to make it look self hosted and bypass ad blockers.
  - markdown 6y ago
    > It’s true, but GA also does a pretty good job at filtering out bot traffic for example Surely you jest. I had to quit using GA because the tech behemoth with all it's tens of thousands of software and algo experts couldn't figure out how to filter out referrer spam from their analytics. Such a simple problem you'd think they'd solve it in a day. I gave them 2yrs and they still couldn't solve that problem so I left.
    - kevin_thibedeau 6y ago
      Parent speaks the truth. I quit because of the garbage referers coming out of Russia that Google would do nothing about.
      - esperent 6y ago
        How do you know they were garbage?
        markdown 6y ago
        https://www.google.com/search?q=%22buttons-for-websites%22+referrer+spam https://www.google.com/search?q=%22buttons-for-websites%22+r...
chatmasta 6y ago
GoAccess is cool but it won't help you much if you have a static website or if you serve the majority of requests from a CDN. In those cases, Matomo (previously piwik) is a good solution for client-side JS-based analytics similar to Google Analytics.
- PenguinCoder 6y ago
  How does Matomo work better with a static site, than GoAccess? GoAccess reads the server logs and creates metrics. Matomo requires javascript; that isn't 'static website'.
  - Carpetsmoker 6y ago
    There are no server logs on GitHub pages, Netlify, etc. Analysing server logs also makes it harder to filter bots, and won't include some useful metrics like screen size.
  - doh 6y ago
    GoAccess reads metrics of a server you have access to. So if you are distributing from a CDN or for instance GitHub or some other hosted solution, then you have no logs to read from. Also usage of javascript doesn't make one page non-static. Static is usually referred to a page that is rendered in full before a request comes in so the content does really change based on the request. More simply, content doesn't change dynamically based on the request. Having javascript to generate metrics doesn't make the page dynamic.
- sradman 6y ago
  I’m not familiar with GoAccess or Matomo but after some Googling it looks like Matomo requires self-hosting it’s PHP/MySQL server. If you are using a CDN then I’m assuming that multiple access.log files need to be aggregated at some point unless the CDN has a service that automates the process. Logstash and the full ELK Stack (or alternative) seem to be required when multiple heterogeneous servers are involved in serving content. Browser-side JavaScript analytics seems to avoid the DevOps surrounding the ELK Stack. GoAccess seems like a minimalist solution when you control a single httpd-style server and can run a local daemon to process the access.log file. As a side note, it looks like you need an Enterprise account to use Cloudflare’s Logpush Service or Logpull API. Amazon’s CloudFront has an advantage here. Avoiding Google Analytics is non-trivial for the lazy and/or price conscious.
celsoazevedo 6y ago
Something like Matomo running on a cheap VPS (less than $5/month) is a good Google Analytics replacement for small projects. It's cheaper than Simple Analytics (mentioned in the article) and no data is sent to a service operated by a 3rd party because it's self-hosted.
- XCSme 6y ago
  I am also working on a similar product to Matomo, it also provides some of the Matomo's premium features for a much cheaper price: https://usertrack.net https://usertrack.net
epoch_100 6y ago
Props to the author for ditching Google. I just want to point out that there are a LOT of non-Google analytics options out there; leaving Google Analytics really isn’t too hard if you’re willing to take the leap. Plug: those who want a modern client-side analytics tool that’s free, self hosted, and open source might consider Shynet [0]. (Disclosure: I maintain it.) It’s a bit simpler/cleaner than Matomo, but exists in the same category. [0] https://github.com/milesmcc/shynet https://github.com/milesmcc/shynet
- tjbiddle 6y ago
  Google Ads still dominates though, and if you're doing paid advertising - you're not just shooting yourself in the foot, but you're lopping off a limb - or two. I wish it wasn't the case. Any non-Google analytics options integrate well with Google Ads, especially for retargeting?
  - sdoering 6y ago
    For retargeting you would probably have to have the DoubleClick code on your site nonetheless. But at least serving European users you would only be able to activate it after a clear opt in via a consent manager. Using marketing parameters (like utm-... in GA or Matomo) or the likes in any other tool still gives you clear marketing performance metrics in your analytics solution of choice, though. And a lot of "pure" tracking tools would - at least currently - not fall under the opt in rules of thew GDPR. But the moment you link them with your advertising profiles (like in GA with the DoubleClick integration) your whole tracking setup also becomes opt in. So you would probably loose at least some share of your traffic in analytics and not be able to clearly do marketing analysis any more. At least not as exact as before. I had clients loosing 80% of their traffic stats in analytics after shifting to opt in.
  - knicholes 6y ago
    Take a look at Adobe's Experience Platform. Adobe Analytics is great. We take the experience events and profile events and build a unified profile upon which you can build segments used for targeting in Adobe's Advertising Cloud, Adobe Campaign, Adobe Target, and Adobe Experience Manager, and probably more... The Advertising Cloud is integrated with many third-party advertisers. I'm just a software engineer, though. We probably demand an extremely high price.
vladoh 6y ago
Very cool project! It is good that more people are trying to go Google-free. Your project does server side analytics, though, while Google Analytics is client side. Both different advantages and disadvantages. Server side analytics advantages: - No overhead on your website, because no external resources are loaded - Not affected by content blockers - Can track resources other than web pages, like images or redirects - Show resources that could not be found Client side analytics advantages: - Can capture more data about the user, like device size, browser and OS - Better track user behavior, like time spent on each page - Provide real-time feedback about visitors - Usually easier to setup (3/8) I've been playing around with different analytics options on my blog recently - Netlify on the server and Plausible Analytics on the client side. The results from 7 days comparison turned to be quite different [0]. The server side tool shows much more views across all pages in my blog. I think this is because the pages are accessed many times from automated systems that don't run JavaScript and are therefore not counted on the client side. This happens when you post a link on Twitter or WhatsApp for example, because their websites or clients then make a request to fetch the metadata of the page and display a card preview. Therefore, it counts on the server, but not on the client side. A perfect example is the draft article, which I only sent per WhatsApp to my wife for proof reading. The difference there is just 2 views (I guess one preview each on my and on her phone). I sent all other links to several people and posted them on Twitter and Reddit. You can also somewhat see this effect in the referrers [1]. The server side tool counted many hits from a RSS reader and Google Translate (my parents in law reading my blog in Bulgarian :D). The preview requests from Twitter and WhatsApp are probably counted as direct traffic. I think client side analytics more realistically count the number of people reading my blog (even if not many). However, there are cases that can be captured only on the server side (for example Google Translate). So, the best way would be to use both or maybe even make a service that combines client and server side - I think there is a lot of potential there! [0] https://pbs.twimg.com/media/EY-IqtdXsAAJvu9?format=jpg&name=medium https://pbs.twimg.com/media/EY-IqtdXsAAJvu9?format=jpg&name=... [1] https://pbs.twimg.com/media/EY-J0c8WoAEwn2J?format=jpg&name=medium https://pbs.twimg.com/media/EY-J0c8WoAEwn2J?format=jpg&name=...
- akdas 6y ago
  There's one more advantage to client side analytics, though maybe you meant to cover that under "Usually easier to setup": it's the only option if you host your site on a platform like Github pages. The problem is, you don't have access to server-side logs on these platforms.
johnchristopher 6y ago
Nice. Can it be used to compare this year's first quarter mobile usage for dutch speaking visitors who came from Facebook to last year's same profiles but who came from the newsletter ?
- luxurytent 6y ago
  GoatCounter and others use non-identifiable hashes to track a unique visit, but they only retain that hash for some time[1]. I think in your example, you'd have to use a solution that uniquely identifies a session and indeed, keeps track of it. [1] https://github.com/zgoat/goatcounter/blob/master/docs/sessions.markdown#goatcounters-solution https://github.com/zgoat/goatcounter/blob/master/docs/sessio...
  - Carpetsmoker 6y ago
    I'm not sure that's even necessary for the parent's requirement? You should be able to get the parent's data out of it since the location and campaign are stored; there just isn't a UI for it.
    - johnchristopher 6y ago
      > there just isn't a UI for it Exactly, the marketing/communication/sales people need those bits in a UI, they need tools to analyze those bits easily, mark them for specific analyzes and to share reports.
      - Carpetsmoker 6y ago
        Sure, and there's definitely value in that. I can't speak for other products, but GoatCounter intentionally doesn't try to solve every possible analytics use case. Adding advanced features frequently comes with the trade-off that it makes things harder for people who don't use those features, so by limiting the scope it gets easier/better for some, and worse/harder for others. I found this is the case for software in general. There certainly should be an UI to quickly compare how the current quarter is doing compared to the last quarter (I wrote some code for that last week, but not done yet), but doing stuff like adding extra search parameters like "from mobile" and "from Netherlands" seem awfully specific to me. While certainly useful for some, most people are not hard-core marketers and adding stuff like that is just "noise" for a lot of people which makes stuff harder to use for them. Matomo can do all of that, and I also found it hard to use. Or, tl;dr: Matomo is already good at being Matomo, let's do something different :-) Related comment from a while ago: https://lobste.rs/s/e8zgt0/github_is_now_free_for_teams#c_r4vyje https://lobste.rs/s/e8zgt0/github_is_now_free_for_teams#c_r4...
wldlyinaccurate 6y ago
I'm not associated with them at all, but Fathom are worth looking into if you want an analytics platform that respects user privacy: https://usefathom.com/ https://usefathom.com/
- JackWritesCode 6y ago
  Hey, thanks for the shoutout. We're a great option for so many reasons but here are a few recent things" * We allow our users to bypass ad-blockers and exclude themselves from tracking their own page views (https://usefathom.com/blog/custom-domains-embed-code https://usefathom.com/blog/custom-domains-embed-code) * We are both full time on Fathom, bootstrapped (we've turned down millions in VC) and profitable (https://usefathom.com/blog/quit https://usefathom.com/blog/quit) * We are adding in unlimited uptime monitoring on Friday (SMS, Telegram, Slack and email notifications) * We run on auto-scaling infrastructure and are used by individuals, small businesses, governments and some of the biggest companies in the world
  - saagarjha 6y ago
    Without launching into moral judgement, I’d just like to mention that people who block your analytics code from loading normally are often sending you a pretty strong signal that they would not like to be tracked. Do you have something that might take this into account?
    - JackWritesCode 6y ago
      We don’t share that stance. Most people install ad-blockers to block dodgy advertisers, Facebook and other privacy-invasive websites.
      - gxnxcxcx 6y ago
        Does Fathom honor DNT requests from visitors?
        6y ago
zubspace 6y ago
I like GoAccess for following reasons: * No javascript. * Does not need a database. * Realtime stats in html format. I use it in my blog, but also believe, that the numbers are completely inflated. I don't trust them. This has been discussed on Github a few times [1], so don't expect accurate numbers (yet). It's also hard to see what's going on recently on you server, because you only get totals. I'd love if I could change the time interval of the shown html stats. I like the way GoAccess is going though and I hope it will improve. [1] https://github.com/allinurl/goaccess/issues/964#issuecomment-350607068 https://github.com/allinurl/goaccess/issues/964#issuecomment...
- darekkay 6y ago
  Interesting, I've switched from Google Analytics to GoAccess ~5 years ago. I let both analytics run for a month and compared the results. The relative numbers were very similar (so I get the same information about what blog posts are most popular), but the absolute numbers were in fact lower for GoAccess. It might be because tech blog visitors are using AdBlockers more often (and hence block GA). > I'd love if I could change the time interval of the shown html stats. GoAccess displays the data that you pass it. So while it doesn't have any date filter option (at least the last time I've checked), you can just filter your logs beforehand. There's even a more simple solution that I'm using: Set the logrotate to a specific time frame (e.g. weekly), so you can pass "access.log" to GoAccess to only get the latest stats. You can still pass "access.log*" to get ALL stats at the same time.
  - zubspace 6y ago
    About the numbers: thanks for your input. I guess, you get more accurate results the more visitors you have. On my small sized blog I doubt the numbers and think, that GoAccess does not filter out some bots. You can try to determine them and filter them out, but well, that takes some time. However, even if the numbers may not be accurate, you still see overall trends, which is valuable. And thanks for the log-rotation trick. I will definitely make use of it.
    - sdoering 6y ago
tobilg 6y ago
If you run on AWS and want cheap and privacy-focussed website analytics, you might also look at https://ownstats.cloud/ https://ownstats.cloud/
- wharfjumper 6y ago
  Thanks. That fits our use-case. Now just need to figure out how to do the visualizations...
slykar 6y ago
The thing I'm interested in when using Google Analytics is tracking user path to see the bounce rate in a checkout process for example. You can calculate conversion rates for different user segments. People who just want to see "how many visits I got" don't benefit from GA. Developers often miss the point of GA, because they do not work in sales.
- sdoering 6y ago
  Bounce Rate in the checkout funnel? Do you mean the "Exit Rate" at specific steps in the funnel? Just to clarify, because as far as I know from some years in the industry, in every tool I got to know a "Bounce" is defined as a single tracking hit from a user id without further measured activity. Meaning after the first measured page view (the entry to the site) leaving the site couldn't be called a bounce. None the less: If one is interested in a user privacy aware solution to replace GA with a focus on funnels, eCommerce, and such I would probably (up to a certain scale of traffic) recommend to take a look at Matomo (former Piwik). Can be run on your own server, has a lot of the basic functions of GA, a great API, can be used to do goal/path analysis as well as marketing performance reporting. If a company/site reaches a certain scale I would probably recommend to use a paid solution like Adobe Analytics (or paid GA), after having done an evaluation into the real needs of said company/site.
- nicbou 6y ago
  I run a fairly simple website supported by donations and affiliate income. Even then, Analytics offers more than view counts. Here are a few questions it can answer for me: - Is supporting IE11 financially justified? - Which articles generate the most income, and why? - Which components are useful, and which are just noise? - Where are my visitors from? (it affects how I can help them)
  - Nextgrid 6y ago
    > Is supporting IE11 financially justified? Server logs can tell you that. > Which articles generate the most income, and why? Your affiliate partner will tell you which links bring what income assuming you're using unique links. Otherwise the server logs will give you a good page view count. Sure, it'll include bots, but bots that have no interest in the content would tend to be randomly distributed across all articles and thus not skew the results too much. > Which components are useful, and which are just noise? Not sure what you mean by components? If you're talking about different links or forms (like a newsletter) then the forms would also generate log entries on the server and can be counted there. > Where are my visitors from? Server logs and GeoIP. I see nothing here that justifies ratting out your users to Google just to save a tiny bit of effort.
markosaric 6y ago
Nice work! Every time I see the "de-Google-ing" posts, there are people in comments saying but can it do this or can it integrate with that Google product. I'm working on a Google Analytics alternative myself [1] and we make it clear that it is not meant as a clone or a full blown replacement of Google Analytics. Some people are fine running GA and are happy to integrate with Google Ads and the rest of the Google ecosystem. On the other hand, some would prefer to focus more on privacy of their visitors or on not having to get cookie / GDPR consent or on having a faster loading website or support a more independent web etc. And alternative solutions to Google products such as these are more meant for those use cases. [1]: https://plausible.io/ https://plausible.io/
darekkay 6y ago
Similar HN Posts from the past: * https://news.ycombinator.com/item?id=22813168 https://news.ycombinator.com/item?id=22813168 * https://news.ycombinator.com/item?id=21890027 https://news.ycombinator.com/item?id=21890027 * https://news.ycombinator.com/item?id=19883876 https://news.ycombinator.com/item?id=19883876 * https://news.ycombinator.com/item?id=18810035 https://news.ycombinator.com/item?id=18810035
traceroute66 6y ago
I was going to say "what about Matomo". But then Matomo are still lurking around in the dark ages of Python 2 and are not worthy of recommendation until they pull their finger out and sort out their analytics scripts that rely on that obsolete thing (https://github.com/matomo-org/matomo-log-analytics/pull/242 https://github.com/matomo-org/matomo-log-analytics/pull/242, https://github.com/matomo-org/matomo-log-analytics/issues/3 https://github.com/matomo-org/matomo-log-analytics/issues/3 etc. etc.)
- sdoering 6y ago
  I am totally not a friend of anyone still using py2.x (esp. as it has EOLed [1]. Non the less one doesn't need to use log analysis and py2.X if you want to use Matomo as a GA replacement in the frontend. So yeah - they should fix the log analysis part with py2 -> py3. But that doesn't invalidate the whole tool imho. [1]: https://www.python.org/dev/peps/pep-0373/ https://www.python.org/dev/peps/pep-0373/
AdriaanvRossum 6y ago
Thanks for the mention of Simple Analytics [1] Bartek. We do love to be a paid service while we know it doesn’t suit everybody’s need. This way we don’t need to find any other way of making money (with the data of our customers). [1] https://simpleanalytics.com https://simpleanalytics.com
spossy 6y ago
https://statcounter.com/ https://statcounter.com/ have been around forever and offer a good alternative as well.
AndrewStephens 6y ago
I like this blog post and I support anyone who removes a third party tracker from their site. There are companies that live and die by analytics and demographics but your personal blog doesn't need the information that GAnalytics sucks up. GoAccess (suggested in a article) is a fine choice, although I found it did not do a good job of filtering out bot hits. For most people this might not be a big deal but it annoyed me. In the end I just wrote[0] a simple hit counter that triggers off a js beacon. [0] https://github.com/andrewstephens75/visitlog https://github.com/andrewstephens75/visitlog
87zuhjkas 6y ago
There's also Cloudflare Analytics, no js tracking or serverside logging required in case you are behind CF.
monus 6y ago
I’m not associated with Countly ( https://count.ly/ https://count.ly/ ), but I heard good things from my friends using it. It’s open source and makes money with enterprise edition.
fareesh 6y ago
Has anyone had the experience of seeing over-reporting of some metrics in Google Analytics as compared to an internal tracking system? Is this data generally seen to be 100% reliable?
sputr 6y ago
Annoyed with cookie consent on your page? Easy: use a 1st party self-hosted tracker (matomo/piwik), anomnize last digits of IP, respect DNT and provide an optout (matomo provides an embeddable widget) on your privacy policy page. And bam, no legal need for cookie consent OR notification! No popup at all! And you still get perfectly usable statistics for most applications.
- kevingrahl 6y ago
  I’m no expert here but wouldn’t that go against the E-privacy Directive 2009/136/EC, according to which you must: Receive users’ consent before you use any cookies except strictly necessary cookies?
  - JeanMarcS 6y ago
    It seems that there is currently no clear answer to this [0]. Couldn’t find the forum post it’s referring to. [0]: https://github.com/matomo-org/matomo/issues/15425#issuecomment-617790122 https://github.com/matomo-org/matomo/issues/15425#issuecomme...

203 points by luxurytent 6y ago

GoAccess is really cool. Matomo is free and self-hosted if you want more of the features of Google Analytics like tracking time on page.

XCSme 6y ago
I am also working on a similar product to Matomo, it's not free but it also provides some of the Matomo's premium features for a much cheaper price: https://usertrack.net https://usertrack.net

gpanders 6y ago

I noticed this post mentions GoatCounter but says that it doesn't meet his needs because it isn't self-hosted. It is not by default, but you certainly can self host it, and the author has been quite open about that being a viable path for people interested in doing so. I self host GoatCounter myself and it works very well.

Carpetsmoker 6y ago
It looks like I have some work to do on clarifying the homepage here, hah.
luxurytent 6y ago
Hey, yeah. This is my bad! I didn’t dig deep into GoatCounter but on second glance I can see the link to source. I amended my article to clarify what I know (not much)
galfarragem 6y ago
+1 to Goatcounter. Not affiliated with it, just an happy (free) user on a low traffic/hobby project. Simple and no BS. More that enough if you just want to know the big picture and care about privacy. The data I get is this (my data is public but it’s up to you): https://slowernews.goatcounter.com/ https://slowernews.goatcounter.com/
owenshen24 6y ago
Also a happy donating user for the free plan. (No affiliation, I just like the project.)

johnghanks 6y ago

So... more work, slower, and fewer features than GA.

llukas 6y ago
Reduced surveillance sounds like a welcome feature.
- johnghanks 6y ago
  Here's a tip: you and the users of your site are being tracked anyways.
  - nicbou 6y ago
    Yes, but the data is used by you and only you. It's not shared with Google. It's not merged with data from other sites and used against your users. The collected data is under your control, and it's used solely for its intended purpose.
b20000 6y ago
google is only interested in selling you ads. They don't care whether you actually get anything out of their analytics product. If it's free, you ARE the product.
- johnghanks 6y ago
  And? GA is a good product.
  - b20000 6y ago
    it is not, it only gives you the data in a way that is convenient for them to sell you ads. it does not give you the data in a way that you want or is best for your business.
- tjbiddle 6y ago
  Correction: If it's free, your users are the product. And an addition: If you're enterprise and need Google 360, which is very much not free ($100k+/year I believe), not only are you paying - but your customers are still the product!

girzel 6y ago

I feel like all I need is a good script to filter out bots and spiders. I could take the rest from there.

luxurytent 6y ago
This is an interesting thought. I wonder if I could pre process my log files to get rid of this noise ahead of piping to a log analyzer.
- girzel 6y ago
  Or even a logging module in the web server itself...
- Carpetsmoker 6y ago
  I worked on a library to do this for my own analytics thing[1]; adding a CLI so you can filter bots from logfiles isn't too hard: just need to parse the log lines to a http.Request{} with the correct fields filled in (mostly User-Agent, but also looks at IP address). One of the signals is from various JS properties though (navigator.webdriver, window._phantom), so you'll miss out on that. I have some other ideas as well. Either way, it's better than nothing? https://github.com/zgoat/isbot https://github.com/zgoat/isbot
  - luxurytent 6y ago
    This is cool! I've bookmarked it for future review. Thanks for sharing
- nicbou 6y ago
  You could do this with nginx for sure. I don't know what the impact on performance would be.

esperent 6y ago

One possible benefit that I don't see discussed much in this context is bypassing ad blockers. I run a tech focused website and Google analytics registers about 10k visits/month. I figure that a good chunk of my visitors have ublock and so they don't show up. Presumably, alternative analytics or self hosted analytics are not blocked, so I'd get more accurate stats. Is this a correct assumption?

dmkii 6y ago
It’s true, but GA also does a pretty good job at filtering out bot traffic for example, so it depends on what your definition of more accurate is. There are also ways to send GA hits through a sub domain to make it look self hosted and bypass ad blockers.
- markdown 6y ago
  > It’s true, but GA also does a pretty good job at filtering out bot traffic for example Surely you jest. I had to quit using GA because the tech behemoth with all it's tens of thousands of software and algo experts couldn't figure out how to filter out referrer spam from their analytics. Such a simple problem you'd think they'd solve it in a day. I gave them 2yrs and they still couldn't solve that problem so I left.
  - kevin_thibedeau 6y ago
    Parent speaks the truth. I quit because of the garbage referers coming out of Russia that Google would do nothing about.
    - esperent 6y ago
      How do you know they were garbage?
      - markdown 6y ago
        https://www.google.com/search?q=%22buttons-for-websites%22+referrer+spam https://www.google.com/search?q=%22buttons-for-websites%22+r...

chatmasta 6y ago

GoAccess is cool but it won't help you much if you have a static website or if you serve the majority of requests from a CDN. In those cases, Matomo (previously piwik) is a good solution for client-side JS-based analytics similar to Google Analytics.

PenguinCoder 6y ago
How does Matomo work better with a static site, than GoAccess? GoAccess reads the server logs and creates metrics. Matomo requires javascript; that isn't 'static website'.
- Carpetsmoker 6y ago
  There are no server logs on GitHub pages, Netlify, etc. Analysing server logs also makes it harder to filter bots, and won't include some useful metrics like screen size.
- doh 6y ago
  GoAccess reads metrics of a server you have access to. So if you are distributing from a CDN or for instance GitHub or some other hosted solution, then you have no logs to read from. Also usage of javascript doesn't make one page non-static. Static is usually referred to a page that is rendered in full before a request comes in so the content does really change based on the request. More simply, content doesn't change dynamically based on the request. Having javascript to generate metrics doesn't make the page dynamic.
sradman 6y ago
I’m not familiar with GoAccess or Matomo but after some Googling it looks like Matomo requires self-hosting it’s PHP/MySQL server. If you are using a CDN then I’m assuming that multiple access.log files need to be aggregated at some point unless the CDN has a service that automates the process. Logstash and the full ELK Stack (or alternative) seem to be required when multiple heterogeneous servers are involved in serving content. Browser-side JavaScript analytics seems to avoid the DevOps surrounding the ELK Stack. GoAccess seems like a minimalist solution when you control a single httpd-style server and can run a local daemon to process the access.log file. As a side note, it looks like you need an Enterprise account to use Cloudflare’s Logpush Service or Logpull API. Amazon’s CloudFront has an advantage here. Avoiding Google Analytics is non-trivial for the lazy and/or price conscious.

celsoazevedo 6y ago

Something like Matomo running on a cheap VPS (less than $5/month) is a good Google Analytics replacement for small projects. It's cheaper than Simple Analytics (mentioned in the article) and no data is sent to a service operated by a 3rd party because it's self-hosted.

XCSme 6y ago
I am also working on a similar product to Matomo, it also provides some of the Matomo's premium features for a much cheaper price: https://usertrack.net https://usertrack.net

epoch_100 6y ago

Props to the author for ditching Google. I just want to point out that there are a LOT of non-Google analytics options out there; leaving Google Analytics really isn’t too hard if you’re willing to take the leap. Plug: those who want a modern client-side analytics tool that’s free, self hosted, and open source might consider Shynet [0]. (Disclosure: I maintain it.) It’s a bit simpler/cleaner than Matomo, but exists in the same category. [0] https://github.com/milesmcc/shynet https://github.com/milesmcc/shynet

tjbiddle 6y ago
Google Ads still dominates though, and if you're doing paid advertising - you're not just shooting yourself in the foot, but you're lopping off a limb - or two. I wish it wasn't the case. Any non-Google analytics options integrate well with Google Ads, especially for retargeting?
- sdoering 6y ago
  For retargeting you would probably have to have the DoubleClick code on your site nonetheless. But at least serving European users you would only be able to activate it after a clear opt in via a consent manager. Using marketing parameters (like utm-... in GA or Matomo) or the likes in any other tool still gives you clear marketing performance metrics in your analytics solution of choice, though. And a lot of "pure" tracking tools would - at least currently - not fall under the opt in rules of thew GDPR. But the moment you link them with your advertising profiles (like in GA with the DoubleClick integration) your whole tracking setup also becomes opt in. So you would probably loose at least some share of your traffic in analytics and not be able to clearly do marketing analysis any more. At least not as exact as before. I had clients loosing 80% of their traffic stats in analytics after shifting to opt in.
- knicholes 6y ago
  Take a look at Adobe's Experience Platform. Adobe Analytics is great. We take the experience events and profile events and build a unified profile upon which you can build segments used for targeting in Adobe's Advertising Cloud, Adobe Campaign, Adobe Target, and Adobe Experience Manager, and probably more... The Advertising Cloud is integrated with many third-party advertisers. I'm just a software engineer, though. We probably demand an extremely high price.

vladoh 6y ago

Very cool project! It is good that more people are trying to go Google-free. Your project does server side analytics, though, while Google Analytics is client side. Both different advantages and disadvantages. Server side analytics advantages: - No overhead on your website, because no external resources are loaded - Not affected by content blockers - Can track resources other than web pages, like images or redirects - Show resources that could not be found Client side analytics advantages: - Can capture more data about the user, like device size, browser and OS - Better track user behavior, like time spent on each page - Provide real-time feedback about visitors - Usually easier to setup (3/8) I've been playing around with different analytics options on my blog recently - Netlify on the server and Plausible Analytics on the client side. The results from 7 days comparison turned to be quite different [0]. The server side tool shows much more views across all pages in my blog. I think this is because the pages are accessed many times from automated systems that don't run JavaScript and are therefore not counted on the client side. This happens when you post a link on Twitter or WhatsApp for example, because their websites or clients then make a request to fetch the metadata of the page and display a card preview. Therefore, it counts on the server, but not on the client side. A perfect example is the draft article, which I only sent per WhatsApp to my wife for proof reading. The difference there is just 2 views (I guess one preview each on my and on her phone). I sent all other links to several people and posted them on Twitter and Reddit. You can also somewhat see this effect in the referrers [1]. The server side tool counted many hits from a RSS reader and Google Translate (my parents in law reading my blog in Bulgarian :D). The preview requests from Twitter and WhatsApp are probably counted as direct traffic. I think client side analytics more realistically count the number of people reading my blog (even if not many). However, there are cases that can be captured only on the server side (for example Google Translate). So, the best way would be to use both or maybe even make a service that combines client and server side - I think there is a lot of potential there! [0] https://pbs.twimg.com/media/EY-IqtdXsAAJvu9?format=jpg&name=medium https://pbs.twimg.com/media/EY-IqtdXsAAJvu9?format=jpg&name=... [1] https://pbs.twimg.com/media/EY-J0c8WoAEwn2J?format=jpg&name=medium https://pbs.twimg.com/media/EY-J0c8WoAEwn2J?format=jpg&name=...

akdas 6y ago
There's one more advantage to client side analytics, though maybe you meant to cover that under "Usually easier to setup": it's the only option if you host your site on a platform like Github pages. The problem is, you don't have access to server-side logs on these platforms.

johnchristopher 6y ago

Nice. Can it be used to compare this year's first quarter mobile usage for dutch speaking visitors who came from Facebook to last year's same profiles but who came from the newsletter ?

luxurytent 6y ago
GoatCounter and others use non-identifiable hashes to track a unique visit, but they only retain that hash for some time[1]. I think in your example, you'd have to use a solution that uniquely identifies a session and indeed, keeps track of it. [1] https://github.com/zgoat/goatcounter/blob/master/docs/sessions.markdown#goatcounters-solution https://github.com/zgoat/goatcounter/blob/master/docs/sessio...
- Carpetsmoker 6y ago
  I'm not sure that's even necessary for the parent's requirement? You should be able to get the parent's data out of it since the location and campaign are stored; there just isn't a UI for it.
  - johnchristopher 6y ago
    > there just isn't a UI for it Exactly, the marketing/communication/sales people need those bits in a UI, they need tools to analyze those bits easily, mark them for specific analyzes and to share reports.
    - Carpetsmoker 6y ago
      Sure, and there's definitely value in that. I can't speak for other products, but GoatCounter intentionally doesn't try to solve every possible analytics use case. Adding advanced features frequently comes with the trade-off that it makes things harder for people who don't use those features, so by limiting the scope it gets easier/better for some, and worse/harder for others. I found this is the case for software in general. There certainly should be an UI to quickly compare how the current quarter is doing compared to the last quarter (I wrote some code for that last week, but not done yet), but doing stuff like adding extra search parameters like "from mobile" and "from Netherlands" seem awfully specific to me. While certainly useful for some, most people are not hard-core marketers and adding stuff like that is just "noise" for a lot of people which makes stuff harder to use for them. Matomo can do all of that, and I also found it hard to use. Or, tl;dr: Matomo is already good at being Matomo, let's do something different :-) Related comment from a while ago: https://lobste.rs/s/e8zgt0/github_is_now_free_for_teams#c_r4vyje https://lobste.rs/s/e8zgt0/github_is_now_free_for_teams#c_r4...

wldlyinaccurate 6y ago

I'm not associated with them at all, but Fathom are worth looking into if you want an analytics platform that respects user privacy: https://usefathom.com/ https://usefathom.com/

JackWritesCode 6y ago
Hey, thanks for the shoutout. We're a great option for so many reasons but here are a few recent things" * We allow our users to bypass ad-blockers and exclude themselves from tracking their own page views (https://usefathom.com/blog/custom-domains-embed-code https://usefathom.com/blog/custom-domains-embed-code) * We are both full time on Fathom, bootstrapped (we've turned down millions in VC) and profitable (https://usefathom.com/blog/quit https://usefathom.com/blog/quit) * We are adding in unlimited uptime monitoring on Friday (SMS, Telegram, Slack and email notifications) * We run on auto-scaling infrastructure and are used by individuals, small businesses, governments and some of the biggest companies in the world
- saagarjha 6y ago
  Without launching into moral judgement, I’d just like to mention that people who block your analytics code from loading normally are often sending you a pretty strong signal that they would not like to be tracked. Do you have something that might take this into account?
  - JackWritesCode 6y ago
    We don’t share that stance. Most people install ad-blockers to block dodgy advertisers, Facebook and other privacy-invasive websites.
    - gxnxcxcx 6y ago
      Does Fathom honor DNT requests from visitors?
      - 6y ago

zubspace 6y ago

I like GoAccess for following reasons: * No javascript. * Does not need a database. * Realtime stats in html format. I use it in my blog, but also believe, that the numbers are completely inflated. I don't trust them. This has been discussed on Github a few times [1], so don't expect accurate numbers (yet). It's also hard to see what's going on recently on you server, because you only get totals. I'd love if I could change the time interval of the shown html stats. I like the way GoAccess is going though and I hope it will improve. [1] https://github.com/allinurl/goaccess/issues/964#issuecomment-350607068 https://github.com/allinurl/goaccess/issues/964#issuecomment...

darekkay 6y ago
Interesting, I've switched from Google Analytics to GoAccess ~5 years ago. I let both analytics run for a month and compared the results. The relative numbers were very similar (so I get the same information about what blog posts are most popular), but the absolute numbers were in fact lower for GoAccess. It might be because tech blog visitors are using AdBlockers more often (and hence block GA). > I'd love if I could change the time interval of the shown html stats. GoAccess displays the data that you pass it. So while it doesn't have any date filter option (at least the last time I've checked), you can just filter your logs beforehand. There's even a more simple solution that I'm using: Set the logrotate to a specific time frame (e.g. weekly), so you can pass "access.log" to GoAccess to only get the latest stats. You can still pass "access.log*" to get ALL stats at the same time.
- zubspace 6y ago
  About the numbers: thanks for your input. I guess, you get more accurate results the more visitors you have. On my small sized blog I doubt the numbers and think, that GoAccess does not filter out some bots. You can try to determine them and filter them out, but well, that takes some time. However, even if the numbers may not be accurate, you still see overall trends, which is valuable. And thanks for the log-rotation trick. I will definitely make use of it.
  - sdoering 6y ago

tobilg 6y ago

If you run on AWS and want cheap and privacy-focussed website analytics, you might also look at https://ownstats.cloud/ https://ownstats.cloud/

wharfjumper 6y ago
Thanks. That fits our use-case. Now just need to figure out how to do the visualizations...

slykar 6y ago

The thing I'm interested in when using Google Analytics is tracking user path to see the bounce rate in a checkout process for example. You can calculate conversion rates for different user segments. People who just want to see "how many visits I got" don't benefit from GA. Developers often miss the point of GA, because they do not work in sales.

sdoering 6y ago
Bounce Rate in the checkout funnel? Do you mean the "Exit Rate" at specific steps in the funnel? Just to clarify, because as far as I know from some years in the industry, in every tool I got to know a "Bounce" is defined as a single tracking hit from a user id without further measured activity. Meaning after the first measured page view (the entry to the site) leaving the site couldn't be called a bounce. None the less: If one is interested in a user privacy aware solution to replace GA with a focus on funnels, eCommerce, and such I would probably (up to a certain scale of traffic) recommend to take a look at Matomo (former Piwik). Can be run on your own server, has a lot of the basic functions of GA, a great API, can be used to do goal/path analysis as well as marketing performance reporting. If a company/site reaches a certain scale I would probably recommend to use a paid solution like Adobe Analytics (or paid GA), after having done an evaluation into the real needs of said company/site.
nicbou 6y ago
I run a fairly simple website supported by donations and affiliate income. Even then, Analytics offers more than view counts. Here are a few questions it can answer for me: - Is supporting IE11 financially justified? - Which articles generate the most income, and why? - Which components are useful, and which are just noise? - Where are my visitors from? (it affects how I can help them)
- Nextgrid 6y ago
  > Is supporting IE11 financially justified? Server logs can tell you that. > Which articles generate the most income, and why? Your affiliate partner will tell you which links bring what income assuming you're using unique links. Otherwise the server logs will give you a good page view count. Sure, it'll include bots, but bots that have no interest in the content would tend to be randomly distributed across all articles and thus not skew the results too much. > Which components are useful, and which are just noise? Not sure what you mean by components? If you're talking about different links or forms (like a newsletter) then the forms would also generate log entries on the server and can be counted there. > Where are my visitors from? Server logs and GeoIP. I see nothing here that justifies ratting out your users to Google just to save a tiny bit of effort.

markosaric 6y ago

Nice work! Every time I see the "de-Google-ing" posts, there are people in comments saying but can it do this or can it integrate with that Google product. I'm working on a Google Analytics alternative myself [1] and we make it clear that it is not meant as a clone or a full blown replacement of Google Analytics. Some people are fine running GA and are happy to integrate with Google Ads and the rest of the Google ecosystem. On the other hand, some would prefer to focus more on privacy of their visitors or on not having to get cookie / GDPR consent or on having a faster loading website or support a more independent web etc. And alternative solutions to Google products such as these are more meant for those use cases. [1]: https://plausible.io/ https://plausible.io/

darekkay 6y ago

Similar HN Posts from the past: * https://news.ycombinator.com/item?id=22813168 https://news.ycombinator.com/item?id=22813168 * https://news.ycombinator.com/item?id=21890027 https://news.ycombinator.com/item?id=21890027 * https://news.ycombinator.com/item?id=19883876 https://news.ycombinator.com/item?id=19883876 * https://news.ycombinator.com/item?id=18810035 https://news.ycombinator.com/item?id=18810035

traceroute66 6y ago

I was going to say "what about Matomo". But then Matomo are still lurking around in the dark ages of Python 2 and are not worthy of recommendation until they pull their finger out and sort out their analytics scripts that rely on that obsolete thing (https://github.com/matomo-org/matomo-log-analytics/pull/242 https://github.com/matomo-org/matomo-log-analytics/pull/242, https://github.com/matomo-org/matomo-log-analytics/issues/3 https://github.com/matomo-org/matomo-log-analytics/issues/3 etc. etc.)

sdoering 6y ago
I am totally not a friend of anyone still using py2.x (esp. as it has EOLed [1]. Non the less one doesn't need to use log analysis and py2.X if you want to use Matomo as a GA replacement in the frontend. So yeah - they should fix the log analysis part with py2 -> py3. But that doesn't invalidate the whole tool imho. [1]: https://www.python.org/dev/peps/pep-0373/ https://www.python.org/dev/peps/pep-0373/

AdriaanvRossum 6y ago

Thanks for the mention of Simple Analytics [1] Bartek. We do love to be a paid service while we know it doesn’t suit everybody’s need. This way we don’t need to find any other way of making money (with the data of our customers). [1] https://simpleanalytics.com https://simpleanalytics.com

spossy 6y ago

https://statcounter.com/ https://statcounter.com/ have been around forever and offer a good alternative as well.

AndrewStephens 6y ago

I like this blog post and I support anyone who removes a third party tracker from their site. There are companies that live and die by analytics and demographics but your personal blog doesn't need the information that GAnalytics sucks up. GoAccess (suggested in a article) is a fine choice, although I found it did not do a good job of filtering out bot hits. For most people this might not be a big deal but it annoyed me. In the end I just wrote[0] a simple hit counter that triggers off a js beacon. [0] https://github.com/andrewstephens75/visitlog https://github.com/andrewstephens75/visitlog

87zuhjkas 6y ago

There's also Cloudflare Analytics, no js tracking or serverside logging required in case you are behind CF.

monus 6y ago

I’m not associated with Countly ( https://count.ly/ https://count.ly/ ), but I heard good things from my friends using it. It’s open source and makes money with enterprise edition.

fareesh 6y ago

Has anyone had the experience of seeing over-reporting of some metrics in Google Analytics as compared to an internal tracking system? Is this data generally seen to be 100% reliable?

sputr 6y ago

Annoyed with cookie consent on your page? Easy: use a 1st party self-hosted tracker (matomo/piwik), anomnize last digits of IP, respect DNT and provide an optout (matomo provides an embeddable widget) on your privacy policy page. And bam, no legal need for cookie consent OR notification! No popup at all! And you still get perfectly usable statistics for most applications.

kevingrahl 6y ago
I’m no expert here but wouldn’t that go against the E-privacy Directive 2009/136/EC, according to which you must: Receive users’ consent before you use any cookies except strictly necessary cookies?
- JeanMarcS 6y ago
  It seems that there is currently no clear answer to this [0]. Couldn’t find the forum post it’s referring to. [0]: https://github.com/matomo-org/matomo/issues/15425#issuecomment-617790122 https://github.com/matomo-org/matomo/issues/15425#issuecomme...