11 ms·

Migrating from Google Analytics

352 points by gorkemcetin 8y ago

leowoo91 8y ago
Good to see new tooling on analytics. Another alternative is Matomo (Piwik, formerly) https://github.com/matomo-org/matomo https://github.com/matomo-org/matomo
- gorkemcetin 8y ago
  Maybe not very new, as Countly is around for 7 years :-)
  - ksec 8y ago
    But no one has ever heard of it Recommend it around popular Internet forum. ( Although there are various submission but never gained any traction ) One reason could be Countly doesn't offer transparent pricing. People tends to switch off when they see call for quote.
    - gorkemcetin 8y ago
      More than 15000 mobile apps are already using it, Gartner recommends it - but you are right - more traction is always better. Re transparent pricing, it is a complicated process when you sell an enterprise product with many addons, support options, configurations etc, that is why on-prem customizable solutions don't usually have price tags.
- hajderr 8y ago
  Good stuff, using Matomo for my sites too. Haven't heard about Countly so that was interesting to learn too!
firatdemirel 8y ago
That's a great migration for the open-source community. As Thomas mentioned on his post, Google is a cornerstone of "centralized internet" and we should make this kind of migrations to stop them. #2pac
dceddia 8y ago
I always like to hear about new (to me) analytics options. Currently I use GA + Clicky, and I've considered getting rid of GA for a while. One option that looks attractive is Fathom, which can be used either as a SaaS or self-hosted, and is more focused on privacy than most. Clicky: https://clicky.com https://clicky.com Fathom: https://usefathom.com/ https://usefathom.com/
- bjoko 8y ago
  Enterprise-grade email analytics: https://mailspice.com https://mailspice.com
- fiatjaf 8y ago
  I've just rewritten my 2-year-old https://trackingco.de/ https://trackingco.de/.
- ksec 8y ago
  I have always wondered, it is either something is doggy about Clicky, or others are really expensive. Clickly could do a million page view for $20/month, while others, even the simple ones that aren't as feature rich such as Fathom, offer anywhere from $29 / $50, to Matomo which is over $250 per month. What's the Catch?
  - gorkemcetin 8y ago
    Check if your analytics vendor can do drilling and segmentation, and be able to dig into raw data. If that is possible (+which is a hard thing to implement), then costs are way higher.
teej 8y ago
> Which pages were featured and on which social media platforms? This was the “killer feature” of the Google Analytics replacement I built for myself a few years back. I would grab referrers for social and scrape them to give a more useful report of the source of traffic. Unfortunately this is challenging or impossible for many social platforms due to HTTPS everywhere, the prevalence of outbound link scrubbing, and app-driven embedded web views. I still think it would be a killer feature to know who tweeted you out or which subreddit you are trending on but it’s just not feasible to do based on a website pixel alone.
- ndnxhs 8y ago
  How does https stop this? I understand many websites will redirect before loading a link to clear the referrer data which is a good thing since the referrer header has been abused far too much. I imagine originally its intent was for website owners to see what other websites linked to them but now it gets used to track the users
georgewfraser 8y ago
If you are thinking of migrating away from GA, I highly recommend you move to a data warehouse based solution, where you store each event permanently in a data warehouse like Snowflake or BigQuery. There are two client side pixels that you can self-host: snowplow.js and segment. It’s hard to find instructions for self hosting segment but I’ve made an example at https://github.com/fivetran/self-hosted-analytics-js https://github.com/fivetran/self-hosted-analytics-js The advantage of doing it this way is you preserve the event level data forever and you can write arbitrary SQL against it. You will need a BI tool to visualize; there are several excellent ones with free or cheap tiers for small companies.
- jbrooksuk 8y ago
  On BI solutions, we’ve been using Metabase[1] a lot at work and it’s been fantastic. [1] https://Metabase.com https://Metabase.com
  - paulgrimes1 8y ago
    Same - we’re getting some great insights from Metabase.
- buremba 8y ago
  That's the path many data-driven companies take lately. Most companies start with plug-and-play solutions such as GA or Mixpanel but as they start to dig into their customer data, they either hire data engineers or use ETL solutions to collect their raw event data into a data warehouse and BI tools in order to track their own metrics. That way, you will have more control and be able to ask any question you want. We have been working on a tool that basically connects to your Segment or Snowplow data and run ad-hoc analysis similar to Mixpanel and GA so that you don't need to adopt generic BI tools and write SQL every time you create a new report. I was going to create a Show HN post but since your comment is quite relevant to the topic, I wanted to share the product analytics tools that we have been working on, https://rakam.io https://rakam.io. The announcement blog post is also here: https://blog.rakam.io/update-we-built-rakam-ui-from-scratch-and-made-it-even-better/ https://blog.rakam.io/update-we-built-rakam-ui-from-scratch-... P.S: I also genuinely appreciate the work the people have done at Countly, it's often not easy to use ETL tools to set up your own data pipeline and create your own metrics yourself so they're a great alternative if you don't want to get stuck with GA or third-party SaaS alternatives.
alexnewman 8y ago
Yea, it's crippleware. Anything good is payfor
- the_common_man 8y ago
  What is crippleware? GA?
  - alexnewman 8y ago
    countly
- bpicolo 8y ago
  Google definitely takes money for GA for large sites
darekkay 8y ago
> What posts are popular this week? > How is the site doing compared to last week? If this is really all the information you need, try out a server-side solution like GoAccess [0]. It is a little bit harder to set up than copy&pasting an HTML snippet, but it is well worth it (privacy, performance, etc). [0] https://goaccess.io/ https://goaccess.io/
- danyork 8y ago
  The challenge with any server-side solution is the huge amount of caching that takes place within the web infrastructure. Many sites sit behind CDNs, but even without CDNs, many ISPs and enterprises run their own caching proxy servers on the edge of their networks. Browsers also have their own caching of recent pages. The result of all the caching is that it is possible that very few of the visitors actually connect to the origin server. For that reason, most analytics systems use come kind of client-side tracking code that runs within the client web browser. That way it works regardless of whatever caching happens.
  - darekkay 8y ago
    Server-side analytics is not a silver bullet, but it's great for the quoted use case. While it has the mentioned disadvantages, so do client-side solutions. Many people run adblockers and it's easy - sometimes even a default - to simply block Google Analytics & co. So yes, if you really need exact statistics, you should probably run both client-side and server-side analytics. But for a rough "what's popular?" estimate, there's no need to impose (one more) JS file on your users. I've switched from GA to GoAccess a few years ago (running both for a few months), and while the absolute numbers were off by ~10%, the ratio was almost the same. But then again, I'm just running a blog and not a business.
radium3d 8y ago
I'm curious how these alternative options (countly, matomo, snowplow, etc) handle fake referral traffic compared to how Google analytics handles it.
- georgewfraser 8y ago
  Not well! Your best bet is to * Sync your (snowplow, segment, ...) data to a data warehouse * Try to clean up your data with SQL * Keep running GA so you can compare your numbers to GA
  - radium3d 8y ago
    That may be a helpful open source machine learning project for a self-hosted analytics system perhaps.
  - coderintherye 8y ago
    That's exactly what we do. Snowplow does a good job, but having the comparison points in GA is very helpful.
- tombrossman 8y ago
  Has Google Analytics finally begun blocking referrer spam? I know Piwik/Matomo have been doing this by default for a while: https://matomo.org/blog/2015/05/stopping-referrer-spam/ https://matomo.org/blog/2015/05/stopping-referrer-spam/ The blacklist is on GitHub and is useful for other analytics software, such as GoAccess which is what I use.
- Findus23 8y ago
  Matomo uses a simple community-contributed list [1] of domains that were reported by multiple people to create referrer-spam. Of course you can use the same list for every other software. [1] https://github.com/matomo-org/referrer-spam-blacklist/blob/master/spammers.txt https://github.com/matomo-org/referrer-spam-blacklist/blob/m...
jagthebeetle 8y ago
A small point (although I don't claim it militates against the central impulse of the article), is that serving analytics.js through gtag.js is obviously going to inflate the tag size unnecessarily. The latter is a unified interface for all of Google's various page snippets. Also, though not recommended, there's no reason you can't self-host GA, although practically speaking Google-hosted GA is likelier cached more often and on pretty snappy geolocated servers.
RA_Fisher 8y ago
Lately I've been pondering whether client-side analytic scripts generate truncated data by design? That is, a payload goes to the client but before a response can be sent to some backend server the client terminates (one example being a tab close) before the analytic code finishes. If this is the case my understanding is that it'd result in an unknowable underestimation of http responses. I'm still exploring the space and would be interested to know if others had thought about this?
- lmkg 8y ago
  There is a browser API called Beacon whose purpose is to address this specific edge-case. Google Analytics supports it as an option. https://developer.mozilla.org/en-US/docs/Web/API/Beacon_API https://developer.mozilla.org/en-US/docs/Web/API/Beacon_API
- buremba 8y ago
  Most analytics providers have JS SDKs that persist the data to localStorage. When the user comes back to the webpage, the SDK tries to send the persisted data to the server. It's not an ultimate solution but it usually covers most of the cases where Beacon support is not available.
goodroot 8y ago
Nice post! I have been thinking much about this for my own written works... My strange conclusion was to not capture analytics at all. My loose metric is now: which posts inspire people to email me directly? I have decided I do not want to know what you read, how long you read it, where you came from... Only whether or not you felt something. And data will not tell me this.
- tlavoie 8y ago
  Interesting! I was reading all the comments, wondering if anyone would express the concept that maybe, just maybe, all of this might not be that useful after all. More specifically, how often does all of this really need to be down to the individual person (or IP address) at all? Even if you know that piece of information at an ephemeral level, my own suspicion is that aggregate data should be sufficient for any non-creepy use case. Perhaps one way to phrase the question, how would having personal-level details in the analytics change the actions you might perform based on available data?
blakesterz 8y ago
Anyone else miss Urchin? Google bought them and used it to build Analytics. For me at least, Urchin was vastly superior. I feel like Analytics is built for sites that run Google Ads to sell more ads. Urchin was for people who ran websites who wanted to learn more about their site and readers.
- toomuchtodo 8y ago
  I loved the hosted version of Urchin that processed web server logs locally (or would grab them remotely with ssh/sftp/ftp). A shame that product was deprecated post-acquisition.
- whalesalad 8y ago
  My favorite analytics tool was Mint from Shaun Inman.
- caro_douglos 8y ago
  I always reminisce about it when I see that urchin.js snippet.. It was *superior. Often times when people ran into a wall within GA they would flock to Urchin and its fairly reasonable price. https://www.cardinalpath.com/what-is-google-urchin-the-difference-between-urchin-and-google-analytics/ https://www.cardinalpath.com/what-is-google-urchin-the-diffe...
pablo-massa 8y ago
I have the same concerns with Google Analytics and tried to install Matomo [1] for my personal website [2] few months ago, seems a more robust tool than Countly [3] to me, maybe I'm wrong. But I had an error installing Matomo and got not help in the official forum [4], if someone here can help me I will appreciate it, I'm not a developer (I'm also considering get rid of analytics tools for good, anyone?). Thanks. [1] https://matomo.org/ https://matomo.org/ [2] https://count.ly/ https://count.ly/ [3] https://pablomassa.com/ https://pablomassa.com/ [4] https://forum.matomo.org/t/fatal-error-on-installation/29949 https://forum.matomo.org/t/fatal-error-on-installation/29949
- toastal 8y ago
  This project used to be called Piwik. 10 years later and they still only support MySQL; a lot of people really wanted at least Postgres support.
- chinathrow 8y ago
  From [4] I learn: "I have a Hostgator shared hosting with PHP 5.5." PHP 5.5 is EOL since mid of 2017. Can't you switch to newer version such as 7.2? Matomo itself does not even display the required version according to [1]. Their FAQ talks about 5.3 - which is horribly ancient too. [1] https://matomo.org/docs/installation/ https://matomo.org/docs/installation/
  - Findus23 8y ago
    Hi, I have to agree that one should be using PHP 7.2. It also gives a nice performance boost to Matomo. The required PHP version for Matomo is shown in [1] (5.5.9 or greater) Can you please send me a link to the FAQ page mentioning 5.3 (e.g. to lukas@matomo.org) so they can be updated? [1] https://matomo.org/docs/requirements/ https://matomo.org/docs/requirements/
    - chinathrow 8y ago
      My mistake, I was skimming the docs only but overlooked the specially linked requirements. I would go further though and remove outdated PHP versions from that page and only recommend maintained versions. Additionally, the error described by OP looks like autoloading is broken.
homero 8y ago
Why countly and not piwik?
evantahler 8y ago
Any interest in helping me build a countly alternative?
- chvid 8y ago
  I think there is but what would you make different?
  - justinclift 8y ago
    Not have a 1 liner installation script that's a security nightmare? :) eg, this is the Countly one (specifically run as root): wget -qO- http://c.ly/install | bash If someone manages to break into the c.ly redirection service, or the website/CDN/etc serving that, new users would likely be in for a bad time. And the problem could be very subtle, if it's done by someone clueful. Just for added er.. goodness (/s), the Countly instructions also need people to: Disable SELinux on Red Hat or CentOS if it's enabled. Countly may not work on a server where SELinux is enabled. In order to disable SELinux, run "setenforce 0". sigh
Envision_Envi 8y ago
Waiting For new tools..will see how accurate they work compared to google analytics.
_up 8y ago
Is anyone using Yandex.metrica? It seems to offer a lot of features and is even completely free. But nobody seems to be talking about it. Where is the catch?
pawal 8y ago
Is there a privacy friendly analytics tool that does not set cookies and store data Forever? I don't really care about perfect user analytics, just good enough. Maybe by analysing logs. In the 90's there was s lot of good tools like this, but now everybody has gone cloud. I can't imagine the tools offered today are compatible with GDPR.
- aparks517 8y ago
  I work on https://getmirrorshades.com https://getmirrorshades.com It doesn’t use cookies. Data is aggregated immediately and purged on a tight schedule. May be worth seeing if it is sufficient for your use case.
- dvko 8y ago
  https://GoAccess.io https://GoAccess.io is great for analysing access logs. That said, there are solutions where a cookie allows for greater privacy because it allows you to leave sensitive data on the client versus having to store it on the server. For example, https://usefathom.com/ https://usefathom.com/ is an open-source self-hosted analytics tool that uses a cookie to enhance privacy.
- Findus23 8y ago
  Hi, You can configure Matomo [1] to both not use any cookies [2] and to automatically delete just the raw data or all data that is older than x months. Log Analytics is also possible. If you want something that is far more minimalistic, but also Open Source and self-hostable, you can take a look at [3]. (Not sure about how they use cookies) (Disclaimer: I am part of the Matomo team) [1] https://matomo.org/ https://matomo.org/ [2] https://matomo.org/faq/general/faq_157/ https://matomo.org/faq/general/faq_157/ [3] https://usefathom.com/ https://usefathom.com/
- gorkemcetin 8y ago
  Countly (if you want) can be configured so it doesn't set cookies [1] You can also configure to remove any data older than N days/months. It is also GDPR compliant [2] [1] https://resources.count.ly/docs/countly-sdk-for-web#section-tracking-users-with-javascript-disabled https://resources.count.ly/docs/countly-sdk-for-web#section-... [2] https://resources.count.ly/docs/compliance-hub https://resources.count.ly/docs/compliance-hub
Ylodi 8y ago
Is there any cookieless analytics solution?
- moehm 8y ago
  You can disable Cookies in Matomo. https://matomo.org/faq/general/faq_157/ https://matomo.org/faq/general/faq_157/
- pkz 8y ago
  Matomo can be set up for cookie less tracking. It will of course impact some metrics. More here: https://matomo.org/faq/general/faq_157/ https://matomo.org/faq/general/faq_157/
- gorkemcetin 8y ago
  Yes, most of the analytics platforms out there can work without a cookie, including Countly.
Tsubasachan 8y ago
Considering GA is blocked by every adblocker I wouldn't rely too much on it anyway.
Brosper 8y ago
Only advantage is that it weight less? I don't feel convinced to switch :(
- jstanley 8y ago
  The advantage is that you're not selling out your user's browsing habits to the biggest surveillance company in the world. That your pages are now lighter too is just a bonus.
harianus 8y ago
I think it's great we are moving away from big data collectors and running our own servers. What I usually see is people storing their users' data on servers that are situated in locations where governments can get access to the data on the servers without the owner knowing about it. It's maybe going far in protecting the privacy of your users, but it's something a lot of solutions on the internet don't think about. That's why I moved Simple Analytics' (https://simpleanalytics.io https://simpleanalytics.io) servers to Iceland where the law will forbid peeking in data before actually informing the owner of the server. I encrypted my server so if anything happens I can just turn it off and it will be nearly impossible to get any user data.
- akudha 8y ago
  I encrypted my server Could you please share which Iceland host you are using? Also, what is the process you're following to encrypt your server?
  - harianus 8y ago
    I'm using 1984 [1]. I'm writing a blog post on this soon [2], but in short I use Ubuntu Server LVM [3], unlock my system via Dropbear SSH (a very small SSH client)[4] and did move the entry point of incoming data to another server which will store the incoming data only when the main server is down. This is because I want to keep recording incoming data and my encrypted server can't boot without me entering a password. So in case of a power failure the data of my customers will still be recorded. [1] https://1984.is https://1984.is [2] https://blog.simpleanalytics.io https://blog.simpleanalytics.io [3] https://en.wikipedia.org/wiki/Logical_Volume_Manager_%28Linux%29 https://en.wikipedia.org/wiki/Logical_Volume_Manager_%28Linu... [4] https://matt.ucc.asn.au/dropbear/dropbear.html https://matt.ucc.asn.au/dropbear/dropbear.html
envisionintel 8y ago
A good one..!! Everything has some advantages and some disadvantages.
Sir_Cmpwn 8y ago
I know the average HN user sold their soul a long time ago, but just in case... let me pitch an alternative: don't spy on your users at all. You can live without your masturbatory dashboards and get all of the technical info you need by grepping through your access logs.
johnchristopher 8y ago
Hmm. Most viewed pages, time spent... okay. But GA provide segments that marketers need. How do you get that kind of data without google of the fa pixel ?

352 points by gorkemcetin 8y ago

Good to see new tooling on analytics. Another alternative is Matomo (Piwik, formerly) https://github.com/matomo-org/matomo https://github.com/matomo-org/matomo

gorkemcetin 8y ago
Maybe not very new, as Countly is around for 7 years :-)
- ksec 8y ago
  But no one has ever heard of it Recommend it around popular Internet forum. ( Although there are various submission but never gained any traction ) One reason could be Countly doesn't offer transparent pricing. People tends to switch off when they see call for quote.
  - gorkemcetin 8y ago
    More than 15000 mobile apps are already using it, Gartner recommends it - but you are right - more traction is always better. Re transparent pricing, it is a complicated process when you sell an enterprise product with many addons, support options, configurations etc, that is why on-prem customizable solutions don't usually have price tags.
hajderr 8y ago
Good stuff, using Matomo for my sites too. Haven't heard about Countly so that was interesting to learn too!

firatdemirel 8y ago

That's a great migration for the open-source community. As Thomas mentioned on his post, Google is a cornerstone of "centralized internet" and we should make this kind of migrations to stop them. #2pac

dceddia 8y ago

I always like to hear about new (to me) analytics options. Currently I use GA + Clicky, and I've considered getting rid of GA for a while. One option that looks attractive is Fathom, which can be used either as a SaaS or self-hosted, and is more focused on privacy than most. Clicky: https://clicky.com https://clicky.com Fathom: https://usefathom.com/ https://usefathom.com/

bjoko 8y ago
Enterprise-grade email analytics: https://mailspice.com https://mailspice.com
fiatjaf 8y ago
I've just rewritten my 2-year-old https://trackingco.de/ https://trackingco.de/.
ksec 8y ago
I have always wondered, it is either something is doggy about Clicky, or others are really expensive. Clickly could do a million page view for $20/month, while others, even the simple ones that aren't as feature rich such as Fathom, offer anywhere from $29 / $50, to Matomo which is over $250 per month. What's the Catch?
- gorkemcetin 8y ago
  Check if your analytics vendor can do drilling and segmentation, and be able to dig into raw data. If that is possible (+which is a hard thing to implement), then costs are way higher.

teej 8y ago

> Which pages were featured and on which social media platforms? This was the “killer feature” of the Google Analytics replacement I built for myself a few years back. I would grab referrers for social and scrape them to give a more useful report of the source of traffic. Unfortunately this is challenging or impossible for many social platforms due to HTTPS everywhere, the prevalence of outbound link scrubbing, and app-driven embedded web views. I still think it would be a killer feature to know who tweeted you out or which subreddit you are trending on but it’s just not feasible to do based on a website pixel alone.

ndnxhs 8y ago
How does https stop this? I understand many websites will redirect before loading a link to clear the referrer data which is a good thing since the referrer header has been abused far too much. I imagine originally its intent was for website owners to see what other websites linked to them but now it gets used to track the users

georgewfraser 8y ago

If you are thinking of migrating away from GA, I highly recommend you move to a data warehouse based solution, where you store each event permanently in a data warehouse like Snowflake or BigQuery. There are two client side pixels that you can self-host: snowplow.js and segment. It’s hard to find instructions for self hosting segment but I’ve made an example at https://github.com/fivetran/self-hosted-analytics-js https://github.com/fivetran/self-hosted-analytics-js The advantage of doing it this way is you preserve the event level data forever and you can write arbitrary SQL against it. You will need a BI tool to visualize; there are several excellent ones with free or cheap tiers for small companies.

jbrooksuk 8y ago
On BI solutions, we’ve been using Metabase[1] a lot at work and it’s been fantastic. [1] https://Metabase.com https://Metabase.com
- paulgrimes1 8y ago
  Same - we’re getting some great insights from Metabase.
buremba 8y ago
That's the path many data-driven companies take lately. Most companies start with plug-and-play solutions such as GA or Mixpanel but as they start to dig into their customer data, they either hire data engineers or use ETL solutions to collect their raw event data into a data warehouse and BI tools in order to track their own metrics. That way, you will have more control and be able to ask any question you want. We have been working on a tool that basically connects to your Segment or Snowplow data and run ad-hoc analysis similar to Mixpanel and GA so that you don't need to adopt generic BI tools and write SQL every time you create a new report. I was going to create a Show HN post but since your comment is quite relevant to the topic, I wanted to share the product analytics tools that we have been working on, https://rakam.io https://rakam.io. The announcement blog post is also here: https://blog.rakam.io/update-we-built-rakam-ui-from-scratch-and-made-it-even-better/ https://blog.rakam.io/update-we-built-rakam-ui-from-scratch-... P.S: I also genuinely appreciate the work the people have done at Countly, it's often not easy to use ETL tools to set up your own data pipeline and create your own metrics yourself so they're a great alternative if you don't want to get stuck with GA or third-party SaaS alternatives.

alexnewman 8y ago

Yea, it's crippleware. Anything good is payfor

the_common_man 8y ago
What is crippleware? GA?
- alexnewman 8y ago
  countly
bpicolo 8y ago
Google definitely takes money for GA for large sites

darekkay 8y ago

> What posts are popular this week? > How is the site doing compared to last week? If this is really all the information you need, try out a server-side solution like GoAccess [0]. It is a little bit harder to set up than copy&pasting an HTML snippet, but it is well worth it (privacy, performance, etc). [0] https://goaccess.io/ https://goaccess.io/

danyork 8y ago
The challenge with any server-side solution is the huge amount of caching that takes place within the web infrastructure. Many sites sit behind CDNs, but even without CDNs, many ISPs and enterprises run their own caching proxy servers on the edge of their networks. Browsers also have their own caching of recent pages. The result of all the caching is that it is possible that very few of the visitors actually connect to the origin server. For that reason, most analytics systems use come kind of client-side tracking code that runs within the client web browser. That way it works regardless of whatever caching happens.
- darekkay 8y ago
  Server-side analytics is not a silver bullet, but it's great for the quoted use case. While it has the mentioned disadvantages, so do client-side solutions. Many people run adblockers and it's easy - sometimes even a default - to simply block Google Analytics & co. So yes, if you really need exact statistics, you should probably run both client-side and server-side analytics. But for a rough "what's popular?" estimate, there's no need to impose (one more) JS file on your users. I've switched from GA to GoAccess a few years ago (running both for a few months), and while the absolute numbers were off by ~10%, the ratio was almost the same. But then again, I'm just running a blog and not a business.

radium3d 8y ago

I'm curious how these alternative options (countly, matomo, snowplow, etc) handle fake referral traffic compared to how Google analytics handles it.

georgewfraser 8y ago
Not well! Your best bet is to * Sync your (snowplow, segment, ...) data to a data warehouse * Try to clean up your data with SQL * Keep running GA so you can compare your numbers to GA
- radium3d 8y ago
  That may be a helpful open source machine learning project for a self-hosted analytics system perhaps.
- coderintherye 8y ago
  That's exactly what we do. Snowplow does a good job, but having the comparison points in GA is very helpful.
tombrossman 8y ago
Has Google Analytics finally begun blocking referrer spam? I know Piwik/Matomo have been doing this by default for a while: https://matomo.org/blog/2015/05/stopping-referrer-spam/ https://matomo.org/blog/2015/05/stopping-referrer-spam/ The blacklist is on GitHub and is useful for other analytics software, such as GoAccess which is what I use.
Findus23 8y ago
Matomo uses a simple community-contributed list [1] of domains that were reported by multiple people to create referrer-spam. Of course you can use the same list for every other software. [1] https://github.com/matomo-org/referrer-spam-blacklist/blob/master/spammers.txt https://github.com/matomo-org/referrer-spam-blacklist/blob/m...

jagthebeetle 8y ago

A small point (although I don't claim it militates against the central impulse of the article), is that serving analytics.js through gtag.js is obviously going to inflate the tag size unnecessarily. The latter is a unified interface for all of Google's various page snippets. Also, though not recommended, there's no reason you can't self-host GA, although practically speaking Google-hosted GA is likelier cached more often and on pretty snappy geolocated servers.

RA_Fisher 8y ago

Lately I've been pondering whether client-side analytic scripts generate truncated data by design? That is, a payload goes to the client but before a response can be sent to some backend server the client terminates (one example being a tab close) before the analytic code finishes. If this is the case my understanding is that it'd result in an unknowable underestimation of http responses. I'm still exploring the space and would be interested to know if others had thought about this?

lmkg 8y ago
There is a browser API called Beacon whose purpose is to address this specific edge-case. Google Analytics supports it as an option. https://developer.mozilla.org/en-US/docs/Web/API/Beacon_API https://developer.mozilla.org/en-US/docs/Web/API/Beacon_API
buremba 8y ago
Most analytics providers have JS SDKs that persist the data to localStorage. When the user comes back to the webpage, the SDK tries to send the persisted data to the server. It's not an ultimate solution but it usually covers most of the cases where Beacon support is not available.

goodroot 8y ago

Nice post! I have been thinking much about this for my own written works... My strange conclusion was to not capture analytics at all. My loose metric is now: which posts inspire people to email me directly? I have decided I do not want to know what you read, how long you read it, where you came from... Only whether or not you felt something. And data will not tell me this.

tlavoie 8y ago
Interesting! I was reading all the comments, wondering if anyone would express the concept that maybe, just maybe, all of this might not be that useful after all. More specifically, how often does all of this really need to be down to the individual person (or IP address) at all? Even if you know that piece of information at an ephemeral level, my own suspicion is that aggregate data should be sufficient for any non-creepy use case. Perhaps one way to phrase the question, how would having personal-level details in the analytics change the actions you might perform based on available data?

blakesterz 8y ago

Anyone else miss Urchin? Google bought them and used it to build Analytics. For me at least, Urchin was vastly superior. I feel like Analytics is built for sites that run Google Ads to sell more ads. Urchin was for people who ran websites who wanted to learn more about their site and readers.

toomuchtodo 8y ago
I loved the hosted version of Urchin that processed web server logs locally (or would grab them remotely with ssh/sftp/ftp). A shame that product was deprecated post-acquisition.
whalesalad 8y ago
My favorite analytics tool was Mint from Shaun Inman.
caro_douglos 8y ago
I always reminisce about it when I see that urchin.js snippet.. It was *superior. Often times when people ran into a wall within GA they would flock to Urchin and its fairly reasonable price. https://www.cardinalpath.com/what-is-google-urchin-the-difference-between-urchin-and-google-analytics/ https://www.cardinalpath.com/what-is-google-urchin-the-diffe...

pablo-massa 8y ago

I have the same concerns with Google Analytics and tried to install Matomo [1] for my personal website [2] few months ago, seems a more robust tool than Countly [3] to me, maybe I'm wrong. But I had an error installing Matomo and got not help in the official forum [4], if someone here can help me I will appreciate it, I'm not a developer (I'm also considering get rid of analytics tools for good, anyone?). Thanks. [1] https://matomo.org/ https://matomo.org/ [2] https://count.ly/ https://count.ly/ [3] https://pablomassa.com/ https://pablomassa.com/ [4] https://forum.matomo.org/t/fatal-error-on-installation/29949 https://forum.matomo.org/t/fatal-error-on-installation/29949

toastal 8y ago
This project used to be called Piwik. 10 years later and they still only support MySQL; a lot of people really wanted at least Postgres support.
chinathrow 8y ago
From [4] I learn: "I have a Hostgator shared hosting with PHP 5.5." PHP 5.5 is EOL since mid of 2017. Can't you switch to newer version such as 7.2? Matomo itself does not even display the required version according to [1]. Their FAQ talks about 5.3 - which is horribly ancient too. [1] https://matomo.org/docs/installation/ https://matomo.org/docs/installation/
- Findus23 8y ago
  Hi, I have to agree that one should be using PHP 7.2. It also gives a nice performance boost to Matomo. The required PHP version for Matomo is shown in [1] (5.5.9 or greater) Can you please send me a link to the FAQ page mentioning 5.3 (e.g. to lukas@matomo.org) so they can be updated? [1] https://matomo.org/docs/requirements/ https://matomo.org/docs/requirements/
  - chinathrow 8y ago
    My mistake, I was skimming the docs only but overlooked the specially linked requirements. I would go further though and remove outdated PHP versions from that page and only recommend maintained versions. Additionally, the error described by OP looks like autoloading is broken.

homero 8y ago

Why countly and not piwik?

evantahler 8y ago

Any interest in helping me build a countly alternative?

chvid 8y ago
I think there is but what would you make different?
- justinclift 8y ago
  Not have a 1 liner installation script that's a security nightmare? :) eg, this is the Countly one (specifically run as root): wget -qO- http://c.ly/install | bash If someone manages to break into the c.ly redirection service, or the website/CDN/etc serving that, new users would likely be in for a bad time. And the problem could be very subtle, if it's done by someone clueful. Just for added er.. goodness (/s), the Countly instructions also need people to: Disable SELinux on Red Hat or CentOS if it's enabled. Countly may not work on a server where SELinux is enabled. In order to disable SELinux, run "setenforce 0". sigh

Envision_Envi 8y ago

Waiting For new tools..will see how accurate they work compared to google analytics.

_up 8y ago

Is anyone using Yandex.metrica? It seems to offer a lot of features and is even completely free. But nobody seems to be talking about it. Where is the catch?

pawal 8y ago

Is there a privacy friendly analytics tool that does not set cookies and store data Forever? I don't really care about perfect user analytics, just good enough. Maybe by analysing logs. In the 90's there was s lot of good tools like this, but now everybody has gone cloud. I can't imagine the tools offered today are compatible with GDPR.

aparks517 8y ago
I work on https://getmirrorshades.com https://getmirrorshades.com It doesn’t use cookies. Data is aggregated immediately and purged on a tight schedule. May be worth seeing if it is sufficient for your use case.
dvko 8y ago
https://GoAccess.io https://GoAccess.io is great for analysing access logs. That said, there are solutions where a cookie allows for greater privacy because it allows you to leave sensitive data on the client versus having to store it on the server. For example, https://usefathom.com/ https://usefathom.com/ is an open-source self-hosted analytics tool that uses a cookie to enhance privacy.
Findus23 8y ago
Hi, You can configure Matomo [1] to both not use any cookies [2] and to automatically delete just the raw data or all data that is older than x months. Log Analytics is also possible. If you want something that is far more minimalistic, but also Open Source and self-hostable, you can take a look at [3]. (Not sure about how they use cookies) (Disclaimer: I am part of the Matomo team) [1] https://matomo.org/ https://matomo.org/ [2] https://matomo.org/faq/general/faq_157/ https://matomo.org/faq/general/faq_157/ [3] https://usefathom.com/ https://usefathom.com/
gorkemcetin 8y ago
Countly (if you want) can be configured so it doesn't set cookies [1] You can also configure to remove any data older than N days/months. It is also GDPR compliant [2] [1] https://resources.count.ly/docs/countly-sdk-for-web#section-tracking-users-with-javascript-disabled https://resources.count.ly/docs/countly-sdk-for-web#section-... [2] https://resources.count.ly/docs/compliance-hub https://resources.count.ly/docs/compliance-hub

Ylodi 8y ago

Is there any cookieless analytics solution?

moehm 8y ago
You can disable Cookies in Matomo. https://matomo.org/faq/general/faq_157/ https://matomo.org/faq/general/faq_157/
pkz 8y ago
Matomo can be set up for cookie less tracking. It will of course impact some metrics. More here: https://matomo.org/faq/general/faq_157/ https://matomo.org/faq/general/faq_157/
gorkemcetin 8y ago
Yes, most of the analytics platforms out there can work without a cookie, including Countly.

Tsubasachan 8y ago

Considering GA is blocked by every adblocker I wouldn't rely too much on it anyway.

Brosper 8y ago

Only advantage is that it weight less? I don't feel convinced to switch :(

jstanley 8y ago
The advantage is that you're not selling out your user's browsing habits to the biggest surveillance company in the world. That your pages are now lighter too is just a bonus.

harianus 8y ago

I think it's great we are moving away from big data collectors and running our own servers. What I usually see is people storing their users' data on servers that are situated in locations where governments can get access to the data on the servers without the owner knowing about it. It's maybe going far in protecting the privacy of your users, but it's something a lot of solutions on the internet don't think about. That's why I moved Simple Analytics' (https://simpleanalytics.io https://simpleanalytics.io) servers to Iceland where the law will forbid peeking in data before actually informing the owner of the server. I encrypted my server so if anything happens I can just turn it off and it will be nearly impossible to get any user data.

akudha 8y ago
I encrypted my server Could you please share which Iceland host you are using? Also, what is the process you're following to encrypt your server?
- harianus 8y ago
  I'm using 1984 [1]. I'm writing a blog post on this soon [2], but in short I use Ubuntu Server LVM [3], unlock my system via Dropbear SSH (a very small SSH client)[4] and did move the entry point of incoming data to another server which will store the incoming data only when the main server is down. This is because I want to keep recording incoming data and my encrypted server can't boot without me entering a password. So in case of a power failure the data of my customers will still be recorded. [1] https://1984.is https://1984.is [2] https://blog.simpleanalytics.io https://blog.simpleanalytics.io [3] https://en.wikipedia.org/wiki/Logical_Volume_Manager_%28Linux%29 https://en.wikipedia.org/wiki/Logical_Volume_Manager_%28Linu... [4] https://matt.ucc.asn.au/dropbear/dropbear.html https://matt.ucc.asn.au/dropbear/dropbear.html

envisionintel 8y ago

A good one..!! Everything has some advantages and some disadvantages.

Sir_Cmpwn 8y ago

I know the average HN user sold their soul a long time ago, but just in case... let me pitch an alternative: don't spy on your users at all. You can live without your masturbatory dashboards and get all of the technical info you need by grepping through your access logs.

johnchristopher 8y ago

Hmm. Most viewed pages, time spent... okay. But GA provide segments that marketers need. How do you get that kind of data without google of the fa pixel ?