3 ms·

What kind of server-side analytics are people using today, for personal blogs and things? Projects like GoAccess which eat an nginx log file and output some ana

by joppy 6y ago

What kind of server-side analytics are people using today, for personal blogs and things? Projects like GoAccess which eat an nginx log file and output some analytics seem like a nice middle ground for those of us who want some feedback on how people are using a website, without needing all the bells and whistles of something more like Google Analytics (not to mention the fact that it doesn't need any Javascript loaded or anything). Personally I've found GoAccess pretty good, but the interface a little difficult to use and understand, so I'm looking for projects like it.

PaulRobinson 6y ago
Server side was how it was always done back in the early days of the web, and analog[0] was state of the art. Around 1999/2000 there was a rise of ISPs needing to install reverse proxy caches because the growth of consumer access meant they were getting seriously contended on upstream access. I was working at the time at a UK 0845 white label ISP called Telinco (was behind Connect Free, Totalise, Current Bun and other 0845 ISPs), and to my knowledge we were the first in the UK to install a Netapps cache. It was the moment we realised (by checking the logs to see if it was working), just how much porn our customers were accessing. Those caches blow server side analytics to pieces, because frequently you wouldn't even know the user had hit the page. What server side analytics was useful for is what we'd now call Observability: they gave reasonable Latency, Error Rate and Throughput metrics, which combined with some other system logs might also give you a sense of Saturation. As such, they were not too useful for marketing. Google Analytics was the first product that allowed high fidelity analytics even if reverse proxy caches (and even browser caches), were all over the place. And here we are. In a World where we are tightly surveilled by corporate entities in order to try and get us to click on things. Bit sad really. I'd encourage people to think about what they need these analytics for. If it's marketing, you might just as well using GA: it's the best product out there. We just need to lobby for better regulation (at least GDPR and cookie setting popovers give us choices on that regard now). If you're stroking your ego, consider whether such an invasive technology is worth the price, and if you need those numbers. If you're making sure your infrastructure can handle the traffic, use server side analytics alone. Parse your logs using the huge number of tools out there able to do that in near-realtime, and leave your users' browsers free of tracking cookies and javascript. [0] https://www.web42.com/analog/ https://www.web42.com/analog/
- Nextgrid 6y ago
  Caches are irrelevant now that the world has moved to HTTPS.
  - divbzero 6y ago
    There are no public caches with HTTPS, but there are still private browser and CDN caches to contend with. To ensure your origin server is hit on subsequent HTTPS requests, you would still need to configure response headers for cache validation [1] to be Cache-Control: no-cache instead of Cache-Control: private [1]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ca...
    - Nextgrid 6y ago
      Yes, but those are caches that you control, so when it comes to analytics you would get the logs off them too in order to get accurate metrics.
dig1 6y ago
Webalizer [1] can be alternative. No fancy UI, but gets job done. For anything heavier and serious, ELK [2]. [1] http://www.webalizer.org/ http://www.webalizer.org/ [2] https://www.elastic.co/what-is/elk-stack https://www.elastic.co/what-is/elk-stack
stephenr 6y ago
I've setup GoAccess for a client's site, the problem is it doesn't have a great HA solution. You either ship all your logs to one place (and hope that place doesn't go offline) or ship your logs to multiple places and hope both destinations are in sync. We've opted for #2 right now (hint: it's not perfect) but it's made me think about writing an alternative. Rather than shipping all the logs all around, my plan is to have each source (i.e. web server) run a process on it's own logs, and use something like Redis to store the aggregated statistics.

by joppy 6y ago

PaulRobinson 6y ago

Server side was how it was always done back in the early days of the web, and analog[0] was state of the art. Around 1999/2000 there was a rise of ISPs needing to install reverse proxy caches because the growth of consumer access meant they were getting seriously contended on upstream access. I was working at the time at a UK 0845 white label ISP called Telinco (was behind Connect Free, Totalise, Current Bun and other 0845 ISPs), and to my knowledge we were the first in the UK to install a Netapps cache. It was the moment we realised (by checking the logs to see if it was working), just how much porn our customers were accessing. Those caches blow server side analytics to pieces, because frequently you wouldn't even know the user had hit the page. What server side analytics was useful for is what we'd now call Observability: they gave reasonable Latency, Error Rate and Throughput metrics, which combined with some other system logs might also give you a sense of Saturation. As such, they were not too useful for marketing. Google Analytics was the first product that allowed high fidelity analytics even if reverse proxy caches (and even browser caches), were all over the place. And here we are. In a World where we are tightly surveilled by corporate entities in order to try and get us to click on things. Bit sad really. I'd encourage people to think about what they need these analytics for. If it's marketing, you might just as well using GA: it's the best product out there. We just need to lobby for better regulation (at least GDPR and cookie setting popovers give us choices on that regard now). If you're stroking your ego, consider whether such an invasive technology is worth the price, and if you need those numbers. If you're making sure your infrastructure can handle the traffic, use server side analytics alone. Parse your logs using the huge number of tools out there able to do that in near-realtime, and leave your users' browsers free of tracking cookies and javascript. [0] https://www.web42.com/analog/ https://www.web42.com/analog/

Nextgrid 6y ago
Caches are irrelevant now that the world has moved to HTTPS.
- divbzero 6y ago
  There are no public caches with HTTPS, but there are still private browser and CDN caches to contend with. To ensure your origin server is hit on subsequent HTTPS requests, you would still need to configure response headers for cache validation [1] to be Cache-Control: no-cache instead of Cache-Control: private [1]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ca...
  - Nextgrid 6y ago
    Yes, but those are caches that you control, so when it comes to analytics you would get the logs off them too in order to get accurate metrics.

dig1 6y ago

Webalizer [1] can be alternative. No fancy UI, but gets job done. For anything heavier and serious, ELK [2]. [1] http://www.webalizer.org/ http://www.webalizer.org/ [2] https://www.elastic.co/what-is/elk-stack https://www.elastic.co/what-is/elk-stack

stephenr 6y ago

I've setup GoAccess for a client's site, the problem is it doesn't have a great HA solution. You either ship all your logs to one place (and hope that place doesn't go offline) or ship your logs to multiple places and hope both destinations are in sync. We've opted for #2 right now (hint: it's not perfect) but it's made me think about writing an alternative. Rather than shipping all the logs all around, my plan is to have each source (i.e. web server) run a process on it's own logs, and use something like Redis to store the aggregated statistics.