8 ms·

This is why I use server side analytics almost exclusively. You just can't rely on every browser running all your javascript.

strommen 11y ago
How do you filter out crawlers, bots, pre-fetchers, etc.? I've personally never even tried using server-side analytics because I assumed there would be so much noise it wouldn't be worth the trouble.
- TillE 11y ago
  Every well-behaved bot (ie, almost all of them) has its own user agent string. You can just grep -v those out.
  - mjn 11y ago
    I wish that were true, but I'd estimate that less than half of the bot hits on my website properly identify themselves as bots. Most of the crawlers running on DigitalOcean/AWS/etc. seem to use a user-agent string lifted from one of the common browsers.
    - kuschku 11y ago
      Check for the order of the headers and the TLS protocols it claims to support – those are useful for identification.
- th0ma5 11y ago
  I must be getting old, never? There's a lot of techniques like doings stats on blank pixels and such that have been around for a long time. A lot of ad blocking techniques start at the DNS of the hosted resource, so having a single domain for everything (if practical) can present some reliable information.
pgrote 11y ago
What interfaces akin to Google Analytics read web server logs?
- steventhedev 11y ago
  aside from the ELK stack, you can also send views/events from your server to GA (this is what I do). Staccato[0] is my go to gem for this [0]: https://github.com/tpitale/staccato https://github.com/tpitale/staccato
- tombrossman 11y ago
  Piwik can. Run a self-hosted instance or pay them to run it, and it handles server logs no problem. You do give up the ability to get live stats, but you get better performance and the ability to track more visitors (read: people like me who have blocked GA for years...) Added bonus: Referrer spam is automatically blocked by default. https://piwik.org/log-analytics/ https://piwik.org/log-analytics/ EDIT: Almost forgot GoAccess, if you are okay with a terminal app and want live stats (can also be scripted to generate HTML reports) - http://goaccess.io/ http://goaccess.io/
  - chuckouellet 11y ago
    Charles, Snipcart co-founder here. As mentioned by the end of the post, we're actively looking to expand our analytics game a bit. Piwik seems like a good place to start, so thanks for sharing. Cheers!
    - tombrossman 11y ago
      Try running it alongside GA and comparing stats. I did this for a while and switched to Piwik as it consistently counted more visits (legitimate visits, obviously). Also, please consider adding your Google Analytics use to your privacy policy. They require it, even though most people ignore this requirement and there is no enforcement. Side note, I just tried @ replying to your companies post on Twitter and got a @your account may not be allowed to perform this action' http://i.imgur.com/9uoEn9B.png http://i.imgur.com/9uoEn9B.png Strange, never saw that before...
EvilMonkeyMat 11y ago
With ad blockers, user tracking gets harder and harder, and this sucks when you want to understand the effects of changes on your own website, such as changing user menus, moving content blocks from a position to another, or even to a different page, etc. The value of partnerships with other websites gets harder to evaluate too. Hosting my websites on Linux environments, I never stopped using server-side generated statistics (based on Apache's access logs) with tools like AWStats. Used together, I feel that client and server-side stats give a much better image of what really happens on my websites. But in the end, I feel awstats isn't enough, and I'll take a look at other solutions that have been pointed out in these comments. Thanks!
- rando3826 11y ago
  Is Apache reporting back to the apache foundation about how you are using your apache software? Maybe your expectations about user tracking are the problem.
  - EvilMonkeyMat 11y ago
    I'm pretty sure they aren't, but I must admit, I have never checked. I supposed that would be known by now... One thing is sure, access logs are located on my own VM, and are processed locally by awstats. The only thing that binds requests to an actual user is the IP address, and one must contact ISPs to have more information. I don't have the authority to do that.

strommen 11y ago

How do you filter out crawlers, bots, pre-fetchers, etc.? I've personally never even tried using server-side analytics because I assumed there would be so much noise it wouldn't be worth the trouble.

TillE 11y ago
Every well-behaved bot (ie, almost all of them) has its own user agent string. You can just grep -v those out.
- mjn 11y ago
  I wish that were true, but I'd estimate that less than half of the bot hits on my website properly identify themselves as bots. Most of the crawlers running on DigitalOcean/AWS/etc. seem to use a user-agent string lifted from one of the common browsers.
  - kuschku 11y ago
    Check for the order of the headers and the TLS protocols it claims to support – those are useful for identification.
th0ma5 11y ago
I must be getting old, never? There's a lot of techniques like doings stats on blank pixels and such that have been around for a long time. A lot of ad blocking techniques start at the DNS of the hosted resource, so having a single domain for everything (if practical) can present some reliable information.

pgrote 11y ago

What interfaces akin to Google Analytics read web server logs?

steventhedev 11y ago
aside from the ELK stack, you can also send views/events from your server to GA (this is what I do). Staccato[0] is my go to gem for this [0]: https://github.com/tpitale/staccato https://github.com/tpitale/staccato
tombrossman 11y ago
Piwik can. Run a self-hosted instance or pay them to run it, and it handles server logs no problem. You do give up the ability to get live stats, but you get better performance and the ability to track more visitors (read: people like me who have blocked GA for years...) Added bonus: Referrer spam is automatically blocked by default. https://piwik.org/log-analytics/ https://piwik.org/log-analytics/ EDIT: Almost forgot GoAccess, if you are okay with a terminal app and want live stats (can also be scripted to generate HTML reports) - http://goaccess.io/ http://goaccess.io/
- chuckouellet 11y ago
  Charles, Snipcart co-founder here. As mentioned by the end of the post, we're actively looking to expand our analytics game a bit. Piwik seems like a good place to start, so thanks for sharing. Cheers!
  - tombrossman 11y ago
    Try running it alongside GA and comparing stats. I did this for a while and switched to Piwik as it consistently counted more visits (legitimate visits, obviously). Also, please consider adding your Google Analytics use to your privacy policy. They require it, even though most people ignore this requirement and there is no enforcement. Side note, I just tried @ replying to your companies post on Twitter and got a @your account may not be allowed to perform this action' http://i.imgur.com/9uoEn9B.png http://i.imgur.com/9uoEn9B.png Strange, never saw that before...

EvilMonkeyMat 11y ago

With ad blockers, user tracking gets harder and harder, and this sucks when you want to understand the effects of changes on your own website, such as changing user menus, moving content blocks from a position to another, or even to a different page, etc. The value of partnerships with other websites gets harder to evaluate too. Hosting my websites on Linux environments, I never stopped using server-side generated statistics (based on Apache's access logs) with tools like AWStats. Used together, I feel that client and server-side stats give a much better image of what really happens on my websites. But in the end, I feel awstats isn't enough, and I'll take a look at other solutions that have been pointed out in these comments. Thanks!

rando3826 11y ago
Is Apache reporting back to the apache foundation about how you are using your apache software? Maybe your expectations about user tracking are the problem.
- EvilMonkeyMat 11y ago
  I'm pretty sure they aren't, but I must admit, I have never checked. I supposed that would be known by now... One thing is sure, access logs are located on my own VM, and are processed locally by awstats. The only thing that binds requests to an actual user is the IP address, and one must contact ISPs to have more information. I don't have the authority to do that.