5 ms·

What interfaces akin to Google Analytics read web server logs?

by pgrote 11y ago

steventhedev 11y ago
aside from the ELK stack, you can also send views/events from your server to GA (this is what I do). Staccato[0] is my go to gem for this [0]: https://github.com/tpitale/staccato https://github.com/tpitale/staccato
tombrossman 11y ago
Piwik can. Run a self-hosted instance or pay them to run it, and it handles server logs no problem. You do give up the ability to get live stats, but you get better performance and the ability to track more visitors (read: people like me who have blocked GA for years...) Added bonus: Referrer spam is automatically blocked by default. https://piwik.org/log-analytics/ https://piwik.org/log-analytics/ EDIT: Almost forgot GoAccess, if you are okay with a terminal app and want live stats (can also be scripted to generate HTML reports) - http://goaccess.io/ http://goaccess.io/
- chuckouellet 11y ago
  Charles, Snipcart co-founder here. As mentioned by the end of the post, we're actively looking to expand our analytics game a bit. Piwik seems like a good place to start, so thanks for sharing. Cheers!
  - tombrossman 11y ago
    Try running it alongside GA and comparing stats. I did this for a while and switched to Piwik as it consistently counted more visits (legitimate visits, obviously). Also, please consider adding your Google Analytics use to your privacy policy. They require it, even though most people ignore this requirement and there is no enforcement. Side note, I just tried @ replying to your companies post on Twitter and got a @your account may not be allowed to perform this action' http://i.imgur.com/9uoEn9B.png http://i.imgur.com/9uoEn9B.png Strange, never saw that before...
    - chuckouellet 11y ago
      We received some mentions in the last minutes, and we just reviewed our account settings, nothing seems problematic on our end. Weird indeed. But thanks for the feedback!
    - HNaTTY 11y ago
      > (legitimate visits, obviously) That's an important point; I'm seeing referral spam in my GA reports which I never noticed before 2015. I manually discount it when compiling reports, and my understanding is that these spammers hit GA UAs at random without even loading your website.
      - snowwrestler 11y ago
        It's incredible easy to filter this out in GA. Just set a filter or segment that only counts traffic from your known hostnames.
- xorcist 11y ago
  Piwik is astonishingly good. It'll give you trouble if you have a lot of traffic, but for moderate sites it's pretty much a completely self hosted drop in replacement for Google Analytics. You can track conversions, set up campaign goals and view statistics overlays for example. Not all features are available when using the log import backend, as compared to the javascript thingy, but most are and it'll give you an overview on how your ad blocking customers are doing.
- luxpir 11y ago
  Quick note that with hourly, or more frequent, cron jobs you get quasi live data. Mentioned a little about logimport in a recent post on stripping external calls and pointless js/css/font loads from our site. Otherwise well put, same experience here. Btw, tail -f /log.log is always fun for live data...
  - roel_v 11y ago
    Yeah, I build a system in the late 1990's that did something like that. Required a dedicated server though, the web server farm would copy over their log files to this analytics machine with cron jobs, which would then dynamically aggregate that data and update the stats cache.
hippich 11y ago
keep in mind that you can push stats into GA from the server. I actually made PoC for pushing server stats to the GA - https://github.com/hippich/server-analytics https://github.com/hippich/server-analytics The same way you can gather all available stats on server side and push these yourself. You can override IP too. Obviously not everything will be available, like screen resolution, but you still can capture a lot of stuff.

What interfaces akin to Google Analytics read web server logs?

by pgrote 11y ago

What interfaces akin to Google Analytics read web server logs?

aside from the ELK stack, you can also send views/events from your server to GA (this is what I do). Staccato[0] is my go to gem for this [0]: https://github.com/tpitale/staccato https://github.com/tpitale/staccato

tombrossman 11y ago

Piwik can. Run a self-hosted instance or pay them to run it, and it handles server logs no problem. You do give up the ability to get live stats, but you get better performance and the ability to track more visitors (read: people like me who have blocked GA for years...) Added bonus: Referrer spam is automatically blocked by default. https://piwik.org/log-analytics/ https://piwik.org/log-analytics/ EDIT: Almost forgot GoAccess, if you are okay with a terminal app and want live stats (can also be scripted to generate HTML reports) - http://goaccess.io/ http://goaccess.io/

chuckouellet 11y ago
Charles, Snipcart co-founder here. As mentioned by the end of the post, we're actively looking to expand our analytics game a bit. Piwik seems like a good place to start, so thanks for sharing. Cheers!
- tombrossman 11y ago
  Try running it alongside GA and comparing stats. I did this for a while and switched to Piwik as it consistently counted more visits (legitimate visits, obviously). Also, please consider adding your Google Analytics use to your privacy policy. They require it, even though most people ignore this requirement and there is no enforcement. Side note, I just tried @ replying to your companies post on Twitter and got a @your account may not be allowed to perform this action' http://i.imgur.com/9uoEn9B.png http://i.imgur.com/9uoEn9B.png Strange, never saw that before...
  - chuckouellet 11y ago
    We received some mentions in the last minutes, and we just reviewed our account settings, nothing seems problematic on our end. Weird indeed. But thanks for the feedback!
  - HNaTTY 11y ago
    > (legitimate visits, obviously) That's an important point; I'm seeing referral spam in my GA reports which I never noticed before 2015. I manually discount it when compiling reports, and my understanding is that these spammers hit GA UAs at random without even loading your website.
    - snowwrestler 11y ago
      It's incredible easy to filter this out in GA. Just set a filter or segment that only counts traffic from your known hostnames.
xorcist 11y ago
Piwik is astonishingly good. It'll give you trouble if you have a lot of traffic, but for moderate sites it's pretty much a completely self hosted drop in replacement for Google Analytics. You can track conversions, set up campaign goals and view statistics overlays for example. Not all features are available when using the log import backend, as compared to the javascript thingy, but most are and it'll give you an overview on how your ad blocking customers are doing.
luxpir 11y ago
Quick note that with hourly, or more frequent, cron jobs you get quasi live data. Mentioned a little about logimport in a recent post on stripping external calls and pointless js/css/font loads from our site. Otherwise well put, same experience here. Btw, tail -f /log.log is always fun for live data...
- roel_v 11y ago
  Yeah, I build a system in the late 1990's that did something like that. Required a dedicated server though, the web server farm would copy over their log files to this analytics machine with cron jobs, which would then dynamically aggregate that data and update the stats cache.

hippich 11y ago

keep in mind that you can push stats into GA from the server. I actually made PoC for pushing server stats to the GA - https://github.com/hippich/server-analytics https://github.com/hippich/server-analytics The same way you can gather all available stats on server side and push these yourself. You can override IP too. Obviously not everything will be available, like screen resolution, but you still can capture a lot of stuff.