3 ms·

Note: I build and maintain such systems for a living. There's a lot of context that's missing from your post, some questions that can help us guide you in the

Note: I build and maintain such systems for a living. There's a lot of context that's missing from your post, some questions that can help us guide you in the right direction: 1) Can your website call out to external services, or are you limited to operating behind a company network? 2) Is this more of an ad-hoc analysis or do you want to invest in a framework to be able to track such metrics systematically over time? 3) How important is data accuracy? Adblock can easily mess with client-side metrics. 4) How real-time do metrics need to be? The big trade-off here is speed vs accuracy. 5) How long do you intend to keep this data? This is a pretty big concern with regards to privacy and storage costs. If you'd rather not share some of these answers on a public forum, feel free to shoot me an email.

psankar 7y ago
1) The website cannot call to external services, which is the primary reason why we thought about implementing it from scratch. 2) We want to invest in building a good framework to track such metrics systematically over time 3) We have some non-web API clients too. Adblock is not a problem. 4) Accuracy is better. Speed is not that critical and could even be a few minutes delayed. 5) The data will be kept for a few months at least, if not years. The storage costs are not a big problem. This is for enterprise based solution, where the deployment may happen in a private network.
- karanke 7y ago
  The standard setup nowadays is something like this: http://bit.ly/2MFAAt9 http://bit.ly/2MFAAt9. You can use different technologies based on your use case, but you probably need all the pieces outlined above. As someone else has mentioned, if you're looking for trade-offs between different technologies, I'd recommend "Designing Data-Intensive Applications" by Kleppmann.
  - namanyayg 7y ago
    How does Snowplow compare to this?
- jrumbut 7y ago
  I would say what you need to consider more than anything is making sure you have the right data, and that the data can be combined. This is the hard part of analytics for an app that is more back office oriented, understanding what will be needed to get truly accurate and useful information to support the reports people will want in 2 years, 5 years, whatever time frame is long enough for things to really change in your environment. Try to think in an adversarial way, what question could someone come up with that I can't answer. The user who has seen the most errors? Usage trends by department? One place this might lead is wanting to put a way to link request/error logs back to an application level user account (in a way that respects security and privacy), this can become great debugging info too.

psankar 7y ago

1) The website cannot call to external services, which is the primary reason why we thought about implementing it from scratch. 2) We want to invest in building a good framework to track such metrics systematically over time 3) We have some non-web API clients too. Adblock is not a problem. 4) Accuracy is better. Speed is not that critical and could even be a few minutes delayed. 5) The data will be kept for a few months at least, if not years. The storage costs are not a big problem. This is for enterprise based solution, where the deployment may happen in a private network.

karanke 7y ago
The standard setup nowadays is something like this: http://bit.ly/2MFAAt9 http://bit.ly/2MFAAt9. You can use different technologies based on your use case, but you probably need all the pieces outlined above. As someone else has mentioned, if you're looking for trade-offs between different technologies, I'd recommend "Designing Data-Intensive Applications" by Kleppmann.
- namanyayg 7y ago
  How does Snowplow compare to this?
jrumbut 7y ago
I would say what you need to consider more than anything is making sure you have the right data, and that the data can be combined. This is the hard part of analytics for an app that is more back office oriented, understanding what will be needed to get truly accurate and useful information to support the reports people will want in 2 years, 5 years, whatever time frame is long enough for things to really change in your environment. Try to think in an adversarial way, what question could someone come up with that I can't answer. The user who has seen the most errors? Usage trends by department? One place this might lead is wanting to put a way to link request/error logs back to an application level user account (in a way that respects security and privacy), this can become great debugging info too.