4 ms·

Firstly, I'm super biased as I'm the CEO of a product analytics company where one of the value props is getting the data in real time. I agree with this post th

Firstly, I'm super biased as I'm the CEO of a product analytics company where one of the value props is getting the data in real time. I agree with this post that people will look at single data points out of context and weight the evidence much more strongly than they should be. Analytics should be one of many tools you use that informs your understanding of how customers are using your product. I also agree that I see a lot of early stage startups invest way too much in building out real-time analytics stores without thinking about what they value they get out of it. That's not something you should do until you're in the 100+ engineer range. That said, this is a post-hoc justification of why having real-time analytics is bad. Delaying the data by 24 hours doesn't automatically make significance testing better or force people to incorporate more context when interpreting their data. If that's the issue, fix that problem, don't blame the tools. There's a ton of positive value to having real time data. Just off the top of my head: 1) If you've instrumented something incorrectly you can see that and fix it right away 2) Even worse, if you've accidentally messed up a feature with a release you can know about it right away. This happened to one of our customers recently and without a real-time analytics system they wouldn't have caught it quickly (and this is a tech startup that's well regarded for their engineering that everyone here would know the name of). 3) You can observe significant changes to your user base right away, eg during a launch or if you're getting a lot of new users from a specific channel. 4) It allows you to have more confidence in deploying multiple times a day. It's a little crazy to me that the deployment of a product is faster than our ability to measure it. I think the real issue is that we're in the early days of analytics and people don't have a great understanding of how to leverage their tools properly. It's like web search before Google or file sync before Dropbox. People will look at single data points and conclude totally crazy things. They won't have an understanding of basic things like the amount of fluctuation on a week to week basis. The largest analytics provider in the world (Google) gives away their product as an ancillary service to drive you to purchase more ads, not to have a better understanding of how your customers are using your product. I'm hopeful this will change over the next 5 years though (and for us to be a part of that!)

ThomPete 9y ago
Let me give you another angle on this. Asking people to become data analysts when they just want to run a business and have a million other things to worry about is exactly the wrong way to think about this. People shouldn't need to become data analysts to understand the data the software should be able to analyze and give simple advice when it has it. I have designed my share of analytics software from the classical dashboards to more complex things like a market replay for Nasdaq. At the end of the day, analytics primarily is useful if it can be used to make decisions with. If you really want to build a useful analytics software tool you have to build something that analysis and either do things for me automatically (like seeing a lot of people are suddenly coming to your site because of a keyword and then quickly do some google advertising) or it has to be able to tell you something you can use (ex. You should buy extra hotdogs for Sunday as you are about to get a lot of customers because of bad weather after/before the football match.
- sskates 9y ago
  Agree with everything you're saying about data. Given how hard an analytics system is to set up properly, it's one of the last pieces you should be adding in to get an understanding of how to build your business.
scott_karana 9y ago
Finding products breakages through analytics is very backwards. If a team is deploying code that untested (or with that poor a process), could you really trust them to make informed judgements either? That needs to be fixed earlier than your product. UX improvements or mistakes in a conversion pipeline, sure though. Fair points there. :)
- TheReveller 9y ago
  I believe that's naive. There is always a non-zero chance that deploying new code will cause an issue that is not covered by your unit tests or integration tests. Good testing methodology means adding to your tests when you find such a case, not promising that all possible cases are covered all the time, because that's unreasonable, and the assumption that there's no way it could break on deploy is the kind of hubris that leads to breakages when you deploy.
  - scott_karana 9y ago
    I agree with "defence in depth": analytics can help catch problems... However, many here are pitching it as a primary selling point. Relying exclusively on it is a design/process failure, in my book.
rusk 9y ago
There's a ton of positive value to having real time data. Just off the top of my head: 1) If you've instrumented something incorrectly ... 2) Even worse, if you've accidentally messed up ... 3) You can observe significant changes ... 4) It allows you to have more confidence in deploying multiple times ... Embracing both yours, and the author's perspectives I think I have a compromise. It seems to me the author's gripe is more specifically that his customers want to "do the same large window analytics in realtime" whereas here you highlight particular usecases where realtime analytics are simple. In a previous life I was producing near-realtime analytics for network activity for populations numbering 10s of millions, all the time, in realtime, except the definition of realtime was stretched to allow a 15 minute latency. Reducing this 15 minute latency was oft requested e.g to give help-desk operators an instant view of what is happening for a customer or, to automate responses to particular well-known system issues in realtime. It wouldn't have been impossible to reduce our 15 minute window but given various architectural limitation of the time it would have been expensive and upon analysing these customer requirements it made more sense to simply siphon off the realtime event-data at our monitoring points into a parallel dataflow. TL;DR you do different things with realtime vs aggregate data. Do you really need the expense of a system that does both equally well?

ThomPete 9y ago

Let me give you another angle on this. Asking people to become data analysts when they just want to run a business and have a million other things to worry about is exactly the wrong way to think about this. People shouldn't need to become data analysts to understand the data the software should be able to analyze and give simple advice when it has it. I have designed my share of analytics software from the classical dashboards to more complex things like a market replay for Nasdaq. At the end of the day, analytics primarily is useful if it can be used to make decisions with. If you really want to build a useful analytics software tool you have to build something that analysis and either do things for me automatically (like seeing a lot of people are suddenly coming to your site because of a keyword and then quickly do some google advertising) or it has to be able to tell you something you can use (ex. You should buy extra hotdogs for Sunday as you are about to get a lot of customers because of bad weather after/before the football match.

sskates 9y ago
Agree with everything you're saying about data. Given how hard an analytics system is to set up properly, it's one of the last pieces you should be adding in to get an understanding of how to build your business.

scott_karana 9y ago

Finding products breakages through analytics is very backwards. If a team is deploying code that untested (or with that poor a process), could you really trust them to make informed judgements either? That needs to be fixed earlier than your product. UX improvements or mistakes in a conversion pipeline, sure though. Fair points there. :)

TheReveller 9y ago
I believe that's naive. There is always a non-zero chance that deploying new code will cause an issue that is not covered by your unit tests or integration tests. Good testing methodology means adding to your tests when you find such a case, not promising that all possible cases are covered all the time, because that's unreasonable, and the assumption that there's no way it could break on deploy is the kind of hubris that leads to breakages when you deploy.
- scott_karana 9y ago
  I agree with "defence in depth": analytics can help catch problems... However, many here are pitching it as a primary selling point. Relying exclusively on it is a design/process failure, in my book.

rusk 9y ago

There's a ton of positive value to having real time data. Just off the top of my head: 1) If you've instrumented something incorrectly ... 2) Even worse, if you've accidentally messed up ... 3) You can observe significant changes ... 4) It allows you to have more confidence in deploying multiple times ... Embracing both yours, and the author's perspectives I think I have a compromise. It seems to me the author's gripe is more specifically that his customers want to "do the same large window analytics in realtime" whereas here you highlight particular usecases where realtime analytics are simple. In a previous life I was producing near-realtime analytics for network activity for populations numbering 10s of millions, all the time, in realtime, except the definition of realtime was stretched to allow a 15 minute latency. Reducing this 15 minute latency was oft requested e.g to give help-desk operators an instant view of what is happening for a customer or, to automate responses to particular well-known system issues in realtime. It wouldn't have been impossible to reduce our 15 minute window but given various architectural limitation of the time it would have been expensive and upon analysing these customer requirements it made more sense to simply siphon off the realtime event-data at our monitoring points into a parallel dataflow. TL;DR you do different things with realtime vs aggregate data. Do you really need the expense of a system that does both equally well?