3 ms·
I really love Mixpanel, it is the "best in breed" when it comes to event analytics. I don't think many people would argue with that, it's awesome. In fact, we u
by bryanh 14y ago
I really love Mixpanel, it is the "best in breed" when it comes to event analytics. I don't think many people would argue with that, it's awesome. In fact, we use it at Zapier for pretty much everything (basic funnels, A/B testing, retention, engagement, etc...) and we haven't even reached our full potential with it yet. But I still have some pretty fundamental beefs with the state of analytics software in general.
About the article, the problem stems from the fact that proper analytics is hard and is (arguably) getting harder with more advanced packages.Shouldn't it be going in the opposite direction?
It is a lot easier to track discrete downloads or pageviews than some other, more insightful metric, so people will naturally gravitate to the cheaper metrics. Until this is reversed, bullshit metrics will reign.
Anyways, my beefs:
First: how do you decide what data to send into the package?
The more data you send, the better (sure), but at a certain point you are just duplicating your internal datastore, so that is too much, right? But not enough and you'll miss a chance to understand a phenomena that you didn't predict seeing (isn't that the point?). After you decide, then you write a crapton of code to send it all (what about backfilling data when you want to track something new?).
Second: once you are collecting the data, how do you know what metrics to actively track?
This is definitely existential, but it's back to the core problem: doing analytics properly is hard. Why couldn't the software let me define some properties about the type of app I am running and suggest some strategies (you have a subscription SaaS app? Try tracking paid plan retention, signup funnels, etc...). Maybe it could go even further with reverse funnels, as in: what events are the most important and work backwards. I could see some automation and discovery possibilities there.
Third: do I really have to dig around trying to find something useful?
All the data is there, the software should tell me what is useful or interesting. It's definitely a hard problem, but I would throw money at software that could send me this email: "Looks like users who experienced event "ABC" also performed your highest priority event "Signup" at a 13% higher rate. This observation is 99% confident." Of course, you'd need to investigate a littler deeper to see if that isn't just a fluke or something stupidly obvious (like: people who view a page signup at a higher rate than those who don't), but at least I might learn something.
I know this is certainly a pipe dream as of today, but I vow to shower someone with money if they can do this.
In my opinion, the next generation of analytics software won't just have more bells and whistles, it will fundamentally shorten the time to some sort of real "AHA!" insight.
- suhail 14y agoWe deal with this every day at Mixpanel. Education is a tough problem to solve. Here's how we think about it with respect to your questions: "First: how do you decide what data to send into the package?" My recommendation is you think about this gradually. Don't instrument everything. Start by picking 5 metrics you really, really care about. One of them should be your One Key Metric (OKM) - a metric that you would be the company on if you could only pick one thing to measure. I think the pressure of picking 5 helps you decide what to measure. "Second: once you are collecting the data, how do you know what metrics to actively track?" When you start gradually, there isn't much to pay attention to. Stay focused only on a set of metrics. Add more metrics when you want to understand those specific metrics more deeply. Add more data to split those metrics into groups to dig even deeper. Keep driving your OKM up as much as possible--sometimes you have to do it with something that correlates to it. "Third: do I really have to dig around trying to find something useful?" Today, the answer is yes. Businesses are fundamentally very different. Analytics is a lot like science experiments in high school: You start with a hypothesis, you test to see if you were right, and if you were then you came up with a conclusion or answer to your experiment. It's hard work to build a good business. I think the first step is for companies like us to help you understand your data. The next step that I think is more important than simply pulling "insights" out automatically is to help you take action on it so you grow.
- bryanh 14y agoThanks for the response. I just signed up fresh to Mixpanel to play around and you guys have solid technical on-boarding around just installing and collecting any data, but not a lot of details on what to collect. I'm sure that getting someone to collect just anything is a better first step than some sort of high level tutorial, which is why you do it that way. I like the OKM idea a lot. Perhaps that could be a pretty solid center stone for on-boarding a new user. I just remember being overwhelmed by the blank slate every time I signed up for any analytics package.
- suhail 14y agoWe generally reach out on the support side proactively to help. Anyone on our team is happy to think it through for you. If you want to learn more in the abstract we recently came out with an education series: http://mixpanel.com/education http://mixpanel.com/education