6 ms·

Andreessen and Mixpanel Call for an End to “Bullshit Metrics”

116 points by billclerico 14y ago

physcab 14y ago
The truth is that there is no one single metric that rules them all. You have to look at a suite of metrics to fully understand how users are using your product. It might be nice for YouTube to measure numbers of views a video has, but what if the majority of those views come from the same person? What if that person doesn't come back to YouTube? It might also be tempting to just measure the retention of users. But that doesn't give you the whole picture either. Is a user who comes in on Day 1 and drops $99 but never comes back again worse than a user who uses your app everyday but never pays? For real businesses (read: those with actual business plans) the closest single metric of importance is Customer Lifetime Value. And the equation is very simple. Make your cost of customer acquisition less than your LTV and you will be making money.
- ritchiea 14y ago
  First of all, they are not saying there is one true metric that rules them all. They said each site will have one key metric, which makes a lot of sense. Each site will most want to measure how users engage with their domain. The example mixpanel offers is that Instagram wants to know how many users uploaded a photo. This matters way more than pageviews, etc. It tells you something about user engagement. The bigger issue I see in your comment is that sites at different stages in their lifecycle will want to measure different things. Customer lifetime value is the most important number for a mature business but early stage startups are nowhere near getting a realistic number for customer lifetime value. If I am the founder of a startup I want detailed knowledge of how users are interacting with the product I created to address a problem domain. I can get that info immediately after I start acquiring users, well before I have an idea of whether I have a viable business and what my customer lifetime value is going to be.
  - simonholroyd 14y ago
    It seems to me, if Instagram could really only chose one metric to track it would be an app visit, not a photo upload. Instagram must have users who are primarily content creators and another larger group (probably a superset) that are content consumers. Not focusing on the actions of the content consumers would seem really bad for business since what early-stage Instagram's investors care about is eyeballs not photographs and what late-stage Instagram's advertisers care about is eyeballs not photographs. Really Instagram probably has a few OKMs (visits, uploads, hearts, comments, follows). All of those are important and should be tracked. In their documentation and educational videos, I've not seen Mixpanel focus on visits as a key metric. I'm not sure why.
- ChuckMcM 14y ago
  I really resonate with this rant, at the end of the day the 'metrics' that should make sense for investors in web type properties all involve the word 'revenue.' Like 'revenue per thousand pageviews' or 'revenue per customer' or 'net revenue.' Because without revenue the web site is essentially on a path to non-existence. I like Fred Wilson's analysis [1] of different revenue models it puts this stuff in perspective. The old "drive traffic too then harvest it with AdSense" model is losing a lot of steam. [1] http://www.avc.com/a_vc/mba-mondays/ http://www.avc.com/a_vc/mba-mondays/
Avitas 14y ago
This will not happen. Individuals and corporations use these because they make them look better. Should companies selling, say, food products be mandated to show pictures of the actual delivered product on advertisements, menus and packaging? That's not going to happen without additional legislation. Should resumes of individuals highlight workplace weaknesses, medical conditions, personality faults, past failures, major conflicts, etc.? The web is no different. Unless individuals and corporations are persuaded to do otherwise, they will continue to use metrics as they please.
throwaway-29382 14y ago
Good thing they don't use any BS metrics on their about page. https://mixpanel.com/about/ https://mixpanel.com/about/
- suhail 14y ago
  We don't. Data points (or actions as we say on the about page) correlate directly with the amount of revenue we make. 85-90% of all data points are paid for by our customers. It's how we make money: http://mixpanel.com/pricing http://mixpanel.com/pricing It actually does a better job correlating with our growth than purely the number of customers we have. We've also been public about that.
  - physcab 14y ago
    It makes sense that data points correlate to your growth, but what's interesting is that data points do not necessarily mean higher value for your customers. If there was a metric that only required one data point to measure and that metric could be the basis for a successful company, people would probably use less data points and gain more value. But most people don't know what to measure, so they default to measure everything. Thats good for you, but doesn't necessarily mean its good for them.
    - suhail 14y ago
      That's a valid point. The thing is, our growth in data largely correlates with customer satisfaction. If a customer finds no value but sends us a lot of data, they stop because they are paying for our product. That causes a decrease in data points. Admittedly, we do use a different metric to determine how "valuable" our product is in a more honest way. We're talking about non-bullshit metrics that highly correlate to growth. Not all valuable products necessarily grow.
      - mbesto 14y ago
        What qualifications (or quantifications) do you use to ensure that your value is more honest than someone else's? At the end of the day, value is determined by someone who is not yourself. This article for more me is pretty straight forward. It's a well crafted PR play (whether made by commercial means or not is irrelevant), that draws readers to putting more importance on what metrics are used to determine valuation. Something that they can draw from using a service such as Mixpanel. The reality is that you, nor I, can definitely provide one metric that clearly correlates to financial valuation. Every case is different and every case warrants negotiation. Mixpanel appears to be a great service btw. I hope my tone doesn't dictate otherwise.
joe-mccann 14y ago
The need for new metrics! Written about extensively here months ago: http://subprint.com/blog/the-need-for-new-metrics http://subprint.com/blog/the-need-for-new-metrics
BrianEatWorld 14y ago
I think theres a difference between advertising or celebrating "bullshit" metrics and actually using them. For our most recent press releases, we reported many of these "bullshit" metrics because thats what gets acclaim and attracts attention to help the company gain the necessary notoriety to thrive. Internally, we still use real, actionable metrics to drive our development. We just don't report them to outsiders because its either sensitive information and likely uninteresting outside of the context modeling or users experience.
- 001sky 14y ago
  Internally, we still use real, actionable metrics to drive our development. We just don't report them to outsiders -- Pretty much how the world works.
karolisd 14y ago
This seems to be focused on web app start ups. What the most important metrics to track for eCommerce?
- suhail 14y ago
  It depends on what you're willing to be public about. I think your OKM is people buying something and looking at all the metrics around that. Great metrics for e-commerce are churn, lifetime value, repetition of purchases, average revenue per user. On the marketing side, I think it's useful to understand what kinds of marketing produce certain kinds of repeat revenue. Publicly, I think understanding how many customers purchased something in the past 30 days is useful. Potentially, the number of customers that purchased something above $X.
  - joshwa 14y ago
    OKM for eCommerce is revenue growth TY/LY by week, closely followed by average order value, average items per order, and conversion rate
bryanh 14y ago
I really love Mixpanel, it is the "best in breed" when it comes to event analytics. I don't think many people would argue with that, it's awesome. In fact, we use it at Zapier for pretty much everything (basic funnels, A/B testing, retention, engagement, etc...) and we haven't even reached our full potential with it yet. But I still have some pretty fundamental beefs with the state of analytics software in general. About the article, the problem stems from the fact that proper analytics is hard and is (arguably) getting harder with more advanced packages.Shouldn't it be going in the opposite direction? It is a lot easier to track discrete downloads or pageviews than some other, more insightful metric, so people will naturally gravitate to the cheaper metrics. Until this is reversed, bullshit metrics will reign. Anyways, my beefs: First: how do you decide what data to send into the package? The more data you send, the better (sure), but at a certain point you are just duplicating your internal datastore, so that is too much, right? But not enough and you'll miss a chance to understand a phenomena that you didn't predict seeing (isn't that the point?). After you decide, then you write a crapton of code to send it all (what about backfilling data when you want to track something new?). Second: once you are collecting the data, how do you know what metrics to actively track? This is definitely existential, but it's back to the core problem: doing analytics properly is hard. Why couldn't the software let me define some properties about the type of app I am running and suggest some strategies (you have a subscription SaaS app? Try tracking paid plan retention, signup funnels, etc...). Maybe it could go even further with reverse funnels, as in: what events are the most important and work backwards. I could see some automation and discovery possibilities there. Third: do I really have to dig around trying to find something useful? All the data is there, the software should tell me what is useful or interesting. It's definitely a hard problem, but I would throw money at software that could send me this email: "Looks like users who experienced event "ABC" also performed your highest priority event "Signup" at a 13% higher rate. This observation is 99% confident." Of course, you'd need to investigate a littler deeper to see if that isn't just a fluke or something stupidly obvious (like: people who view a page signup at a higher rate than those who don't), but at least I might learn something. I know this is certainly a pipe dream as of today, but I vow to shower someone with money if they can do this. In my opinion, the next generation of analytics software won't just have more bells and whistles, it will fundamentally shorten the time to some sort of real "AHA!" insight.
cwilson 14y ago
The first most important metric: When it goes up and to the right, it means your company doesn't die. Death can be alluded in many different ways, depending on the nature of your business, but it usually means making money, hitting a milestone that allows you to raise money, acquiring users, and so forth. Once your company evolves, and has beaten death for the foreseeable future, a new set of metrics come into play and are now the most important. These may be similar to the death-evading set above, or they may be something new like increasing revenue, retaining users, increasing engagement, optimizing CLV, and so forth. The point? If a metric means your company is not dying, or your company is growing, it matters. If it doesn't fit into those buckets? Probably doesn't matter.
mattmaroon 14y ago
A little ironic coming from the firm that invested in Groupon, inventor of the "Adjusted consolidated segment operating income".
evolve2k 14y ago
I would have liked the article to pony up what the companies own 'non BS' metrics are. Seemed a little flat to 'call BS' on everyone else then not deliver the goods itself.
kevinyun 14y ago
People will always be measuring everything by the numbers. Look at your number of Facebook friends, your LinkedIn connections, your GPA in college, your annual income. It's always a rational argument to say that bigger is better. In a world where metrics are quantified like this, this is the natural way of thinking. It's going to be difficult to seep through the bullshit metrics, especially when you're such a small startup with other numbers that are too embarrassing to display to the world -- but, really, when you think about it this is the only way to find and fix the problems that you have.
mmaunder 14y ago
Everything except bottom line profitability is a bullshit metric.
- wildwood 14y ago
  Until you can't make payroll because the deals that you booked as profit two months ago haven't paid you any money yet. Then cash flow suddenly becomes essential.
robryan 14y ago
An example in the article, number of pictures uploaded on Instagram, it is not clear that this is a good metric, at least until they work out a way for each additional image to bring in more revenue.
endlessvoid94 14y ago
Gaming companies have been doing this already for years.
deleted 14y ago
[deleted]
mrdavidjcole 14y ago
This article inspired me to create a twitter handle to highlight the best(?) examples of this: http://twitter.com/bullshitmetrics http://twitter.com/bullshitmetrics Anything recent come to mind?
JacobJans 14y ago
Analytics only matter if they help you make better decisions. Anything else, is, indeed, just vanity.
fidanov 14y ago
So they call to replace one bullshit metrics with other bullshit metrics. Nice. Not once in this article there were the words "Profit" or "Revenue" and yet they talk about business and companies. Here is a surprise for you. Real companies are interested in real profits from real revenue and you count revenue and profit only after the money is actually in your bank account. Otherwise one day you get the kind of terms of service just like Instragram these days.
- marcuskaz 14y ago
  Yeh, if there is one true metric it would be profit
majani 14y ago
Actually, I find that a lot of the metrics mentioned like uniques and pageviews are the ones that affect revenue for ad-driven startups. That's as real as it gets with metrics. When it comes to ad-driven startups, you could actually say that the "real" metrics suggested, like active users and engagement, are the actual bullshit metrics. For many ad companies like Google(search) or YouTube, high engagement by the users adds nothing to the bottom line.

116 points by billclerico 14y ago

The truth is that there is no one single metric that rules them all. You have to look at a suite of metrics to fully understand how users are using your product. It might be nice for YouTube to measure numbers of views a video has, but what if the majority of those views come from the same person? What if that person doesn't come back to YouTube? It might also be tempting to just measure the retention of users. But that doesn't give you the whole picture either. Is a user who comes in on Day 1 and drops $99 but never comes back again worse than a user who uses your app everyday but never pays? For real businesses (read: those with actual business plans) the closest single metric of importance is Customer Lifetime Value. And the equation is very simple. Make your cost of customer acquisition less than your LTV and you will be making money.

ritchiea 14y ago
First of all, they are not saying there is one true metric that rules them all. They said each site will have one key metric, which makes a lot of sense. Each site will most want to measure how users engage with their domain. The example mixpanel offers is that Instagram wants to know how many users uploaded a photo. This matters way more than pageviews, etc. It tells you something about user engagement. The bigger issue I see in your comment is that sites at different stages in their lifecycle will want to measure different things. Customer lifetime value is the most important number for a mature business but early stage startups are nowhere near getting a realistic number for customer lifetime value. If I am the founder of a startup I want detailed knowledge of how users are interacting with the product I created to address a problem domain. I can get that info immediately after I start acquiring users, well before I have an idea of whether I have a viable business and what my customer lifetime value is going to be.
- simonholroyd 14y ago
  It seems to me, if Instagram could really only chose one metric to track it would be an app visit, not a photo upload. Instagram must have users who are primarily content creators and another larger group (probably a superset) that are content consumers. Not focusing on the actions of the content consumers would seem really bad for business since what early-stage Instagram's investors care about is eyeballs not photographs and what late-stage Instagram's advertisers care about is eyeballs not photographs. Really Instagram probably has a few OKMs (visits, uploads, hearts, comments, follows). All of those are important and should be tracked. In their documentation and educational videos, I've not seen Mixpanel focus on visits as a key metric. I'm not sure why.
ChuckMcM 14y ago
I really resonate with this rant, at the end of the day the 'metrics' that should make sense for investors in web type properties all involve the word 'revenue.' Like 'revenue per thousand pageviews' or 'revenue per customer' or 'net revenue.' Because without revenue the web site is essentially on a path to non-existence. I like Fred Wilson's analysis [1] of different revenue models it puts this stuff in perspective. The old "drive traffic too then harvest it with AdSense" model is losing a lot of steam. [1] http://www.avc.com/a_vc/mba-mondays/ http://www.avc.com/a_vc/mba-mondays/

Avitas 14y ago

This will not happen. Individuals and corporations use these because they make them look better. Should companies selling, say, food products be mandated to show pictures of the actual delivered product on advertisements, menus and packaging? That's not going to happen without additional legislation. Should resumes of individuals highlight workplace weaknesses, medical conditions, personality faults, past failures, major conflicts, etc.? The web is no different. Unless individuals and corporations are persuaded to do otherwise, they will continue to use metrics as they please.

throwaway-29382 14y ago

Good thing they don't use any BS metrics on their about page. https://mixpanel.com/about/ https://mixpanel.com/about/

suhail 14y ago
We don't. Data points (or actions as we say on the about page) correlate directly with the amount of revenue we make. 85-90% of all data points are paid for by our customers. It's how we make money: http://mixpanel.com/pricing http://mixpanel.com/pricing It actually does a better job correlating with our growth than purely the number of customers we have. We've also been public about that.
- physcab 14y ago
  It makes sense that data points correlate to your growth, but what's interesting is that data points do not necessarily mean higher value for your customers. If there was a metric that only required one data point to measure and that metric could be the basis for a successful company, people would probably use less data points and gain more value. But most people don't know what to measure, so they default to measure everything. Thats good for you, but doesn't necessarily mean its good for them.
  - suhail 14y ago
    That's a valid point. The thing is, our growth in data largely correlates with customer satisfaction. If a customer finds no value but sends us a lot of data, they stop because they are paying for our product. That causes a decrease in data points. Admittedly, we do use a different metric to determine how "valuable" our product is in a more honest way. We're talking about non-bullshit metrics that highly correlate to growth. Not all valuable products necessarily grow.
    - mbesto 14y ago
      What qualifications (or quantifications) do you use to ensure that your value is more honest than someone else's? At the end of the day, value is determined by someone who is not yourself. This article for more me is pretty straight forward. It's a well crafted PR play (whether made by commercial means or not is irrelevant), that draws readers to putting more importance on what metrics are used to determine valuation. Something that they can draw from using a service such as Mixpanel. The reality is that you, nor I, can definitely provide one metric that clearly correlates to financial valuation. Every case is different and every case warrants negotiation. Mixpanel appears to be a great service btw. I hope my tone doesn't dictate otherwise.

joe-mccann 14y ago

The need for new metrics! Written about extensively here months ago: http://subprint.com/blog/the-need-for-new-metrics http://subprint.com/blog/the-need-for-new-metrics

BrianEatWorld 14y ago

I think theres a difference between advertising or celebrating "bullshit" metrics and actually using them. For our most recent press releases, we reported many of these "bullshit" metrics because thats what gets acclaim and attracts attention to help the company gain the necessary notoriety to thrive. Internally, we still use real, actionable metrics to drive our development. We just don't report them to outsiders because its either sensitive information and likely uninteresting outside of the context modeling or users experience.

001sky 14y ago
Internally, we still use real, actionable metrics to drive our development. We just don't report them to outsiders -- Pretty much how the world works.

karolisd 14y ago

This seems to be focused on web app start ups. What the most important metrics to track for eCommerce?

suhail 14y ago
It depends on what you're willing to be public about. I think your OKM is people buying something and looking at all the metrics around that. Great metrics for e-commerce are churn, lifetime value, repetition of purchases, average revenue per user. On the marketing side, I think it's useful to understand what kinds of marketing produce certain kinds of repeat revenue. Publicly, I think understanding how many customers purchased something in the past 30 days is useful. Potentially, the number of customers that purchased something above $X.
- joshwa 14y ago
  OKM for eCommerce is revenue growth TY/LY by week, closely followed by average order value, average items per order, and conversion rate

bryanh 14y ago

I really love Mixpanel, it is the "best in breed" when it comes to event analytics. I don't think many people would argue with that, it's awesome. In fact, we use it at Zapier for pretty much everything (basic funnels, A/B testing, retention, engagement, etc...) and we haven't even reached our full potential with it yet. But I still have some pretty fundamental beefs with the state of analytics software in general. About the article, the problem stems from the fact that proper analytics is hard and is (arguably) getting harder with more advanced packages.Shouldn't it be going in the opposite direction? It is a lot easier to track discrete downloads or pageviews than some other, more insightful metric, so people will naturally gravitate to the cheaper metrics. Until this is reversed, bullshit metrics will reign. Anyways, my beefs: First: how do you decide what data to send into the package? The more data you send, the better (sure), but at a certain point you are just duplicating your internal datastore, so that is too much, right? But not enough and you'll miss a chance to understand a phenomena that you didn't predict seeing (isn't that the point?). After you decide, then you write a crapton of code to send it all (what about backfilling data when you want to track something new?). Second: once you are collecting the data, how do you know what metrics to actively track? This is definitely existential, but it's back to the core problem: doing analytics properly is hard. Why couldn't the software let me define some properties about the type of app I am running and suggest some strategies (you have a subscription SaaS app? Try tracking paid plan retention, signup funnels, etc...). Maybe it could go even further with reverse funnels, as in: what events are the most important and work backwards. I could see some automation and discovery possibilities there. Third: do I really have to dig around trying to find something useful? All the data is there, the software should tell me what is useful or interesting. It's definitely a hard problem, but I would throw money at software that could send me this email: "Looks like users who experienced event "ABC" also performed your highest priority event "Signup" at a 13% higher rate. This observation is 99% confident." Of course, you'd need to investigate a littler deeper to see if that isn't just a fluke or something stupidly obvious (like: people who view a page signup at a higher rate than those who don't), but at least I might learn something. I know this is certainly a pipe dream as of today, but I vow to shower someone with money if they can do this. In my opinion, the next generation of analytics software won't just have more bells and whistles, it will fundamentally shorten the time to some sort of real "AHA!" insight.

cwilson 14y ago

The first most important metric: When it goes up and to the right, it means your company doesn't die. Death can be alluded in many different ways, depending on the nature of your business, but it usually means making money, hitting a milestone that allows you to raise money, acquiring users, and so forth. Once your company evolves, and has beaten death for the foreseeable future, a new set of metrics come into play and are now the most important. These may be similar to the death-evading set above, or they may be something new like increasing revenue, retaining users, increasing engagement, optimizing CLV, and so forth. The point? If a metric means your company is not dying, or your company is growing, it matters. If it doesn't fit into those buckets? Probably doesn't matter.

mattmaroon 14y ago

A little ironic coming from the firm that invested in Groupon, inventor of the "Adjusted consolidated segment operating income".

evolve2k 14y ago

I would have liked the article to pony up what the companies own 'non BS' metrics are. Seemed a little flat to 'call BS' on everyone else then not deliver the goods itself.

kevinyun 14y ago

People will always be measuring everything by the numbers. Look at your number of Facebook friends, your LinkedIn connections, your GPA in college, your annual income. It's always a rational argument to say that bigger is better. In a world where metrics are quantified like this, this is the natural way of thinking. It's going to be difficult to seep through the bullshit metrics, especially when you're such a small startup with other numbers that are too embarrassing to display to the world -- but, really, when you think about it this is the only way to find and fix the problems that you have.

mmaunder 14y ago

Everything except bottom line profitability is a bullshit metric.

wildwood 14y ago
Until you can't make payroll because the deals that you booked as profit two months ago haven't paid you any money yet. Then cash flow suddenly becomes essential.

robryan 14y ago

An example in the article, number of pictures uploaded on Instagram, it is not clear that this is a good metric, at least until they work out a way for each additional image to bring in more revenue.

endlessvoid94 14y ago

Gaming companies have been doing this already for years.

deleted 14y ago

[deleted]

mrdavidjcole 14y ago

This article inspired me to create a twitter handle to highlight the best(?) examples of this: http://twitter.com/bullshitmetrics http://twitter.com/bullshitmetrics Anything recent come to mind?

JacobJans 14y ago

Analytics only matter if they help you make better decisions. Anything else, is, indeed, just vanity.

fidanov 14y ago

So they call to replace one bullshit metrics with other bullshit metrics. Nice. Not once in this article there were the words "Profit" or "Revenue" and yet they talk about business and companies. Here is a surprise for you. Real companies are interested in real profits from real revenue and you count revenue and profit only after the money is actually in your bank account. Otherwise one day you get the kind of terms of service just like Instragram these days.

marcuskaz 14y ago
Yeh, if there is one true metric it would be profit

majani 14y ago

Actually, I find that a lot of the metrics mentioned like uniques and pageviews are the ones that affect revenue for ad-driven startups. That's as real as it gets with metrics. When it comes to ad-driven startups, you could actually say that the "real" metrics suggested, like active users and engagement, are the actual bullshit metrics. For many ad companies like Google(search) or YouTube, high engagement by the users adds nothing to the bottom line.