4 ms·

How VCs See Your KPIs

46 points by dpaluy 8y ago

peterwwillis 8y ago
> I’m a big fan of making KPIs as visible as possible when the time is right, like having big monitors with your KPIs displayed all around the office so everyone can see them. I've been in tech 14 years, and I've never worked anywhere where a manager discussed KPIs with people on my level. But the companies I worked for also had billions in the bank, so maybe this just implies that companies of that size don't need engineers to look at KPIs. Get rid of the stupid TV dashboards and put up a decent Monet. It sucks working in a physical manifestation of a computer.
- edgarvaldes 8y ago
  Why do you think TV dashboards are stupid?
- mi100hael 8y ago
  > But the companies I worked for also had billions in the bank, so maybe this just implies that companies of that size don't need engineers to look at KPIs. Indeed. Things are a little different at a startup when there are ten of you sitting in one big room and the business lives or dies based on what each and every person does in that room.
- leesalminen 8y ago
  We just launched a TV dashboard (using Geckoboard), pulling data from a dozen sources. Everyone seems to dig it, and when a number gets noticeably bigger people feel good.
manishsharan 8y ago
Could someone share a few examples of KPIs,excluding the well know AARRR metrics, tracked by successful startups ?
rabidrat 8y ago
"And tracking towards the wrong goals can be even more counterproductive than failing to track anything at all." The author doesn't seem to recognize that KPIs would be subject to Goodhart's Law. ("When a measure becomes a target, it ceases to be a good measure.")
- tryitnow 8y ago
  Really? I think Goodhart's Law applies more to complex systems with many, many actors, like whole economies (which is what Goodhart originally applied it to). I am willing to bet that something like "retention" or "net new accounts" are relatively resistant to GL.
  - rabidrat 8y ago
    Maybe in a startup you're right, because it's too small for any kind of metric-hacking to go unnoticed and become systemic. But it definitely happens in larger companies; some might say that making the metric of revenue/profit a target is the source of our capitalistic ills today. https://money.cnn.com/2017/08/31/investing/wells-fargo-fake-accounts/index.html https://money.cnn.com/2017/08/31/investing/wells-fargo-fake-...
Alex3917 8y ago
> I’m a big fan of making KPIs as visible as possible when the time is right, like having big monitors with your KPIs displayed all around the office so everyone can see them. > If they don’t have a daily report or tell me it's going to take a few days, the conversation is over. So he's a big fan of having the company KPIs displayed on monitors throughout the office, but if anyone actually uses those monitors to get their info on KPIs then the conversation is over. Got it.
- EnFinlay 8y ago
  Both statements are saying the same thing - KPIs should be highly accessible and visible.
  - Alex3917 8y ago
    He’s saying he wouldn’t invest in any company that doesn’t have a daily email blast with the KPIs. But if you already have dashboards for everyone then surely a daily email is a nice to have rather than a need to have.
    - citrablue 8y ago
      Daily emails are a great notification method (the point of KPIs is to socialize them, and some people do better with push instead of pull). Further, emailed reports are a append-only log of previous reports, which has additional value. Whether that's worth the half day to setup is a different question.
3pt14159 8y ago
> preferably with a dashboard Nope. Daily / weekly / monthly plain text email or emailed spreadsheet is way better. The problem with dashboards is that when the underlying numbers change, they can change the figures for the past and there won't be a record of it. If it's in an email someone can bring up this discrepancy. Also, lots of startups waste a lot of time building a dashboard not realizing that they're now building two apps. "We don't need login, we'll just IP whitelist it." Then a VC wants to see the numbers. Then he wants to forward it to his partners. Now if your dashboard goes down the CEO starts freaking out about how it makes the company look during their raise. The rest of the advice is spot on though. Especially keeping the number of KPIs down. This isn't to rule out pulling in other sets of numbers, but those should be done by a Data Analyst and they should be one-time analyses that can be re-run by the analyst later if needed.
- moritzplassnig 8y ago
  We pushed data automatically into a Google Sheet (e.g. importdata() from Looker). The sheet had a column per month and I simply exported that sheet at the end of a month and shared it with our investors (together with an email update). Super easy, the Google Sheet "dashboard" was available to everybody in the company and our investors received the same information.
- iooi 8y ago
  > Daily / weekly / monthly plain text email or emailed spreadsheet is way better. > Also, lots of startups waste a lot of time building a dashboard not realizing that they're now building two apps. Setting up a Grafana dashboard is a lot easier than setting up crons that send emails. Auth is built in, including OAuth. It's also trivial to have your data adjust to time bounds that you can change on the interface. > when the underlying numbers change, they can change the figures for the past and there won't be a record of it. Any important numbers should be graphed on a time series chart, and you won't have to track down emails and compare across different tabs.
  - 3pt14159 8y ago
    > Setting up a Grafana dashboard is a lot easier than setting up crons that send emails. Auth is built in, including OAuth. Not really. It would take me about 5 minutes to setup an extra worker that sent emails for an existing app. Plus, it's a strategic decision to share numbers with a potential funder. That decision should be made each time, rather than allow ongoing analysis. > Any important numbers should be graphed on a time series chart, and you won't have to track down emails and compare across different tabs. I agree that important numbers should be graphed on a time series chart, what I'm saying is that sometimes changes in code can change numbers in the past. If each email has that month's spreadsheet, complete with numbers going into the past, and someone says "hey wait a minute, I thought churn in July 2017 was 4.5%, I remember because it was our worst month!" Then you can go back to see what the email said.
graycat 8y ago
"Key", in what sense "key"? There needs to be a good reason. Just applying the label is not good progress. (1) To go to the trouble to measure something and publicize it to employees, investors, etc., first need a good reason for measuring that something. E.g., why do we measure MPG, MPH, engine temperature, time since last oil change, ..., unemployment rate, labor participation rate, economic growth rate, inflation rate, prime rate, money supply, balance of payments, mortgage default rate, federal budget deficit/surplus? For each of these, and jointly for several of them, we have good reasons, from owning a car to macroeconomics. (2) When we seek to measure something, we should be clear on what in principle we are trying to measure and, then, pay due attention to the accuracy of our measurements, e.g., reliability and validity. The OP ignored (1) and (2). My wife, Ph.D. in mathematical sociology, understood reliability and validity and much more. Similarly for my brother, Ph.D. in political science. The social sciences are such indefinite fields that they have to be careful about what they are measuring, why, what the measurements mean, and the accuracy. As a graduate student, Ph.D. in applied math, I understood reliability and validity, and at least several, likely nearly all, my fellow graduate students and faculty did also. The OP is by a VC who wants "warm introductions". Well, likely the people he knows don't know about reliability or validity, and the people I know do. So, we won't have colleagues in common with no possibility of a "warm introduction" in either direction. E.g., one of my dissertation advisors was later President at CMU. I'm sure he knew about reliability and validity, but it sounds like we couldn't get a "warm introduction" via him in either direction. The Chair of my Ph.D. orals committee was a Member, US National Academy of Engineering, and I'm sure he knew about reliability and validity; same problem, not much chance of an introduction! Uh, the author of the OP is interested in "huge ideas". Hmm .... From people who don't know about reliability and validity? How about measure preserving in ergodic theory and applications to machine learning and computer security? My startup is supposed to be of high interest to nearly everyone on the Internet -- "huge" enough? But to evaluate the work, need to dig into and evaluate the crucial, core, original applied math secret sauce based on some advanced pure math prerequisites -- the OP author is qualified to evaluate such work? In what sense is he really interested in and able to evaluate or work with "huge ideas"? Or, if he invests and we have our first BoD meeting and his big contribution is to insist that we allocate time, money, and effort to lining the walls with big video screens with "key performance indicators" changing in real time, uh, to "motivate" the employees? Uh, might we have a study of the effect on the employees, motivate them, distract them, or irritate them? My a priori guess -- at best another case of the Hawthorne effect https://en.wikipedia.org/wiki/Hawthorne_effect https://en.wikipedia.org/wiki/Hawthorne_effect
loco5niner 8y ago
> As a four-time CEO, I had to learn early-on how to... For some reason, this line put a bad taste in my mouth. Probably an association with too many bad advice articles on the internet where the author felt the need to establish authority on the subject using a line like this rather than letting the article stand on its own merits.
davidivadavid 8y ago
So basically set up OKRs.

46 points by dpaluy 8y ago

> I’m a big fan of making KPIs as visible as possible when the time is right, like having big monitors with your KPIs displayed all around the office so everyone can see them. I've been in tech 14 years, and I've never worked anywhere where a manager discussed KPIs with people on my level. But the companies I worked for also had billions in the bank, so maybe this just implies that companies of that size don't need engineers to look at KPIs. Get rid of the stupid TV dashboards and put up a decent Monet. It sucks working in a physical manifestation of a computer.

edgarvaldes 8y ago
Why do you think TV dashboards are stupid?
mi100hael 8y ago
> But the companies I worked for also had billions in the bank, so maybe this just implies that companies of that size don't need engineers to look at KPIs. Indeed. Things are a little different at a startup when there are ten of you sitting in one big room and the business lives or dies based on what each and every person does in that room.
leesalminen 8y ago
We just launched a TV dashboard (using Geckoboard), pulling data from a dozen sources. Everyone seems to dig it, and when a number gets noticeably bigger people feel good.

manishsharan 8y ago

Could someone share a few examples of KPIs,excluding the well know AARRR metrics, tracked by successful startups ?

rabidrat 8y ago

"And tracking towards the wrong goals can be even more counterproductive than failing to track anything at all." The author doesn't seem to recognize that KPIs would be subject to Goodhart's Law. ("When a measure becomes a target, it ceases to be a good measure.")

tryitnow 8y ago
Really? I think Goodhart's Law applies more to complex systems with many, many actors, like whole economies (which is what Goodhart originally applied it to). I am willing to bet that something like "retention" or "net new accounts" are relatively resistant to GL.
- rabidrat 8y ago
  Maybe in a startup you're right, because it's too small for any kind of metric-hacking to go unnoticed and become systemic. But it definitely happens in larger companies; some might say that making the metric of revenue/profit a target is the source of our capitalistic ills today. https://money.cnn.com/2017/08/31/investing/wells-fargo-fake-accounts/index.html https://money.cnn.com/2017/08/31/investing/wells-fargo-fake-...

Alex3917 8y ago

> I’m a big fan of making KPIs as visible as possible when the time is right, like having big monitors with your KPIs displayed all around the office so everyone can see them. > If they don’t have a daily report or tell me it's going to take a few days, the conversation is over. So he's a big fan of having the company KPIs displayed on monitors throughout the office, but if anyone actually uses those monitors to get their info on KPIs then the conversation is over. Got it.

EnFinlay 8y ago
Both statements are saying the same thing - KPIs should be highly accessible and visible.
- Alex3917 8y ago
  He’s saying he wouldn’t invest in any company that doesn’t have a daily email blast with the KPIs. But if you already have dashboards for everyone then surely a daily email is a nice to have rather than a need to have.
  - citrablue 8y ago
    Daily emails are a great notification method (the point of KPIs is to socialize them, and some people do better with push instead of pull). Further, emailed reports are a append-only log of previous reports, which has additional value. Whether that's worth the half day to setup is a different question.

3pt14159 8y ago

> preferably with a dashboard Nope. Daily / weekly / monthly plain text email or emailed spreadsheet is way better. The problem with dashboards is that when the underlying numbers change, they can change the figures for the past and there won't be a record of it. If it's in an email someone can bring up this discrepancy. Also, lots of startups waste a lot of time building a dashboard not realizing that they're now building two apps. "We don't need login, we'll just IP whitelist it." Then a VC wants to see the numbers. Then he wants to forward it to his partners. Now if your dashboard goes down the CEO starts freaking out about how it makes the company look during their raise. The rest of the advice is spot on though. Especially keeping the number of KPIs down. This isn't to rule out pulling in other sets of numbers, but those should be done by a Data Analyst and they should be one-time analyses that can be re-run by the analyst later if needed.

moritzplassnig 8y ago
We pushed data automatically into a Google Sheet (e.g. importdata() from Looker). The sheet had a column per month and I simply exported that sheet at the end of a month and shared it with our investors (together with an email update). Super easy, the Google Sheet "dashboard" was available to everybody in the company and our investors received the same information.
iooi 8y ago
> Daily / weekly / monthly plain text email or emailed spreadsheet is way better. > Also, lots of startups waste a lot of time building a dashboard not realizing that they're now building two apps. Setting up a Grafana dashboard is a lot easier than setting up crons that send emails. Auth is built in, including OAuth. It's also trivial to have your data adjust to time bounds that you can change on the interface. > when the underlying numbers change, they can change the figures for the past and there won't be a record of it. Any important numbers should be graphed on a time series chart, and you won't have to track down emails and compare across different tabs.
- 3pt14159 8y ago
  > Setting up a Grafana dashboard is a lot easier than setting up crons that send emails. Auth is built in, including OAuth. Not really. It would take me about 5 minutes to setup an extra worker that sent emails for an existing app. Plus, it's a strategic decision to share numbers with a potential funder. That decision should be made each time, rather than allow ongoing analysis. > Any important numbers should be graphed on a time series chart, and you won't have to track down emails and compare across different tabs. I agree that important numbers should be graphed on a time series chart, what I'm saying is that sometimes changes in code can change numbers in the past. If each email has that month's spreadsheet, complete with numbers going into the past, and someone says "hey wait a minute, I thought churn in July 2017 was 4.5%, I remember because it was our worst month!" Then you can go back to see what the email said.

graycat 8y ago

"Key", in what sense "key"? There needs to be a good reason. Just applying the label is not good progress. (1) To go to the trouble to measure something and publicize it to employees, investors, etc., first need a good reason for measuring that something. E.g., why do we measure MPG, MPH, engine temperature, time since last oil change, ..., unemployment rate, labor participation rate, economic growth rate, inflation rate, prime rate, money supply, balance of payments, mortgage default rate, federal budget deficit/surplus? For each of these, and jointly for several of them, we have good reasons, from owning a car to macroeconomics. (2) When we seek to measure something, we should be clear on what in principle we are trying to measure and, then, pay due attention to the accuracy of our measurements, e.g., reliability and validity. The OP ignored (1) and (2). My wife, Ph.D. in mathematical sociology, understood reliability and validity and much more. Similarly for my brother, Ph.D. in political science. The social sciences are such indefinite fields that they have to be careful about what they are measuring, why, what the measurements mean, and the accuracy. As a graduate student, Ph.D. in applied math, I understood reliability and validity, and at least several, likely nearly all, my fellow graduate students and faculty did also. The OP is by a VC who wants "warm introductions". Well, likely the people he knows don't know about reliability or validity, and the people I know do. So, we won't have colleagues in common with no possibility of a "warm introduction" in either direction. E.g., one of my dissertation advisors was later President at CMU. I'm sure he knew about reliability and validity, but it sounds like we couldn't get a "warm introduction" via him in either direction. The Chair of my Ph.D. orals committee was a Member, US National Academy of Engineering, and I'm sure he knew about reliability and validity; same problem, not much chance of an introduction! Uh, the author of the OP is interested in "huge ideas". Hmm .... From people who don't know about reliability and validity? How about measure preserving in ergodic theory and applications to machine learning and computer security? My startup is supposed to be of high interest to nearly everyone on the Internet -- "huge" enough? But to evaluate the work, need to dig into and evaluate the crucial, core, original applied math secret sauce based on some advanced pure math prerequisites -- the OP author is qualified to evaluate such work? In what sense is he really interested in and able to evaluate or work with "huge ideas"? Or, if he invests and we have our first BoD meeting and his big contribution is to insist that we allocate time, money, and effort to lining the walls with big video screens with "key performance indicators" changing in real time, uh, to "motivate" the employees? Uh, might we have a study of the effect on the employees, motivate them, distract them, or irritate them? My a priori guess -- at best another case of the Hawthorne effect https://en.wikipedia.org/wiki/Hawthorne_effect https://en.wikipedia.org/wiki/Hawthorne_effect

loco5niner 8y ago

> As a four-time CEO, I had to learn early-on how to... For some reason, this line put a bad taste in my mouth. Probably an association with too many bad advice articles on the internet where the author felt the need to establish authority on the subject using a line like this rather than letting the article stand on its own merits.

davidivadavid 8y ago

So basically set up OKRs.