4 ms·
Articles like this really bother me because its really unclear what data here is significant (if any) since all of the data is just quoted as a % without standa
by jamiequint 11y ago
Articles like this really bother me because its really unclear what data here is significant (if any) since all of the data is just quoted as a % without standard error and the sample sizes are very small:
e.g. on comparing companies that do better. You could have a data set of 150 companies whose exit performance (or current value) looks something like this (numbers in millions):
5000,1000,500,400,200,200,100,50,50,30,30,20,20,20,15,15,15,15,0,0,0,0,0,0,0 (repeated 6x to get 150 data points)
Now compare that against a data set that is those exact numbers divided in half:
2500,500,250,200,100,100,50,25,25,15,15,10,10,10,7.5,7.5,7.5,7.5,0,0,0,0,0,0,0 (repeated 6x to get 150 data points)
If you compare these data sets with a Two Sample T-Test you have to go down to 91% confidence to get a significant result (http://www.evanmiller.org/ab-testing/t-test.html#!307.2/986.137554/150;153.6/493.068777/150@91 http://www.evanmiller.org/ab-testing/t-test.html#!307.2/986....)
That may not sound that bad, but now add a super-unicorn to each one of those data sets, a $20B exit. Now the differences aren't even significant at 80% confidence.
e.g. in Item 7 about technical co-founders: "consumer companies with at least one technical co-founder underperform completely non-technical teams by 31%"
Lets say that First Round has 150 consumer businesses and we're just going to look at a binary outcome of something like "valued over $50m". Now lets say that 100 of these consumer companies have technical co-founders and 50 are completely non-technical. Say 40% of the non-technical teams are "successful" by the $50m metric. That means that 30.5% of the technical teams are successful (if they are doing 31% worse by the numbers in the article since 40%/1.31 = 30.5%). That's not a significant result at 80% confidence (http://www.evanmiller.org/ab-testing/chi-squared.html#!31/100;20/50@80 http://www.evanmiller.org/ab-testing/chi-squared.html#!31/10...)
I understand why they published the piece and think it will get a lot of reads, but really wish I could read a version with statistically relevant insights instead.
- whitehouse3 11y agoConsidering... > Venture capitalists are constantly telling the entrepreneurs they invest in to make data-driven decisions. ...the scant amount of real data presented is surprising.
- mfoy_ 11y ago"Data-driven" as in "we looked at some data and made a decision." Whether that was good data or bad data, lots of data or just a couple points, is out of scope. :p
- danieltillett 11y agoThe data they are looking for is can you show more progress so they can come in as late as possible at the lowest valuation. I don’t blame VCs for doing this, but I wish they would be a bit more honest about what they are doing.
- danieltillett 11y agoGiven they only have 300 data points and they have tested at least 10 hypothesises it is almost certain that they have been engaged in a massive example of data dredging [1]. The chance that we can draw anything from this study is next to zero. If this is the sort of data driven work that VCs generate then it might be best that they just tell founders to be data driven and not actually engage in it themselves. 1. https://en.wikipedia.org/wiki/Data_dredging https://en.wikipedia.org/wiki/Data_dredging