3 ms·
Sample size is never directly mentioned, but: > At Triplebyte we evaluate hundreds of engineers every week. > I pulled the editor / language data for every Tr
by jawns 9y ago
Sample size is never directly mentioned, but:
> At Triplebyte we evaluate hundreds of engineers every week.
> I pulled the editor / language data for every Triplebyte interview over the last year.
Based on that, if we suppose "hundreds" means "low hundreds," this suggests a sample size of 5K-15K.
That's not a bad sample size for the total population.
But if 35% use Python and only 5% use Ruby, then you're comparing a sample of 1,750 (on the low end) to a sample of 250. It wouldn't surprise me if the Ruby data is more shaky than the Python data, and thus it's not a fair comparison.
Same goes for a lot of the other metrics where the sample sizes vary.
~ Disclaimer: I'm no statistician (but I play one on Correlated.org).
- tedsanders 9y agoBe careful equating 'evaluate' with 'interview', when TripleByte has incentive to obfuscate. I suspect hundreds of engineers may apply through Triplebyte each week, but they may interview only a small fraction of those. If true, this makes your point about small sample size even more important. I was disappointed their charts didn't have confidence intervals or count numbers. Makes it harder to guess at how the numbers generalize.
- jayant123 9y agoAlso, my guess is that the success rate for interviews would be fairly low, making any comparison of the success rates for different languages/editors even more statistically shaky. They should definitely publish the total sample size and success percentage for these results to be meaningful. Example: Say they hired 5 out of a 100 applicants who used Python, and 2 out of 20 applicants who used Ruby. That's a 5% success rate vs a 10% success rate, a 100% increase. I'd hardly call that a significant result though. Disclaimer: I'm no statistician either