4 ms·

2000 samples is what we needed to get going yes. We're also noticing variable accuracy on different real world dataset. Meaning the algorithm is biased towards

2000 samples is what we needed to get going yes. We're also noticing variable accuracy on different real world dataset. Meaning the algorithm is biased towards our current dataset. Getting more data is one of our top priorities going forward with this project.

stygiansonic 10y ago
A follow-up question: In your experience, what's the cost difference between: 1) Mis-identifying a lead as a non-lead. 2) Mis-identifying a non-lead as a lead. I would guess that (1) would be more costly if non-leads go to somewhere where they aren't followed-up on. But I'd appreciate the insight. Thanks again!
- chipsanddip 10y ago
  Hi, as Per that wrote this is a developer while I do the sales that come out of this, I'll answer this: Mis-identifying a lead as a non-lead is potentially loosing out on a big deal that can make or break your company. You never know what email will lead to a quickly closed $30k ARR sale, which are golden for any SaaS startup. The reverse has almost no consequences unless you're really going to town with the emailing and end up being flagged for spam. Usually people just ignore you (not so smart) or write back that it's not relevant.
- mrborgen 10y ago
  Your assumption is right, in terms of business cost, we'd rather see a false positive than a false negative. I don't think the scikit learn algorithm differentiates between the two types of errors, in terms of cost. Though it seems to give less false negatives than false positives overall, when testing on new datasets. I'll put in the f1 score in the article when I have time.
- SixSigma 10y ago
  Here's a whole post on that subject (or it might be Part 1 :) http://blog.mldb.ai/blog/posts/2016/04/ml-meets-economics2/ http://blog.mldb.ai/blog/posts/2016/04/ml-meets-economics2/
  - Lanzaa 10y ago
    This is exactly the link I was going to post. I think it does a great job showing that accuracy is not always the most appropriate metric for a model.

stygiansonic 10y ago

A follow-up question: In your experience, what's the cost difference between: 1) Mis-identifying a lead as a non-lead. 2) Mis-identifying a non-lead as a lead. I would guess that (1) would be more costly if non-leads go to somewhere where they aren't followed-up on. But I'd appreciate the insight. Thanks again!

chipsanddip 10y ago
Hi, as Per that wrote this is a developer while I do the sales that come out of this, I'll answer this: Mis-identifying a lead as a non-lead is potentially loosing out on a big deal that can make or break your company. You never know what email will lead to a quickly closed $30k ARR sale, which are golden for any SaaS startup. The reverse has almost no consequences unless you're really going to town with the emailing and end up being flagged for spam. Usually people just ignore you (not so smart) or write back that it's not relevant.
mrborgen 10y ago
Your assumption is right, in terms of business cost, we'd rather see a false positive than a false negative. I don't think the scikit learn algorithm differentiates between the two types of errors, in terms of cost. Though it seems to give less false negatives than false positives overall, when testing on new datasets. I'll put in the f1 score in the article when I have time.
SixSigma 10y ago
Here's a whole post on that subject (or it might be Part 1 :) http://blog.mldb.ai/blog/posts/2016/04/ml-meets-economics2/ http://blog.mldb.ai/blog/posts/2016/04/ml-meets-economics2/
- Lanzaa 10y ago
  This is exactly the link I was going to post. I think it does a great job showing that accuracy is not always the most appropriate metric for a model.