3 ms·

As someone with little experience in this area, I appreciated the step-by-step examples of cleaning and transforming the data into a suitable vector format for

As someone with little experience in this area, I appreciated the step-by-step examples of cleaning and transforming the data into a suitable vector format for the training step. Very nice! Is it typical that ~2000 samples (1000 leads, 1000 non-leads) is "enough"? I see that accuracy on the training set was ~86% and that the results are "starting to become useful for our sales team", but I would have thought more samples would have been needed. (I guess that since you're always collecting more real-world data you can continue to train the model and so it should get better?)

inigoesdr 10y ago
The author mentioned that 2000 was the minimum, and they intend to keep training the model with future data.
mrborgen 10y ago
2000 samples is what we needed to get going yes. We're also noticing variable accuracy on different real world dataset. Meaning the algorithm is biased towards our current dataset. Getting more data is one of our top priorities going forward with this project.
- stygiansonic 10y ago
  A follow-up question: In your experience, what's the cost difference between: 1) Mis-identifying a lead as a non-lead. 2) Mis-identifying a non-lead as a lead. I would guess that (1) would be more costly if non-leads go to somewhere where they aren't followed-up on. But I'd appreciate the insight. Thanks again!
  - chipsanddip 10y ago
    Hi, as Per that wrote this is a developer while I do the sales that come out of this, I'll answer this: Mis-identifying a lead as a non-lead is potentially loosing out on a big deal that can make or break your company. You never know what email will lead to a quickly closed $30k ARR sale, which are golden for any SaaS startup. The reverse has almost no consequences unless you're really going to town with the emailing and end up being flagged for spam. Usually people just ignore you (not so smart) or write back that it's not relevant.
  - mrborgen 10y ago
    Your assumption is right, in terms of business cost, we'd rather see a false positive than a false negative. I don't think the scikit learn algorithm differentiates between the two types of errors, in terms of cost. Though it seems to give less false negatives than false positives overall, when testing on new datasets. I'll put in the f1 score in the article when I have time.
  - SixSigma 10y ago
    Here's a whole post on that subject (or it might be Part 1 :) http://blog.mldb.ai/blog/posts/2016/04/ml-meets-economics2/ http://blog.mldb.ai/blog/posts/2016/04/ml-meets-economics2/
    - Lanzaa 10y ago
      This is exactly the link I was going to post. I think it does a great job showing that accuracy is not always the most appropriate metric for a model.
taneq 10y ago
What's "enough" depends entirely on what you're doing with it and how much accuracy you require. In this case, 86% was enough to improves their batting average and so was worth using. If they were trying to detect pedestrians from a self-driving car then they'd need a lot more accuracy and so a lot more training data.
- visarga 10y ago
  86% accurate should be a solid 7x speedup compared to manual selection, not to mention that the algo could also pre-populate description fields and only require a visual check-up instead of manual typing.
denzil_correa 10y ago
> Is it typical that ~2000 samples (1000 leads, 1000 non-leads) is "enough"? In terms of practical machine learning, plotting learning curves are a great way to know if samples are "enough" [0, 1]. If your algorithm under fits (high bias) both errors (training and validation) will be high and in this case, adding training samples would not help. [0] http://www.astroml.org/sklearn_tutorial/practical.html http://www.astroml.org/sklearn_tutorial/practical.html [1] http://blog.fliptop.com/blog/2015/03/02/bias-variance-and-overfitting-machine-learning-overview/ http://blog.fliptop.com/blog/2015/03/02/bias-variance-and-ov...
selectron 10y ago
Statistical uncertainty grows like sqrt(N), and percent statistical uncertainty shrinks like sqrt(N) / N. So 2000 samples has a percent statistical uncertainty of about 2% - this is likely on par with or smaller than the systematic uncertainty (errors which don't depend on the size of the data). This means 2000 samples is probably fine for what they are trying to do, especially because the dataset is balanced with each class having 1000 samples. However the more data you have the better your model will do.

inigoesdr 10y ago

The author mentioned that 2000 was the minimum, and they intend to keep training the model with future data.

mrborgen 10y ago

2000 samples is what we needed to get going yes. We're also noticing variable accuracy on different real world dataset. Meaning the algorithm is biased towards our current dataset. Getting more data is one of our top priorities going forward with this project.

stygiansonic 10y ago
A follow-up question: In your experience, what's the cost difference between: 1) Mis-identifying a lead as a non-lead. 2) Mis-identifying a non-lead as a lead. I would guess that (1) would be more costly if non-leads go to somewhere where they aren't followed-up on. But I'd appreciate the insight. Thanks again!
- chipsanddip 10y ago
  Hi, as Per that wrote this is a developer while I do the sales that come out of this, I'll answer this: Mis-identifying a lead as a non-lead is potentially loosing out on a big deal that can make or break your company. You never know what email will lead to a quickly closed $30k ARR sale, which are golden for any SaaS startup. The reverse has almost no consequences unless you're really going to town with the emailing and end up being flagged for spam. Usually people just ignore you (not so smart) or write back that it's not relevant.
- mrborgen 10y ago
  Your assumption is right, in terms of business cost, we'd rather see a false positive than a false negative. I don't think the scikit learn algorithm differentiates between the two types of errors, in terms of cost. Though it seems to give less false negatives than false positives overall, when testing on new datasets. I'll put in the f1 score in the article when I have time.
- SixSigma 10y ago
  Here's a whole post on that subject (or it might be Part 1 :) http://blog.mldb.ai/blog/posts/2016/04/ml-meets-economics2/ http://blog.mldb.ai/blog/posts/2016/04/ml-meets-economics2/
  - Lanzaa 10y ago
    This is exactly the link I was going to post. I think it does a great job showing that accuracy is not always the most appropriate metric for a model.

taneq 10y ago

What's "enough" depends entirely on what you're doing with it and how much accuracy you require. In this case, 86% was enough to improves their batting average and so was worth using. If they were trying to detect pedestrians from a self-driving car then they'd need a lot more accuracy and so a lot more training data.

visarga 10y ago
86% accurate should be a solid 7x speedup compared to manual selection, not to mention that the algo could also pre-populate description fields and only require a visual check-up instead of manual typing.

denzil_correa 10y ago

> Is it typical that ~2000 samples (1000 leads, 1000 non-leads) is "enough"? In terms of practical machine learning, plotting learning curves are a great way to know if samples are "enough" [0, 1]. If your algorithm under fits (high bias) both errors (training and validation) will be high and in this case, adding training samples would not help. [0] http://www.astroml.org/sklearn_tutorial/practical.html http://www.astroml.org/sklearn_tutorial/practical.html [1] http://blog.fliptop.com/blog/2015/03/02/bias-variance-and-overfitting-machine-learning-overview/ http://blog.fliptop.com/blog/2015/03/02/bias-variance-and-ov...

selectron 10y ago

Statistical uncertainty grows like sqrt(N), and percent statistical uncertainty shrinks like sqrt(N) / N. So 2000 samples has a percent statistical uncertainty of about 2% - this is likely on par with or smaller than the systematic uncertainty (errors which don't depend on the size of the data). This means 2000 samples is probably fine for what they are trying to do, especially because the dataset is balanced with each class having 1000 samples. However the more data you have the better your model will do.