8 ms·

You can probably use deep learning even if you don't have a lot of data

417 points by deepnotderp 9y ago

irq11 9y ago
...but why would you? The fact that there are people "getting their jimmies up" on questions of training massively paramterized statistical models on tiny amounts of data should tell you exactly where we are on the deep-learning hype cycle. For a while there, SVMs were the thing, but now the True Faithful have moved on to neural networks. The argument this writer is making is essentially: "yes, there are lots of free parameters to train, and that means that using it with small data is a bad idea in general, but neural networks have overfitting tools now and they're flexible so you should use them with small data anyway." This is literally the story told by the bulleted points. Neural networks are a tool. Don't use the tool if it isn't appropriate to your work. Maybe you can find a way to hammer a nail with a blowtorch, but it's still a bad idea.
- RandomInteger4 9y ago
  It might be that people want to learn to use deep learning, but they find the NIST AND other sets boring and want to learn on a problem they find interesting, but still produce something that seems to work. Plus, learning to learn how to learn on less can only help the field of learning. That's the goal of one shot learning right?
  - irq11 9y ago
    Obviously, you do whatever you like when you're playing around, but when you find yourself in that situation (i.e. "I want to learn tool X, but all of my problems are inappropriate for X"), it's an indication that you're misusing/misunderstanding X. Again, with the forced metaphors: if I buy a new welder, I'm probably suddenly very keen on welding things. That doesn't mean I'm learning how to weld. A big (the biggest?) part of learning a tool is learning when to use it.
    - jfoutz 9y ago
      In my very limited experience, i hear about cool technology X, and have crazy fantasies about how it actually works. Picking up X and applying it to a problem, no matter how poorly informed, always taught me more about X than anything else. When you get your welder, you kinda have to weld a lot of things to see when it's effective and when it's not. Everybody gets a free pass with the first few months with a new toy. The only way to learn when it's appropriate is to screw up a few times.
      - irq11 9y ago
        Yes, sure. Playing with something is a good way to learn that thing. But if you find yourself saying "I really like using this backhoe, but I'm finding that most of my hole-digging problems are too small for a backhoe. Is there a blog post on using backhoes to make sandcastles?", you have perhaps wandered off the path to enlightenment. I'm really enjoying these metaphors though.
        candiodari 9y ago
        https://www.youtube.com/watch?v=nltqM_Q-cwg https://www.youtube.com/watch?v=nltqM_Q-cwg
      - diegoperini 9y ago
        Here is an approach that worked for us well in our startup, that may help other teams who read this thread. We learned it through making a few mistakes in our decision making process. Whenever we have a new feature which cannot be implemented using existing frameworks, tools, in house technology or existing expertise, we inform the managers to add an extra 2 week to our schedule to evaluate as many options as we can. It is really hard to fit a lot of tools in that time so all the team picks up the work, even the ones that has no past experience or theoretical knowledge on the topic. It actually helps to have those ones in the research group. They are often the ones to be able to tell "since i have no idea in the expertise, i instead searched for this company who apparently ditched this tool because they suffered from this and that". Others who try to acquire the theory on the other hand is able to argue like "X seems to be better than Y". Once we have enough Xs, we already have use cases of X that is proven to be useless. In that attempt, mostly there remain only one X or even none. We either pick up that remaining X or an X that is less scary but a little more boring. Boring is good because a team can argue on a boring thing more easily. Those arguments produce quality code that remains to be used more than a year. A year or two is enough to allocate more research time for the topic, which eventually helps to find or implement a tool-set that can live much longer. Here are some examples we used more than a year and ditched or soon will ditch: Stock Tesseract Server Side OCR -> Properly Trained Server Side Tesseract + Image preprocessing on the mobile device Rethinkdb change feeds -> Postgres LISTEN Bluebird.js -> ReactiveX Ubuntu -> CentOS Forever -> Docker Edit: The reason why I am posting under your comment is in some cases, mistakes can hurt a company in a way that is unrecoverable. My advice my not apply to pet projects.
- simcop2387 9y ago
  Small question here, what are SVNs in this context?
  - irq11 9y ago
    support vector machines. a non-linear classifier that was the hot thing in the AI world before neural networks became practical.
    - candiodari 9y ago
      They're great because they immediately go to the best solution. One epoch is all you need, no matter the size of the SVM. They suck because they're limited to one layer. But there's a good case to be made that machine learning introductions should always be done like this : linear classifier (when that works) -> SVM (when that works) -> NN -> Deep learning.
      - irq11 9y ago
        I contend that all real work should be done this way. And don't forget things like bayes' nets and random forests, which get no love. in other words, KISS.
        snovv_crash 9y ago
        To go even simpler, KNN also works really well if you can get a good weighting for your inputs and measurement noise is low. And KNN works with online/soft-RT datasets as well, where constant learning is required.
        CuriouslyC 9y ago
        I have done a lot of work with classification algorithms, and KNN doesn't get nearly the love it deserves. It rarely turns in exceptional performance, but when used with mahalinobis distance it is extremely robust.
        9y ago
- rahimnathwani 9y ago
  s/SVN/SVM
  - irq11 9y ago
    oops, typo.
- tsiki 9y ago
  I think you're missing the point. The jimmies are getting rustled up because someone provides false information about the performance to make his own argument seem better. This is something anyone should be against.
  - irq11 9y ago
    but they didn't. the writer makes an unconvincing claim that the original post was wrong. the data presented shows only that if you try really hard and get lucky enough, you can probably do as well as a simple regression in this case. the author himself admits that deep learning is probably misapplied here, and that training with such small data is difficult, at best. which again brings us back to the important question (i.e. the point being made by the original post): why would you ever do this?
    - dbecker 9y ago
      if you try really hard and get lucky enough, you can probably do as well as a simple regression in this case. Maybe you aren't familiar with deep learning: but this isn't "trying really hard." This is doing basic stuff that anyone using deep learning probably knows. And deep learning doesn't just "do as well" as the simpler model. It does meaningfully better at all sample sizes.
      - blueblob 9y ago
        Define "meaningfully better." Perhaps you mean statistically significantly better? It may have better accuracy, but it has significantly less interpretability. What does it capture that regression couldn't capture? At least with regression you can interpret the relationship between all of the variables and their relative importance by looking at the coefficients of the regression. With deep learning, the best approaches for explanation are to train another model at the same time that you use for explanation. Additionally, it was proven that a perceptron can learn any function, so in some senses the "deep" part of deep learning is because people are being lazy because at least you could get a better interpretation of the perceptron. I don't mean to imply that there's not a place for deep learning, but I think this isn't a great refutation of the argument that fitting a deep model is somewhat inappropriate for a small dataset.
rkaplan 9y ago
This post doesn't even mention the easiest way to use deep learning without a lot of data: download a pretrained model and fine-tune the last few layers on your small dataset. In many domains (like image classification, the task in this blog post) fine-tuning works extremely well, because the pretrained model has learned generic features in the early layers that are useful for many datasets, not just the one trained on. Even the best skin cancer classifier [1] was pretrained on ImageNet. [1]: http://www.nature.com/articles/nature21056 http://www.nature.com/articles/nature21056
- option_greek 9y ago
  Does transfer learning apply to seq2seq as well ?
- a_bonobo 9y ago
  This is how the great fast.ai course begins - download VGG16, finetune the top layer with a single dense layer, get amazing results. The second or third class shows how to make the top layers a bit more complex to get even better accuracy.
  - alexcnwy 9y ago
    Can't recommend the course highly enough!
    - manibatra 9y ago
      Same!! On lesson 2 and already feel like I know so much! The top down approach to learning is great! Will actually read the book "Making Learning Whole" that inspired them to follow this approach.
    - Steeeve 9y ago
      The 30 minute overview has me excited about going through it. I really like how they have structured it. Thanks to all for recommending it!
    - yamaneko 9y ago
deepnotderp 9y ago
Also, not to mention transfer learning is a big one. Case in point, the silicon valley "not hotdog" classifier which they stopped at hotdog or not due to lack of training data when in reality they could've just used a pre trained net on imagenet. Lol, I was literally cringing through that episode so hard xD
- minimaxir 9y ago
  The developer behind the real-world app mentioned why they did not use pretraining, and why you can't always use pretraining: https://news.ycombinator.com/item?id=14347513 https://news.ycombinator.com/item?id=14347513 > We ended up with a custom architecture trained from scratch due to runtime constraints more so than accuracy reasons (the inference runs on phones, so we have to be efficient with CPU + memory), but that model also ended up being the most accurate model we could build in the time we had. (With more time/resources I have no doubt I could have achieved better accuracy with a heavier model!)
  - deleted 9y ago
    [deleted]
  - deepnotderp 9y ago
    You could still have pre trained the custom architecture.
shadowmint 9y ago
You probably can... but is that really the issue? I think the problem isnt that you cant solve problems with small amounts of data; its that you can't solve 'the problem' at a small scale and then just apply that solution at large scale... and that's not what people want or expect. People expect that if you have an industrial welder than can assemble areoplanes (apparently), then you should easily be able to check it out by welding a few sheets of metal together, and if it welds well on a small scale, it should be representative of how well it welds entire vehicles. ...but thats not how DNN models work. Each solution is a specific selection of hyperparameters for the specific data and specific shape of that data. As we see here, specific even to the volume of data available. It doesnt scale up and it doesn't scale down. To solve a problem you just have to sort of.... just mess around with different solutions until you get a good one. ...and even then, you've got no really strong proof your solution is good; just that its better than the other solutions you've tried. Thats the problem; its really hard to know when DNN are the wrong choice, vs. you're just 'doing it wrong'
zensavona 9y ago
Maybe what you should do is deep learn some data and then do some deep learning with your deep learned [deep] data. Deep.
aub3bhat 9y ago
This debate is meaningless for several reasons: 1. The original argument is a strawman. What do they mean by "data"? Is it survey results, microarrays, "Natural" images, "Natural" language text or readings from an audio sensor? No ML researcher would argue that applying complex models such as CNNs is useful for say survey data. But if the data is domain specific, such as Natural Language text, images taken in particular context, etc. using a model and parameters that exhibit good performance is a good starting point. 2. Unlike how statisticians view data (as say a matrix of measurements or "Data Frame"), machine learning researchers view data at a higher level of representation. E.g. An image is not merely a matrix but rather an object that can be augmented by horizontally flipping, changing contrast etc. In case of text you can render characters using different fonts, colors etc. 3. Finally the example used in the initial blog post, of predicting 1 vs 0 from images is itself incorrect. Sure a statistician would "train" a linear model to predict 1 vs 0, however I as an ML researcher would NOT train any model at all and would just use [1] which has state of the art performance in character recognition in widely varying conditions. When you have only 80 images, why risk assuming that they are sampled in an IID manner from population, instead why not simply use a model thats trained on far larger population. Now the final argument might look suspicious but its crucial in understanding the difference between AI/ML/CV vs Statistics. In AI/ML/CV the assumption is that there are higher level problems (Character recognition, Object recognition, Scene understanding, Audio recognition) which when solved enable us to apply them in wide variety of situations where they appear. Thus when you encounter a problem like digit recognition the answer an ML researcher would give is to use a state of the art model. [1] https://github.com/bgshih/crnn https://github.com/bgshih/crnn
- Sean1708 9y ago
  Are you using some accepted definition of Statistics that I'm unaware of? Because I always thought Machine Learning was a branch of Statistics.
  - aub3bhat 9y ago
    >> Machine Learning was a branch of Statistics Thats incorrect. """ Machine learning is the subfield of computer science that, according to Arthur Samuel in 1959, gives "computers the ability to learn without being explicitly programmed." """ -- Wikipedia
j7ake 9y ago
Of course you can use it but does it perform better than "shallow" methods such as Gaussian processes, SVMs, and multivariate linear regression ? either through theoretical or empirical evidence ?
- minimaxir 9y ago
  The original post used a linear regression (and apparently misimplemented an intended logistic regression); this post sees better results at all sample sizes with a proper deep learning approach.
known 9y ago
aka Wisdom of Crowds
ska 9y ago
It's an interesting conversation but really weakened by failing to take on the generalization problem head on. This is something I see in a lot of discussions about deep nets on smaller data sets, whether transfer or not. The answer "it's built in" is particularly unsatisfying. The plots shown certainly should raise the spectre of overtraining - and rather than handwaving about techniques to avoid it, it would be great to see a detailed discussion of how you convince yourself (i.e. with additional data) that you are reasonably generalizable. Deep learning techniques are no panacea here.
m3kw9 9y ago
Ppl keep saying a lot without even thinking is a relative term. For images a lot means enough to get to x percentage accuracy, for OCR for a single font, a lot means 26 letters + special chars and numbers. Stop saying a lot blindly like every one underatands.
erickscott 9y ago
Andrew Beam's post offered very persuasive evidence that Jeff Leek's intuition (deep learning yields poor performance with small sample sizes) is incorrect. The error bars and the consistent trend of higher accuracy with a properly implemented deep learning model, particularly with smaller sample sizes, is devastating to Leek's original post. I think this is a fantastic example of the speed and self-correcting nature of science in the internet-age. As an aside, @simplystats blocked me on Twitter, which I assume is in response to this tweet: https://twitter.com/ErickRScott/status/871586233599893505 https://twitter.com/ErickRScott/status/871586233599893505 and it seems that I'm likely not the only one blocked: https://twitter.com/jtleek/status/871693250947624961 https://twitter.com/jtleek/status/871693250947624961 What's most concerning about @simplystats blocking activity is the chilling effect it has on discourse between differing perspectives. I've tried to come up with a rationale for why highlighting the most recent evidence in reply to someone who sympathized with Leek's original post (btw, @thomasp85 liked the tweet) is grounds for blocking , but I can't come up with a reasonable idea. Further aside, is irq11 Rafael Irizarry? Update: after emailing the members of @simplystats they have removed the block on my account and offered a reasonable explanation. SimplyStats is a force for good in the world (https://simplystatistics.org/courses/ https://simplystatistics.org/courses/) and I look forward to their future contributions.

417 points by deepnotderp 9y ago

...but why would you? The fact that there are people "getting their jimmies up" on questions of training massively paramterized statistical models on tiny amounts of data should tell you exactly where we are on the deep-learning hype cycle. For a while there, SVMs were the thing, but now the True Faithful have moved on to neural networks. The argument this writer is making is essentially: "yes, there are lots of free parameters to train, and that means that using it with small data is a bad idea in general, but neural networks have overfitting tools now and they're flexible so you should use them with small data anyway." This is literally the story told by the bulleted points. Neural networks are a tool. Don't use the tool if it isn't appropriate to your work. Maybe you can find a way to hammer a nail with a blowtorch, but it's still a bad idea.

RandomInteger4 9y ago
It might be that people want to learn to use deep learning, but they find the NIST AND other sets boring and want to learn on a problem they find interesting, but still produce something that seems to work. Plus, learning to learn how to learn on less can only help the field of learning. That's the goal of one shot learning right?
- irq11 9y ago
  Obviously, you do whatever you like when you're playing around, but when you find yourself in that situation (i.e. "I want to learn tool X, but all of my problems are inappropriate for X"), it's an indication that you're misusing/misunderstanding X. Again, with the forced metaphors: if I buy a new welder, I'm probably suddenly very keen on welding things. That doesn't mean I'm learning how to weld. A big (the biggest?) part of learning a tool is learning when to use it.
  - jfoutz 9y ago
    In my very limited experience, i hear about cool technology X, and have crazy fantasies about how it actually works. Picking up X and applying it to a problem, no matter how poorly informed, always taught me more about X than anything else. When you get your welder, you kinda have to weld a lot of things to see when it's effective and when it's not. Everybody gets a free pass with the first few months with a new toy. The only way to learn when it's appropriate is to screw up a few times.
    - irq11 9y ago
      Yes, sure. Playing with something is a good way to learn that thing. But if you find yourself saying "I really like using this backhoe, but I'm finding that most of my hole-digging problems are too small for a backhoe. Is there a blog post on using backhoes to make sandcastles?", you have perhaps wandered off the path to enlightenment. I'm really enjoying these metaphors though.
      - candiodari 9y ago
        https://www.youtube.com/watch?v=nltqM_Q-cwg https://www.youtube.com/watch?v=nltqM_Q-cwg
    - diegoperini 9y ago
      Here is an approach that worked for us well in our startup, that may help other teams who read this thread. We learned it through making a few mistakes in our decision making process. Whenever we have a new feature which cannot be implemented using existing frameworks, tools, in house technology or existing expertise, we inform the managers to add an extra 2 week to our schedule to evaluate as many options as we can. It is really hard to fit a lot of tools in that time so all the team picks up the work, even the ones that has no past experience or theoretical knowledge on the topic. It actually helps to have those ones in the research group. They are often the ones to be able to tell "since i have no idea in the expertise, i instead searched for this company who apparently ditched this tool because they suffered from this and that". Others who try to acquire the theory on the other hand is able to argue like "X seems to be better than Y". Once we have enough Xs, we already have use cases of X that is proven to be useless. In that attempt, mostly there remain only one X or even none. We either pick up that remaining X or an X that is less scary but a little more boring. Boring is good because a team can argue on a boring thing more easily. Those arguments produce quality code that remains to be used more than a year. A year or two is enough to allocate more research time for the topic, which eventually helps to find or implement a tool-set that can live much longer. Here are some examples we used more than a year and ditched or soon will ditch: Stock Tesseract Server Side OCR -> Properly Trained Server Side Tesseract + Image preprocessing on the mobile device Rethinkdb change feeds -> Postgres LISTEN Bluebird.js -> ReactiveX Ubuntu -> CentOS Forever -> Docker Edit: The reason why I am posting under your comment is in some cases, mistakes can hurt a company in a way that is unrecoverable. My advice my not apply to pet projects.
simcop2387 9y ago
Small question here, what are SVNs in this context?
- irq11 9y ago
  support vector machines. a non-linear classifier that was the hot thing in the AI world before neural networks became practical.
  - candiodari 9y ago
    They're great because they immediately go to the best solution. One epoch is all you need, no matter the size of the SVM. They suck because they're limited to one layer. But there's a good case to be made that machine learning introductions should always be done like this : linear classifier (when that works) -> SVM (when that works) -> NN -> Deep learning.
    - irq11 9y ago
      I contend that all real work should be done this way. And don't forget things like bayes' nets and random forests, which get no love. in other words, KISS.
      - snovv_crash 9y ago
        To go even simpler, KNN also works really well if you can get a good weighting for your inputs and measurement noise is low. And KNN works with online/soft-RT datasets as well, where constant learning is required.
        CuriouslyC 9y ago
        I have done a lot of work with classification algorithms, and KNN doesn't get nearly the love it deserves. It rarely turns in exceptional performance, but when used with mahalinobis distance it is extremely robust.
      - 9y ago
rahimnathwani 9y ago
s/SVN/SVM
- irq11 9y ago
  oops, typo.
tsiki 9y ago
I think you're missing the point. The jimmies are getting rustled up because someone provides false information about the performance to make his own argument seem better. This is something anyone should be against.
- irq11 9y ago
  but they didn't. the writer makes an unconvincing claim that the original post was wrong. the data presented shows only that if you try really hard and get lucky enough, you can probably do as well as a simple regression in this case. the author himself admits that deep learning is probably misapplied here, and that training with such small data is difficult, at best. which again brings us back to the important question (i.e. the point being made by the original post): why would you ever do this?
  - dbecker 9y ago
    if you try really hard and get lucky enough, you can probably do as well as a simple regression in this case. Maybe you aren't familiar with deep learning: but this isn't "trying really hard." This is doing basic stuff that anyone using deep learning probably knows. And deep learning doesn't just "do as well" as the simpler model. It does meaningfully better at all sample sizes.
    - blueblob 9y ago
      Define "meaningfully better." Perhaps you mean statistically significantly better? It may have better accuracy, but it has significantly less interpretability. What does it capture that regression couldn't capture? At least with regression you can interpret the relationship between all of the variables and their relative importance by looking at the coefficients of the regression. With deep learning, the best approaches for explanation are to train another model at the same time that you use for explanation. Additionally, it was proven that a perceptron can learn any function, so in some senses the "deep" part of deep learning is because people are being lazy because at least you could get a better interpretation of the perceptron. I don't mean to imply that there's not a place for deep learning, but I think this isn't a great refutation of the argument that fitting a deep model is somewhat inappropriate for a small dataset.

rkaplan 9y ago

This post doesn't even mention the easiest way to use deep learning without a lot of data: download a pretrained model and fine-tune the last few layers on your small dataset. In many domains (like image classification, the task in this blog post) fine-tuning works extremely well, because the pretrained model has learned generic features in the early layers that are useful for many datasets, not just the one trained on. Even the best skin cancer classifier [1] was pretrained on ImageNet. [1]: http://www.nature.com/articles/nature21056 http://www.nature.com/articles/nature21056

option_greek 9y ago
Does transfer learning apply to seq2seq as well ?
a_bonobo 9y ago
This is how the great fast.ai course begins - download VGG16, finetune the top layer with a single dense layer, get amazing results. The second or third class shows how to make the top layers a bit more complex to get even better accuracy.
- alexcnwy 9y ago
  Can't recommend the course highly enough!
  - manibatra 9y ago
    Same!! On lesson 2 and already feel like I know so much! The top down approach to learning is great! Will actually read the book "Making Learning Whole" that inspired them to follow this approach.
  - Steeeve 9y ago
    The 30 minute overview has me excited about going through it. I really like how they have structured it. Thanks to all for recommending it!
  - yamaneko 9y ago

deepnotderp 9y ago

Also, not to mention transfer learning is a big one. Case in point, the silicon valley "not hotdog" classifier which they stopped at hotdog or not due to lack of training data when in reality they could've just used a pre trained net on imagenet. Lol, I was literally cringing through that episode so hard xD

minimaxir 9y ago
The developer behind the real-world app mentioned why they did not use pretraining, and why you can't always use pretraining: https://news.ycombinator.com/item?id=14347513 https://news.ycombinator.com/item?id=14347513 > We ended up with a custom architecture trained from scratch due to runtime constraints more so than accuracy reasons (the inference runs on phones, so we have to be efficient with CPU + memory), but that model also ended up being the most accurate model we could build in the time we had. (With more time/resources I have no doubt I could have achieved better accuracy with a heavier model!)
- deleted 9y ago
  [deleted]
- deepnotderp 9y ago
  You could still have pre trained the custom architecture.

shadowmint 9y ago

You probably can... but is that really the issue? I think the problem isnt that you cant solve problems with small amounts of data; its that you can't solve 'the problem' at a small scale and then just apply that solution at large scale... and that's not what people want or expect. People expect that if you have an industrial welder than can assemble areoplanes (apparently), then you should easily be able to check it out by welding a few sheets of metal together, and if it welds well on a small scale, it should be representative of how well it welds entire vehicles. ...but thats not how DNN models work. Each solution is a specific selection of hyperparameters for the specific data and specific shape of that data. As we see here, specific even to the volume of data available. It doesnt scale up and it doesn't scale down. To solve a problem you just have to sort of.... just mess around with different solutions until you get a good one. ...and even then, you've got no really strong proof your solution is good; just that its better than the other solutions you've tried. Thats the problem; its really hard to know when DNN are the wrong choice, vs. you're just 'doing it wrong'

zensavona 9y ago

Maybe what you should do is deep learn some data and then do some deep learning with your deep learned [deep] data. Deep.

aub3bhat 9y ago

This debate is meaningless for several reasons: 1. The original argument is a strawman. What do they mean by "data"? Is it survey results, microarrays, "Natural" images, "Natural" language text or readings from an audio sensor? No ML researcher would argue that applying complex models such as CNNs is useful for say survey data. But if the data is domain specific, such as Natural Language text, images taken in particular context, etc. using a model and parameters that exhibit good performance is a good starting point. 2. Unlike how statisticians view data (as say a matrix of measurements or "Data Frame"), machine learning researchers view data at a higher level of representation. E.g. An image is not merely a matrix but rather an object that can be augmented by horizontally flipping, changing contrast etc. In case of text you can render characters using different fonts, colors etc. 3. Finally the example used in the initial blog post, of predicting 1 vs 0 from images is itself incorrect. Sure a statistician would "train" a linear model to predict 1 vs 0, however I as an ML researcher would NOT train any model at all and would just use [1] which has state of the art performance in character recognition in widely varying conditions. When you have only 80 images, why risk assuming that they are sampled in an IID manner from population, instead why not simply use a model thats trained on far larger population. Now the final argument might look suspicious but its crucial in understanding the difference between AI/ML/CV vs Statistics. In AI/ML/CV the assumption is that there are higher level problems (Character recognition, Object recognition, Scene understanding, Audio recognition) which when solved enable us to apply them in wide variety of situations where they appear. Thus when you encounter a problem like digit recognition the answer an ML researcher would give is to use a state of the art model. [1] https://github.com/bgshih/crnn https://github.com/bgshih/crnn

Sean1708 9y ago
Are you using some accepted definition of Statistics that I'm unaware of? Because I always thought Machine Learning was a branch of Statistics.
- aub3bhat 9y ago
  >> Machine Learning was a branch of Statistics Thats incorrect. """ Machine learning is the subfield of computer science that, according to Arthur Samuel in 1959, gives "computers the ability to learn without being explicitly programmed." """ -- Wikipedia

j7ake 9y ago

Of course you can use it but does it perform better than "shallow" methods such as Gaussian processes, SVMs, and multivariate linear regression ? either through theoretical or empirical evidence ?

minimaxir 9y ago
The original post used a linear regression (and apparently misimplemented an intended logistic regression); this post sees better results at all sample sizes with a proper deep learning approach.

known 9y ago

aka Wisdom of Crowds

ska 9y ago

It's an interesting conversation but really weakened by failing to take on the generalization problem head on. This is something I see in a lot of discussions about deep nets on smaller data sets, whether transfer or not. The answer "it's built in" is particularly unsatisfying. The plots shown certainly should raise the spectre of overtraining - and rather than handwaving about techniques to avoid it, it would be great to see a detailed discussion of how you convince yourself (i.e. with additional data) that you are reasonably generalizable. Deep learning techniques are no panacea here.

m3kw9 9y ago

Ppl keep saying a lot without even thinking is a relative term. For images a lot means enough to get to x percentage accuracy, for OCR for a single font, a lot means 26 letters + special chars and numbers. Stop saying a lot blindly like every one underatands.

erickscott 9y ago

Andrew Beam's post offered very persuasive evidence that Jeff Leek's intuition (deep learning yields poor performance with small sample sizes) is incorrect. The error bars and the consistent trend of higher accuracy with a properly implemented deep learning model, particularly with smaller sample sizes, is devastating to Leek's original post. I think this is a fantastic example of the speed and self-correcting nature of science in the internet-age. As an aside, @simplystats blocked me on Twitter, which I assume is in response to this tweet: https://twitter.com/ErickRScott/status/871586233599893505 https://twitter.com/ErickRScott/status/871586233599893505 and it seems that I'm likely not the only one blocked: https://twitter.com/jtleek/status/871693250947624961 https://twitter.com/jtleek/status/871693250947624961 What's most concerning about @simplystats blocking activity is the chilling effect it has on discourse between differing perspectives. I've tried to come up with a rationale for why highlighting the most recent evidence in reply to someone who sympathized with Leek's original post (btw, @thomasp85 liked the tweet) is grounds for blocking , but I can't come up with a reasonable idea. Further aside, is irq11 Rafael Irizarry? Update: after emailing the members of @simplystats they have removed the block on my account and offered a reasonable explanation. SimplyStats is a force for good in the world (https://simplystatistics.org/courses/ https://simplystatistics.org/courses/) and I look forward to their future contributions.