5 ms·

...but why would you? The fact that there are people "getting their jimmies up" on questions of training massively paramterized statistical models on tiny amou

by irq11 9y ago

...but why would you? The fact that there are people "getting their jimmies up" on questions of training massively paramterized statistical models on tiny amounts of data should tell you exactly where we are on the deep-learning hype cycle. For a while there, SVMs were the thing, but now the True Faithful have moved on to neural networks. The argument this writer is making is essentially: "yes, there are lots of free parameters to train, and that means that using it with small data is a bad idea in general, but neural networks have overfitting tools now and they're flexible so you should use them with small data anyway." This is literally the story told by the bulleted points. Neural networks are a tool. Don't use the tool if it isn't appropriate to your work. Maybe you can find a way to hammer a nail with a blowtorch, but it's still a bad idea.

RandomInteger4 9y ago
It might be that people want to learn to use deep learning, but they find the NIST AND other sets boring and want to learn on a problem they find interesting, but still produce something that seems to work. Plus, learning to learn how to learn on less can only help the field of learning. That's the goal of one shot learning right?
- irq11 9y ago
  Obviously, you do whatever you like when you're playing around, but when you find yourself in that situation (i.e. "I want to learn tool X, but all of my problems are inappropriate for X"), it's an indication that you're misusing/misunderstanding X. Again, with the forced metaphors: if I buy a new welder, I'm probably suddenly very keen on welding things. That doesn't mean I'm learning how to weld. A big (the biggest?) part of learning a tool is learning when to use it.
  - jfoutz 9y ago
    In my very limited experience, i hear about cool technology X, and have crazy fantasies about how it actually works. Picking up X and applying it to a problem, no matter how poorly informed, always taught me more about X than anything else. When you get your welder, you kinda have to weld a lot of things to see when it's effective and when it's not. Everybody gets a free pass with the first few months with a new toy. The only way to learn when it's appropriate is to screw up a few times.
    - irq11 9y ago
      Yes, sure. Playing with something is a good way to learn that thing. But if you find yourself saying "I really like using this backhoe, but I'm finding that most of my hole-digging problems are too small for a backhoe. Is there a blog post on using backhoes to make sandcastles?", you have perhaps wandered off the path to enlightenment. I'm really enjoying these metaphors though.
      - candiodari 9y ago
        https://www.youtube.com/watch?v=nltqM_Q-cwg https://www.youtube.com/watch?v=nltqM_Q-cwg
    - diegoperini 9y ago
      Here is an approach that worked for us well in our startup, that may help other teams who read this thread. We learned it through making a few mistakes in our decision making process. Whenever we have a new feature which cannot be implemented using existing frameworks, tools, in house technology or existing expertise, we inform the managers to add an extra 2 week to our schedule to evaluate as many options as we can. It is really hard to fit a lot of tools in that time so all the team picks up the work, even the ones that has no past experience or theoretical knowledge on the topic. It actually helps to have those ones in the research group. They are often the ones to be able to tell "since i have no idea in the expertise, i instead searched for this company who apparently ditched this tool because they suffered from this and that". Others who try to acquire the theory on the other hand is able to argue like "X seems to be better than Y". Once we have enough Xs, we already have use cases of X that is proven to be useless. In that attempt, mostly there remain only one X or even none. We either pick up that remaining X or an X that is less scary but a little more boring. Boring is good because a team can argue on a boring thing more easily. Those arguments produce quality code that remains to be used more than a year. A year or two is enough to allocate more research time for the topic, which eventually helps to find or implement a tool-set that can live much longer. Here are some examples we used more than a year and ditched or soon will ditch: Stock Tesseract Server Side OCR -> Properly Trained Server Side Tesseract + Image preprocessing on the mobile device Rethinkdb change feeds -> Postgres LISTEN Bluebird.js -> ReactiveX Ubuntu -> CentOS Forever -> Docker Edit: The reason why I am posting under your comment is in some cases, mistakes can hurt a company in a way that is unrecoverable. My advice my not apply to pet projects.
simcop2387 9y ago
Small question here, what are SVNs in this context?
- irq11 9y ago
  support vector machines. a non-linear classifier that was the hot thing in the AI world before neural networks became practical.
  - candiodari 9y ago
    They're great because they immediately go to the best solution. One epoch is all you need, no matter the size of the SVM. They suck because they're limited to one layer. But there's a good case to be made that machine learning introductions should always be done like this : linear classifier (when that works) -> SVM (when that works) -> NN -> Deep learning.
    - irq11 9y ago
      I contend that all real work should be done this way. And don't forget things like bayes' nets and random forests, which get no love. in other words, KISS.
      - snovv_crash 9y ago
        To go even simpler, KNN also works really well if you can get a good weighting for your inputs and measurement noise is low. And KNN works with online/soft-RT datasets as well, where constant learning is required.
        CuriouslyC 9y ago
        I have done a lot of work with classification algorithms, and KNN doesn't get nearly the love it deserves. It rarely turns in exceptional performance, but when used with mahalinobis distance it is extremely robust.
      - 9y ago
rahimnathwani 9y ago
s/SVN/SVM
- irq11 9y ago
  oops, typo.
tsiki 9y ago
I think you're missing the point. The jimmies are getting rustled up because someone provides false information about the performance to make his own argument seem better. This is something anyone should be against.
- irq11 9y ago
  but they didn't. the writer makes an unconvincing claim that the original post was wrong. the data presented shows only that if you try really hard and get lucky enough, you can probably do as well as a simple regression in this case. the author himself admits that deep learning is probably misapplied here, and that training with such small data is difficult, at best. which again brings us back to the important question (i.e. the point being made by the original post): why would you ever do this?
  - dbecker 9y ago
    if you try really hard and get lucky enough, you can probably do as well as a simple regression in this case. Maybe you aren't familiar with deep learning: but this isn't "trying really hard." This is doing basic stuff that anyone using deep learning probably knows. And deep learning doesn't just "do as well" as the simpler model. It does meaningfully better at all sample sizes.
    - blueblob 9y ago
      Define "meaningfully better." Perhaps you mean statistically significantly better? It may have better accuracy, but it has significantly less interpretability. What does it capture that regression couldn't capture? At least with regression you can interpret the relationship between all of the variables and their relative importance by looking at the coefficients of the regression. With deep learning, the best approaches for explanation are to train another model at the same time that you use for explanation. Additionally, it was proven that a perceptron can learn any function, so in some senses the "deep" part of deep learning is because people are being lazy because at least you could get a better interpretation of the perceptron. I don't mean to imply that there's not a place for deep learning, but I think this isn't a great refutation of the argument that fitting a deep model is somewhat inappropriate for a small dataset.

by irq11 9y ago

RandomInteger4 9y ago

It might be that people want to learn to use deep learning, but they find the NIST AND other sets boring and want to learn on a problem they find interesting, but still produce something that seems to work. Plus, learning to learn how to learn on less can only help the field of learning. That's the goal of one shot learning right?

irq11 9y ago
Obviously, you do whatever you like when you're playing around, but when you find yourself in that situation (i.e. "I want to learn tool X, but all of my problems are inappropriate for X"), it's an indication that you're misusing/misunderstanding X. Again, with the forced metaphors: if I buy a new welder, I'm probably suddenly very keen on welding things. That doesn't mean I'm learning how to weld. A big (the biggest?) part of learning a tool is learning when to use it.
- jfoutz 9y ago
  In my very limited experience, i hear about cool technology X, and have crazy fantasies about how it actually works. Picking up X and applying it to a problem, no matter how poorly informed, always taught me more about X than anything else. When you get your welder, you kinda have to weld a lot of things to see when it's effective and when it's not. Everybody gets a free pass with the first few months with a new toy. The only way to learn when it's appropriate is to screw up a few times.
  - irq11 9y ago
    Yes, sure. Playing with something is a good way to learn that thing. But if you find yourself saying "I really like using this backhoe, but I'm finding that most of my hole-digging problems are too small for a backhoe. Is there a blog post on using backhoes to make sandcastles?", you have perhaps wandered off the path to enlightenment. I'm really enjoying these metaphors though.
    - candiodari 9y ago
      https://www.youtube.com/watch?v=nltqM_Q-cwg https://www.youtube.com/watch?v=nltqM_Q-cwg
  - diegoperini 9y ago
    Here is an approach that worked for us well in our startup, that may help other teams who read this thread. We learned it through making a few mistakes in our decision making process. Whenever we have a new feature which cannot be implemented using existing frameworks, tools, in house technology or existing expertise, we inform the managers to add an extra 2 week to our schedule to evaluate as many options as we can. It is really hard to fit a lot of tools in that time so all the team picks up the work, even the ones that has no past experience or theoretical knowledge on the topic. It actually helps to have those ones in the research group. They are often the ones to be able to tell "since i have no idea in the expertise, i instead searched for this company who apparently ditched this tool because they suffered from this and that". Others who try to acquire the theory on the other hand is able to argue like "X seems to be better than Y". Once we have enough Xs, we already have use cases of X that is proven to be useless. In that attempt, mostly there remain only one X or even none. We either pick up that remaining X or an X that is less scary but a little more boring. Boring is good because a team can argue on a boring thing more easily. Those arguments produce quality code that remains to be used more than a year. A year or two is enough to allocate more research time for the topic, which eventually helps to find or implement a tool-set that can live much longer. Here are some examples we used more than a year and ditched or soon will ditch: Stock Tesseract Server Side OCR -> Properly Trained Server Side Tesseract + Image preprocessing on the mobile device Rethinkdb change feeds -> Postgres LISTEN Bluebird.js -> ReactiveX Ubuntu -> CentOS Forever -> Docker Edit: The reason why I am posting under your comment is in some cases, mistakes can hurt a company in a way that is unrecoverable. My advice my not apply to pet projects.

simcop2387 9y ago

Small question here, what are SVNs in this context?

irq11 9y ago
support vector machines. a non-linear classifier that was the hot thing in the AI world before neural networks became practical.
- candiodari 9y ago
  They're great because they immediately go to the best solution. One epoch is all you need, no matter the size of the SVM. They suck because they're limited to one layer. But there's a good case to be made that machine learning introductions should always be done like this : linear classifier (when that works) -> SVM (when that works) -> NN -> Deep learning.
  - irq11 9y ago
    I contend that all real work should be done this way. And don't forget things like bayes' nets and random forests, which get no love. in other words, KISS.
    - snovv_crash 9y ago
      To go even simpler, KNN also works really well if you can get a good weighting for your inputs and measurement noise is low. And KNN works with online/soft-RT datasets as well, where constant learning is required.
      - CuriouslyC 9y ago
        I have done a lot of work with classification algorithms, and KNN doesn't get nearly the love it deserves. It rarely turns in exceptional performance, but when used with mahalinobis distance it is extremely robust.
    - 9y ago

rahimnathwani 9y ago

s/SVN/SVM

irq11 9y ago
oops, typo.

tsiki 9y ago

I think you're missing the point. The jimmies are getting rustled up because someone provides false information about the performance to make his own argument seem better. This is something anyone should be against.

irq11 9y ago
but they didn't. the writer makes an unconvincing claim that the original post was wrong. the data presented shows only that if you try really hard and get lucky enough, you can probably do as well as a simple regression in this case. the author himself admits that deep learning is probably misapplied here, and that training with such small data is difficult, at best. which again brings us back to the important question (i.e. the point being made by the original post): why would you ever do this?
- dbecker 9y ago
  if you try really hard and get lucky enough, you can probably do as well as a simple regression in this case. Maybe you aren't familiar with deep learning: but this isn't "trying really hard." This is doing basic stuff that anyone using deep learning probably knows. And deep learning doesn't just "do as well" as the simpler model. It does meaningfully better at all sample sizes.
  - blueblob 9y ago
    Define "meaningfully better." Perhaps you mean statistically significantly better? It may have better accuracy, but it has significantly less interpretability. What does it capture that regression couldn't capture? At least with regression you can interpret the relationship between all of the variables and their relative importance by looking at the coefficients of the regression. With deep learning, the best approaches for explanation are to train another model at the same time that you use for explanation. Additionally, it was proven that a perceptron can learn any function, so in some senses the "deep" part of deep learning is because people are being lazy because at least you could get a better interpretation of the perceptron. I don't mean to imply that there's not a place for deep learning, but I think this isn't a great refutation of the argument that fitting a deep model is somewhat inappropriate for a small dataset.