3 ms·

This is totally counter to my experience. NNs are a real pain involving architectural searches, hyperparameter searches, etc. They take forever to train. Someti

by Gimpei 5y ago

This is totally counter to my experience. NNs are a real pain involving architectural searches, hyperparameter searches, etc. They take forever to train. Sometimes it's even difficult to get them to converge. The easy magical things are GBTs. If you have a classification problem that isn't super non-linear, try a GBT first. There's hardly any tweaking and the training is fast.

sillysaurusx 5y ago
I'm surprised our experiences differ so greatly. It's true that you have to be particularly determined to get results, but it's not true that they take forever to train. At the height of my productivity, I was aiming for at least three completed experiments per day, and always made sure to have at least one going while I was sleeping. It's a bit like saying it takes forever to grow tomato plants. Well, yes. Yes it does. Welcome to gardening. ML is problem-space gardening. Yep, it's hyperparameter searches. Yep, it's architecture searches. No, it doesn't take nearly as long as you're implying, because you can quickly narrow the problem space in log(N) steps. (It would be foolish to do a hyperparam search in linear increments.) Resources have also never been more plentiful. TRC literally gives you 100 TPUs when you sign up. https://jaxtputest.vercel.app/ https://jaxtputest.vercel.app/ That means you can (and I did) run 100 training sessions simultaneously. https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf
- lumost 5y ago
  Tbh it sounds like you got lucky on your first outing with ML and your first NN model worked well enough to deliver a success. Congratulations on achieving this rare milestone. While I’ve seen many data scientists fall into the trap of spending 1 week per iteration even when it’s clear the change produces negligible result. If you stick with the field long enough you’re likely to hit a dog of a problem that should have worked, but ultimately did not. The areas where ML is advancing fastest also suffer from elusive solutions. Language models and image recognition are only recently tractable problems - the folks trying to advance these fields do not experience magic NN improvements.
  - sillysaurusx 5y ago
    Oof. I completely agree that one week per iteration isn’t good. Thanks for pointing this out — I need to write up something about how to be effective. 1 week is fine for a production training run. But that’s after you’ve locked in your hyperparams on small prototypes. Small prototypes should take no more than two days to give results. If your prototype absolutely must be a large model, you should have many (perhaps even a dozen) attempts training simultaneously. It’s hard to imagine someone spending an entire week waiting to see whether their latest tweak worked, then tweaking again and waiting another week… I had models like that, but they were exploratory side projects, not my primary research focus. It’s stuff you try for fun, not for work. That way a week is “whatever” if it doesn’t work out. But when you’re hunting for a result, no way. Prototype results are measured in “N completed runs per day”. Which means you should have multiple runs going at all times. It gets tricky to manage, but… I don’t know. Either you’re right and I got incredibly lucky, or determined programmers can do the same thing I did. I’m not that smart.
    - godelski 5y ago
      > one week per iteration isn’t good. Try training Imagenet. For some reason this is called a "medium" sized data set. I do want to point something out though that's a big problem with a lot of your comments. We're seeing the results of what happens when you "just implement a DL pipeline." Lots of people are using these powerful tools without understanding. It's like handing a monkey a machine gun. We're seeing that models don't generalize like people think they do. We're seeing people use models pre-trained on ImageNet or CelebA and think that there is no bias in them. When people are saying you need to learn theory, this is really what they are concerned with.
- Straw 5y ago
  If you can train an NN in a day, then you can probably train SVD in minutes/hours- so why not start with it? Otherwise you might not even recognize the symptoms of bad hyperparameters fo r the NN resulting in poor performance.

by Gimpei 5y ago

sillysaurusx 5y ago

I'm surprised our experiences differ so greatly. It's true that you have to be particularly determined to get results, but it's not true that they take forever to train. At the height of my productivity, I was aiming for at least three completed experiments per day, and always made sure to have at least one going while I was sleeping. It's a bit like saying it takes forever to grow tomato plants. Well, yes. Yes it does. Welcome to gardening. ML is problem-space gardening. Yep, it's hyperparameter searches. Yep, it's architecture searches. No, it doesn't take nearly as long as you're implying, because you can quickly narrow the problem space in log(N) steps. (It would be foolish to do a hyperparam search in linear increments.) Resources have also never been more plentiful. TRC literally gives you 100 TPUs when you sign up. https://jaxtputest.vercel.app/ https://jaxtputest.vercel.app/ That means you can (and I did) run 100 training sessions simultaneously. https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf

lumost 5y ago
Tbh it sounds like you got lucky on your first outing with ML and your first NN model worked well enough to deliver a success. Congratulations on achieving this rare milestone. While I’ve seen many data scientists fall into the trap of spending 1 week per iteration even when it’s clear the change produces negligible result. If you stick with the field long enough you’re likely to hit a dog of a problem that should have worked, but ultimately did not. The areas where ML is advancing fastest also suffer from elusive solutions. Language models and image recognition are only recently tractable problems - the folks trying to advance these fields do not experience magic NN improvements.
- sillysaurusx 5y ago
  Oof. I completely agree that one week per iteration isn’t good. Thanks for pointing this out — I need to write up something about how to be effective. 1 week is fine for a production training run. But that’s after you’ve locked in your hyperparams on small prototypes. Small prototypes should take no more than two days to give results. If your prototype absolutely must be a large model, you should have many (perhaps even a dozen) attempts training simultaneously. It’s hard to imagine someone spending an entire week waiting to see whether their latest tweak worked, then tweaking again and waiting another week… I had models like that, but they were exploratory side projects, not my primary research focus. It’s stuff you try for fun, not for work. That way a week is “whatever” if it doesn’t work out. But when you’re hunting for a result, no way. Prototype results are measured in “N completed runs per day”. Which means you should have multiple runs going at all times. It gets tricky to manage, but… I don’t know. Either you’re right and I got incredibly lucky, or determined programmers can do the same thing I did. I’m not that smart.
  - godelski 5y ago
    > one week per iteration isn’t good. Try training Imagenet. For some reason this is called a "medium" sized data set. I do want to point something out though that's a big problem with a lot of your comments. We're seeing the results of what happens when you "just implement a DL pipeline." Lots of people are using these powerful tools without understanding. It's like handing a monkey a machine gun. We're seeing that models don't generalize like people think they do. We're seeing people use models pre-trained on ImageNet or CelebA and think that there is no bias in them. When people are saying you need to learn theory, this is really what they are concerned with.
Straw 5y ago
If you can train an NN in a day, then you can probably train SVD in minutes/hours- so why not start with it? Otherwise you might not even recognize the symptoms of bad hyperparameters fo r the NN resulting in poor performance.