5 ms·

Your AI skills are worth less than you think

28 points by secabeen 8y ago

pizza 8y ago
In a gold rush, sell pickaxes. bitcoin boom => GPUs AI boom => TPUs (and GPUs, too), cloud processing platforms, and, generally, AI specific hardware platforms
- partingshots 8y ago
  What are your opinions on the TPUs Google uses versus more traditional GPUs from say Nvidia for machine learning?
  - pizza 8y ago
    Honestly, I’ve never used them before. Cloud notebooks running on virtual GPU servers seems awesome, in theory, to me, though.
m0zg 8y ago
Stopped reading at "manual tuning is going the way of dodo, and good riddance". The guy clearly has no idea whatsoever what he's talking about (figures: he was an SRE at Google, nowhere close to any ML/DL efforts). Hyperparameter tuning is an area of active research and a dark art of sorts, and it will remain that for the foreseeable future for one simple reason: hyperparameters are interdependent and _data dependent_ as well, and nobody, not even Google with its TPUs has the compute to tune them "automatically" to any meaningful extent every time. What you see with the various "automl" efforts is merely transfer learning, refining of existing models on customer's data. You can learn how to do that in one evening, with no prior DL experience if you're a good coder, and in 2-3 evenings if you aren't a good coder. Point is, it's not a difficult problem, just follow a tutorial. That's great, that's how smart researchers build practical models, but that's not really "auto-ml", in the sense that there's no hyperparameter tuning going on (as far as I know) and no new problems are being solved. It's just a classifier: anybody can do that with common off the shelf tools and pre-trained checkpoints. Now achieving state of the art results, solving novel tasks for which there aren't any ready-made solutions, doing DL efficiently, figuring out creative ways to get labeled data, semi-supervised or few-shot methods, etc, that's where it's at right now. And the comp has never been better if you know what you're doing.
- bambax 8y ago
  > figures: he was an SRE at Google, nowhere close to any ML/DL efforts > Ric Szopa is the CTO in Residence at Inovo.vc. Prior to Inovo, Ric was the CTO at MicroscopeIT, a software house specializing in computer vision, robotics, and microscope image processing. Before that he worked on YouTube’s database infrastructure at Google, in Mountain View, California and Zurich, Switzerland. Ric studied Philosophy at the University of Warsaw and Artificial Intelligence at the Katholieke Universiteit Leuven.
- Zephyr314 8y ago
  To be fair, the hyperparameter tuning behind these AutoML systems are getting fairly robust. Google bases theirs on Vizier [0]. The Amazon Sagemaker group has people from the gpyopt project [1]. There is also tons of open source projects out there to help for non-enterprise projects [2] [3]. There are also stand-alone companies that help with this explicitly for enterprises [4] (Caveat, I am a founder). Increasingly I think more time will be spent on the creative/bespoke aspects you mention later in your post, like making sure that you are building a system that actually achieves some business value (vs just getting a better academic-oriented metric result). Hyperparameter tuning is basically trying to do high-dimensional, non-convex optimization on time consuming and expensive to sample functions. Hand tuning is a terrible way to approach this, and is different for each problem as you point out. Experts can leverage their domain expertise and the unique aspects of their data, models, and applications in much better ways. [0]: https://www.kdd.org/kdd2017/papers/view/google-vizier-a-service-for-black-box-optimization https://www.kdd.org/kdd2017/papers/view/google-vizier-a-serv... [1]: https://github.com/SheffieldML/GPyOpt https://github.com/SheffieldML/GPyOpt [2]: https://github.com/Yelp/MOE https://github.com/Yelp/MOE [3]: https://github.com/hyperopt/hyperopt https://github.com/hyperopt/hyperopt [4]: https://sigopt.com https://sigopt.com
  - m0zg 8y ago
    Are you sure this is actually being used for "AutoML" type services? All of the mentioned methods require a parameter search, which is computationally infeasible in a "quick" AutoML use case, and expensive in case you actually need it. That is, you more or less run several training sessions in parallel, and learn from which performs the best in choosing the next parameters. You don't do a full grid search (that's completely unfeasible most of the time), but you at best tweak only a few parameters, and you don't do it every time you train. Hyperparameters aren't just the learning rate and weight decay, it's also the size and number of layers, where and when to quantize and by how much, structure of the network, parameters of pooling, etc etc. I'd say we're still pretty early in the game with all of that, especially when it comes to efficient architectures that demonstrate high accuracy.
justin_evans 8y ago
The term “Artificial Intelligence” is widespread these days. Once, It was considered fictional that robots will act like a human and eventually take human place. But, now we can see that technology is transforming this fiction into reality. I read a post about how AI influencing mobile technology: https://mobinspire.com/blogs/how-ai-is-influencing-mobile-technology/ https://mobinspire.com/blogs/how-ai-is-influencing-mobile-te...

28 points by secabeen 8y ago

In a gold rush, sell pickaxes. bitcoin boom => GPUs AI boom => TPUs (and GPUs, too), cloud processing platforms, and, generally, AI specific hardware platforms

partingshots 8y ago
What are your opinions on the TPUs Google uses versus more traditional GPUs from say Nvidia for machine learning?
- pizza 8y ago
  Honestly, I’ve never used them before. Cloud notebooks running on virtual GPU servers seems awesome, in theory, to me, though.

m0zg 8y ago

Stopped reading at "manual tuning is going the way of dodo, and good riddance". The guy clearly has no idea whatsoever what he's talking about (figures: he was an SRE at Google, nowhere close to any ML/DL efforts). Hyperparameter tuning is an area of active research and a dark art of sorts, and it will remain that for the foreseeable future for one simple reason: hyperparameters are interdependent and _data dependent_ as well, and nobody, not even Google with its TPUs has the compute to tune them "automatically" to any meaningful extent every time. What you see with the various "automl" efforts is merely transfer learning, refining of existing models on customer's data. You can learn how to do that in one evening, with no prior DL experience if you're a good coder, and in 2-3 evenings if you aren't a good coder. Point is, it's not a difficult problem, just follow a tutorial. That's great, that's how smart researchers build practical models, but that's not really "auto-ml", in the sense that there's no hyperparameter tuning going on (as far as I know) and no new problems are being solved. It's just a classifier: anybody can do that with common off the shelf tools and pre-trained checkpoints. Now achieving state of the art results, solving novel tasks for which there aren't any ready-made solutions, doing DL efficiently, figuring out creative ways to get labeled data, semi-supervised or few-shot methods, etc, that's where it's at right now. And the comp has never been better if you know what you're doing.

bambax 8y ago
> figures: he was an SRE at Google, nowhere close to any ML/DL efforts > Ric Szopa is the CTO in Residence at Inovo.vc. Prior to Inovo, Ric was the CTO at MicroscopeIT, a software house specializing in computer vision, robotics, and microscope image processing. Before that he worked on YouTube’s database infrastructure at Google, in Mountain View, California and Zurich, Switzerland. Ric studied Philosophy at the University of Warsaw and Artificial Intelligence at the Katholieke Universiteit Leuven.
Zephyr314 8y ago
To be fair, the hyperparameter tuning behind these AutoML systems are getting fairly robust. Google bases theirs on Vizier [0]. The Amazon Sagemaker group has people from the gpyopt project [1]. There is also tons of open source projects out there to help for non-enterprise projects [2] [3]. There are also stand-alone companies that help with this explicitly for enterprises [4] (Caveat, I am a founder). Increasingly I think more time will be spent on the creative/bespoke aspects you mention later in your post, like making sure that you are building a system that actually achieves some business value (vs just getting a better academic-oriented metric result). Hyperparameter tuning is basically trying to do high-dimensional, non-convex optimization on time consuming and expensive to sample functions. Hand tuning is a terrible way to approach this, and is different for each problem as you point out. Experts can leverage their domain expertise and the unique aspects of their data, models, and applications in much better ways. [0]: https://www.kdd.org/kdd2017/papers/view/google-vizier-a-service-for-black-box-optimization https://www.kdd.org/kdd2017/papers/view/google-vizier-a-serv... [1]: https://github.com/SheffieldML/GPyOpt https://github.com/SheffieldML/GPyOpt [2]: https://github.com/Yelp/MOE https://github.com/Yelp/MOE [3]: https://github.com/hyperopt/hyperopt https://github.com/hyperopt/hyperopt [4]: https://sigopt.com https://sigopt.com
- m0zg 8y ago
  Are you sure this is actually being used for "AutoML" type services? All of the mentioned methods require a parameter search, which is computationally infeasible in a "quick" AutoML use case, and expensive in case you actually need it. That is, you more or less run several training sessions in parallel, and learn from which performs the best in choosing the next parameters. You don't do a full grid search (that's completely unfeasible most of the time), but you at best tweak only a few parameters, and you don't do it every time you train. Hyperparameters aren't just the learning rate and weight decay, it's also the size and number of layers, where and when to quantize and by how much, structure of the network, parameters of pooling, etc etc. I'd say we're still pretty early in the game with all of that, especially when it comes to efficient architectures that demonstrate high accuracy.

justin_evans 8y ago

The term “Artificial Intelligence” is widespread these days. Once, It was considered fictional that robots will act like a human and eventually take human place. But, now we can see that technology is transforming this fiction into reality. I read a post about how AI influencing mobile technology: https://mobinspire.com/blogs/how-ai-is-influencing-mobile-technology/ https://mobinspire.com/blogs/how-ai-is-influencing-mobile-te...