3 ms·

Anything that's not MNIST. Even Fashion-MNIST benefits from hyperparam search. Distributed TF is the standard in Google, will be the standard amongst everybody

Anything that's not MNIST. Even Fashion-MNIST benefits from hyperparam search. Distributed TF is the standard in Google, will be the standard amongst everybody else within a couple of years. They actually just run Estimators, and distribution is somewhat transparent to them. When hyperparam search is this easy with PySpark, why wouldn't you do it? def model_fn(learning_rate, dropout): [TensorFlow Code here] args_dict = {'learning_rate': [0.001, 0.005, 0.01], 'dropout': [0.5, 0.6]} experiment.launch(spark, model_fn, args_dict) For distributed training, we favor Horovod. It requires minimal changes to code, and scales linearly on our DeepLearning11 servers: https://www.oreilly.com/ideas/distributed-tensorflow https://www.oreilly.com/ideas/distributed-tensorflow

p1esk 9y ago
why wouldn't you do it? Perhaps because I don't have hundreds of GPUs laying around? I'm a DL researcher fortunate enough to have a 4 Titan X workstation for my experiments. Most of my peers at other labs only have shared access to such workstations, and with the current trends in GPU pricing, that's not likely to change in the nearest future. More importantly, the lack of compute power is rarely a bottleneck in my research. Reading papers, brainstorming new ideas, and writing/debugging code are the most time consuming parts, it's not like I'm sitting idle while waiting for my models to train. Distributed TF is the standard in Google, will be the standard amongst everybody else within a couple of years. By "everyone else" you mean a few dozen corporations, which can justify spending millions of dollars on hardware to speed up their mission critical ML research?
- jamesblonde 9y ago
  Things can change quickly. When Ethereum goes to proof-of-stake the market will be flooded with cheap 1080Tis. A moderately well-funded lab should (by the end of 2018) be able to buy 100 1080Tis in DeepLearning11 servers for 100K Euro and do cool work in distributed learning and model architecture search. SoTA in Cifar-10 and ImageNet is now model-architecture search. This paper/blog was an eye-opener for me: https://blog.acolyer.org/2018/03/28/deep-learning-scaling-is-predictable-empirically/ https://blog.acolyer.org/2018/03/28/deep-learning-scaling-is...
  - tasuki 9y ago
    What's the ETA on that? I'd love me some cheap 1080Ti!
    - jamesblonde 9y ago
      Prices are already down. Keep an eye out here: https://www.reddit.com/r/nvidia/ https://www.reddit.com/r/nvidia/ Current estimates are that 75% of ethereum mining is done on 1080Tis. So, if that stops overnight - will they sell them or switch to some other bitcoin? My guess is that they will flood the market. Mostly they are under-volted when mining, so they won't burn out within a few weeks if you buy used.

p1esk 9y ago

why wouldn't you do it? Perhaps because I don't have hundreds of GPUs laying around? I'm a DL researcher fortunate enough to have a 4 Titan X workstation for my experiments. Most of my peers at other labs only have shared access to such workstations, and with the current trends in GPU pricing, that's not likely to change in the nearest future. More importantly, the lack of compute power is rarely a bottleneck in my research. Reading papers, brainstorming new ideas, and writing/debugging code are the most time consuming parts, it's not like I'm sitting idle while waiting for my models to train. Distributed TF is the standard in Google, will be the standard amongst everybody else within a couple of years. By "everyone else" you mean a few dozen corporations, which can justify spending millions of dollars on hardware to speed up their mission critical ML research?

jamesblonde 9y ago
Things can change quickly. When Ethereum goes to proof-of-stake the market will be flooded with cheap 1080Tis. A moderately well-funded lab should (by the end of 2018) be able to buy 100 1080Tis in DeepLearning11 servers for 100K Euro and do cool work in distributed learning and model architecture search. SoTA in Cifar-10 and ImageNet is now model-architecture search. This paper/blog was an eye-opener for me: https://blog.acolyer.org/2018/03/28/deep-learning-scaling-is-predictable-empirically/ https://blog.acolyer.org/2018/03/28/deep-learning-scaling-is...
- tasuki 9y ago
  What's the ETA on that? I'd love me some cheap 1080Ti!
  - jamesblonde 9y ago
    Prices are already down. Keep an eye out here: https://www.reddit.com/r/nvidia/ https://www.reddit.com/r/nvidia/ Current estimates are that 75% of ethereum mining is done on 1080Tis. So, if that stops overnight - will they sell them or switch to some other bitcoin? My guess is that they will flood the market. Mostly they are under-volted when mining, so they won't burn out within a few weeks if you buy used.