3 ms·

A practically interesting paper. Some insights: 1.Larger batches requires large learning rate. And this paper shows that learning rate can even scale linearly

A practically interesting paper. Some insights: 1.Larger batches requires large learning rate. And this paper shows that learning rate can even scale linearly with the batch size, which leading to extremely large learning rate/batch sizes. 2.Larger batch causes initial learning difficult, so this paper proposes to have a warm-up period where during the initial epochs, the learning rate grows from a smaller value gradually to a larger one. But if you are not Google/Facebook/Amazon/Microsoft, the experiment setting is unrealistic to you. Best AWS instances didn't come with the 50GBits network. For now, for the others, we would still stick to at most 8 GPUs on a single node, even your soul screams distributed :/

cs702 9y ago
The approach could prove useful in a single machine too -- for example, in cases where increasing batch size might improve the efficiency of shuffling data back and forth from main memory to GPU memory. This provides an easy recipe for trying larger batch sizes to increase GPU usage.
jamesblonde 9y ago
I disagree that this is not an architecture for others. We are a research lab, and we're building a distributed cluster. Cheap infiniband. Lots of gtx1080Tis. It's not that expensive to have a 40-GPU cluster with 10 4U servers (about 100K Euro).
- gm-conspiracy 9y ago
  What mobo/CPU are you using? How much RAM in each server? Can you provide some more detailed specs? Thanks!
  - jamesblonde 9y ago
    CPU is not that important. A motherboard that has PCIe 3.0 and 7 PCIx16 slots is ok for at least 3 GPUs. Each GPU takes typically two slots. Then, a >1400 Watts PSU. Here is a relatively cheap box that also has 8 disk slots (for hadoop): https://exxactcorp.com/index.php/solution/solu_detail/320 https://exxactcorp.com/index.php/solution/solu_detail/320
    - Retric 9y ago
      Some people are using water cooling setups to get 7 GPU's on a 7 PCI slot MB. You need to cut off the DVI that would otherwise occupy another slot, but overall it's not that time consuming and reduces your networking needs. EX: https://www.youtube.com/watch?v=9hsQmcSwGv0 https://www.youtube.com/watch?v=9hsQmcSwGv0
- malux85 9y ago
  Are you going to rent access to your cluster? :)
  - jamesblonde 9y ago
    To users in Sweden at www.hops.site It provides Jupyter/Hadoop/Spark/Tensorflow and GPUs (as a resource in YARN).
- jamesblonde 9y ago
  You can see the potential scale-out you can get for Tensorflow here on this PR: https://github.com/tensorflow/tensorflow/issues/2916 https://github.com/tensorflow/tensorflow/issues/2916
nl 9y ago
10 node GPU clusters are relatively common at eg Universities.
- dgacmu 9y ago
  Concur. I threw 2x GTX 980s in one of our testbeds two years ago just to make it more generally useful, giving us a 20 GPU cluster. Next time we do it, I'll aim for 4x1080ti or the equivalent. It's painful to power and cool those in lower-end university machine rooms, but it's possible. 10x 4x1080ti machines is only about 15kW, which is high but not insane. It's also only about $60k USD. Also something that a pool of faculty can afford. I've been experimenting with really low-end builds for that design recently. https://goo.gl/photos/6bDLjJqGAwhG7hGP9 https://goo.gl/photos/6bDLjJqGAwhG7hGP9 That's a consumer/gaming motherboard (asrock supercarrier), which doesn't have enough PCIe bandwidth to support the cards, but part of what we're researching are ways to reduce synchronization bandwidth. I wouldn't recommend that route as a general approach, though - not flexible enough for future uses. The 8x 1080ti Supermicro build posted a few days ago is probably a better choice: https://news.ycombinator.com/item?id=14508928 https://news.ycombinator.com/item?id=14508928 The problem is that one student can easily tie up the entire cluster for half the duration of her Ph.D. Machine learning people have voracious appetites for compute. :)
  - jamesblonde 9y ago
    Nice. We have added support for GPUs to YARN in Hops Hadoop. And users have CPU and GPU quotas in our version of YARN. Jobs are launched as either Tensorflow on Spark or Distributed Tensorflow on YARN. If the user exceeds her quota, currently we allow them to complete. But give them a ticking off. We could also abort jobs on quota violations, but like you alluded to - after 5 days of training, aborting that job would be a world of pain for us.

cs702 9y ago

The approach could prove useful in a single machine too -- for example, in cases where increasing batch size might improve the efficiency of shuffling data back and forth from main memory to GPU memory. This provides an easy recipe for trying larger batch sizes to increase GPU usage.

jamesblonde 9y ago

I disagree that this is not an architecture for others. We are a research lab, and we're building a distributed cluster. Cheap infiniband. Lots of gtx1080Tis. It's not that expensive to have a 40-GPU cluster with 10 4U servers (about 100K Euro).

gm-conspiracy 9y ago
What mobo/CPU are you using? How much RAM in each server? Can you provide some more detailed specs? Thanks!
- jamesblonde 9y ago
  CPU is not that important. A motherboard that has PCIe 3.0 and 7 PCIx16 slots is ok for at least 3 GPUs. Each GPU takes typically two slots. Then, a >1400 Watts PSU. Here is a relatively cheap box that also has 8 disk slots (for hadoop): https://exxactcorp.com/index.php/solution/solu_detail/320 https://exxactcorp.com/index.php/solution/solu_detail/320
  - Retric 9y ago
    Some people are using water cooling setups to get 7 GPU's on a 7 PCI slot MB. You need to cut off the DVI that would otherwise occupy another slot, but overall it's not that time consuming and reduces your networking needs. EX: https://www.youtube.com/watch?v=9hsQmcSwGv0 https://www.youtube.com/watch?v=9hsQmcSwGv0
malux85 9y ago
Are you going to rent access to your cluster? :)
- jamesblonde 9y ago
  To users in Sweden at www.hops.site It provides Jupyter/Hadoop/Spark/Tensorflow and GPUs (as a resource in YARN).
jamesblonde 9y ago
You can see the potential scale-out you can get for Tensorflow here on this PR: https://github.com/tensorflow/tensorflow/issues/2916 https://github.com/tensorflow/tensorflow/issues/2916

nl 9y ago

10 node GPU clusters are relatively common at eg Universities.

dgacmu 9y ago
Concur. I threw 2x GTX 980s in one of our testbeds two years ago just to make it more generally useful, giving us a 20 GPU cluster. Next time we do it, I'll aim for 4x1080ti or the equivalent. It's painful to power and cool those in lower-end university machine rooms, but it's possible. 10x 4x1080ti machines is only about 15kW, which is high but not insane. It's also only about $60k USD. Also something that a pool of faculty can afford. I've been experimenting with really low-end builds for that design recently. https://goo.gl/photos/6bDLjJqGAwhG7hGP9 https://goo.gl/photos/6bDLjJqGAwhG7hGP9 That's a consumer/gaming motherboard (asrock supercarrier), which doesn't have enough PCIe bandwidth to support the cards, but part of what we're researching are ways to reduce synchronization bandwidth. I wouldn't recommend that route as a general approach, though - not flexible enough for future uses. The 8x 1080ti Supermicro build posted a few days ago is probably a better choice: https://news.ycombinator.com/item?id=14508928 https://news.ycombinator.com/item?id=14508928 The problem is that one student can easily tie up the entire cluster for half the duration of her Ph.D. Machine learning people have voracious appetites for compute. :)
- jamesblonde 9y ago
  Nice. We have added support for GPUs to YARN in Hops Hadoop. And users have CPU and GPU quotas in our version of YARN. Jobs are launched as either Tensorflow on Spark or Distributed Tensorflow on YARN. If the user exceeds her quota, currently we allow them to complete. But give them a ticking off. We could also abort jobs on quota violations, but like you alluded to - after 5 days of training, aborting that job would be a world of pain for us.