8 ms·
Facebook Trains ImageNet in 1 Hour
- quadrature 9y agorelevant paper from facebook https://research.fb.com/publications/ImageNet1kIn1h/ https://research.fb.com/publications/ImageNet1kIn1h/
- boulos 9y agoThis is the URL that jonbaer originally submitted (sadly at an awkward time). I meant to send mail about it (there's no "Really! I vouch for this!" for low point stories that languish), but I see the result worked out anyway.
- qeternity 9y agoI can't seem to find it anywhere but what is the interconnect between servers being used? NVLink is used internally for GPU to GPU communication within a single box...correct? But this sounds like it takes a cluster of 32 of their 8 GPU Big Basin boxes.
- Seanny123 9y agotl;dr they found a clever way to spread the training across 256 GPUs by synchronising the stochastic gradient descent
- randyrand 9y agothis seems like the trivial, most obvious way to parallelize training across GPUs. Not, imo, clever. The important bit here is that they've shown that large mini batch sizes still can maintain accuracy if you slow the learning rate.
- Seanny123 9y agovery personally, clever means "well, that would have taken me a long time to figure out and code properly"
- willvarfar 9y agoLots of clever things are simply self-evident in hindsight.
- randyrand 9y agoSomeone asked 5 days ago on HN how to parallelize gpu learning. I had never thought about the problem before, but still came up with and gave this as the most obvious way. Took me maybe 20 seconds to think of. https://news.ycombinator.com/item?id=14510146 https://news.ycombinator.com/item?id=14510146 So the fact that someone with little experience can come up with this 'clever' technique, means either I'm really clever or it's not that clever. I'll go with the latter.
- opportune 9y agoJust because it's simple to explain at a high level doesn't make it trivial. Plenty of theoretically trivial solutions to problems are absolute pains to implement. I mean there are entire companies that at their core solve relatively "trivial" problems but employ huge numbers of engineers. Just because the core concept is simple to explain doesn't mean it's easy.
- randyrand 9y agoSomeone asked the other day how to parallelize gpu learning. I had never thought about the problem before, but still came up with and gave this as the most obvious way. https://news.ycombinator.com/item?id=14510146 https://news.ycombinator.com/item?id=14510146
- breatheoften 9y agoI'd always conceptualized decreasing batch size as a performance/memory optimization to deal with the fact that datasets don't all fit into memory and to reduce overall training time. You look at batch_size samples and compute the sum of the gradient of the errors to update the network weights so as to reduce the error -- shouldn't a larger batch_size inherently provide more information about the optimal direction of the update? It seems to my naive view like it should be "nice" from an accuracy perspective to look at more samples before making an adjustment to the network weights ...? In general, does changing the batch_size hyperparameter make a lot of difference on different problems ...? Does the right value for batch size tend to be problem specific and/or network architecture specific?
- nullc 9y ago[Warning: far from an expert here] No, batches also help you escape local minima.
- sxyuan 9y ago[Another non-expert commenting] Your comment didn't make sense to me at first, but I think I get it now. Even if you were able to fit the entire dataset into memory, batches are still a good idea, because optimizing on the entire dataset is non-convex and will likely lead you into a local minimum. However, what is a local minimum for one batch may not be a local minimum for the next batch, which helps you escape. This explains why optimization gets harder with very large batch sizes - the gradients for different batches become more similar (as they resemble the "global" gradient more closely), so you become more susceptible to local minima. I think this also explains why the learning rate scaling helps - it increases the variance across gradients, and helps you escape local minima.
- breatheoften 9y agoThis explanation of why larger batches might make optimization harder makes a fair amount of sense to me. This comment on Quora helps a bit as well - with a slightly different approach to characterizing the trade off of batch size/accuracy/runtime performance. https://www.quora.com/Intuitively-how-does-mini-batch-size-affect-the-performance-of-stochastic-gradient-descent https://www.quora.com/Intuitively-how-does-mini-batch-size-a... I wonder if rather than computing a single gradient for a large batch you could simultaneously compute a gradient for the batch and for several subsets of the batch -- then pick or combine the gradient subset(s) that most differ from the full batch result. Not sure if that would work out to a computational efficiency gain. Are there any optimizers that dynamically scale the batch size up/down based on an online metric?
- personjerry 9y agoHow much time would it have taken using Torch instead of Caffe2? (I still don't understand which I should use...)
- technics256 9y agoIt depends on your application. For most general purposes caffe2 will be fine. For research and pushing the limits pytorch is your best bet.
- antirez 9y agoTrivia: the Pieter in the paper is the one of the Redis fame.
- pietern 9y agoHi Salvatore! :D And there's even a tiny Redis dependency (optional though) in the code to generate these results. In particular the collective communication library needs a rendezvous phase where all nodes connect to their peers. Using Redis for this is one of the options. See: https://github.com/facebookincubator/gloo/tree/master/gloo/rendezvous https://github.com/facebookincubator/gloo/tree/master/gloo/r...
- antirez 9y agoHey Pieter! Wow cool :-) Thanks for the info. See you soon!
- horsecaptin 9y agoLooks like Nvidia is doing a proper content marketing push to compete against AMD.
- Eridrus 9y agoInterestingly enough I think the people who gain the most from this paper are cloud providers. Very few orgs will buy 256 GPUs, but with linear scaling, renting them makes a lot of sense.
- jamesblonde 9y agoSome observations: * for synchronous model-based distributed training to scale linearly, the time required to broadcast the model must be much larger than the time required for a worker (GPU) to process a batch * it's not strict synchronous training, as when gradients are computed at a worker, they are transmitted to all workers - so the driver doesn't have to send models to all 32 workers at the same time (8 GPUs per worker makes 256 GPUs in total). * there are extremely large batch sizes (8196) * it's a good network (50 Gb Ethernet, albeit not infiniband) So, the relative amount of work done training at each worker is much higher than the time spent broadcasting the model (which is quite small (~100 MB, i think)) to the workers for each iteration. For larger models with smaller batch sizes, this relationship would break down. The interesting contribution here is that you can have massive batch sizes and Facebook provided a heuristic for adjusting the learning rate to converge with such massive batch sizes.
- pietern 9y agoRe: your second point, it is strictly synchronous, though since there are 8 GPUs per process (thus have 1 process per machine) the gradient reduction is done in 3 phases. First they are reduced within the process, then across processes/machines, and then broadcast within the processes.
- jamesblonde 9y agoI misphrased that point, agreed. It's not classic driver-driven synchronous training, as you would do in tensorflow. It's using all-reduce (not available in tensorflow yet, i think).
- tanilama 9y agoA practically interesting paper. Some insights: 1.Larger batches requires large learning rate. And this paper shows that learning rate can even scale linearly with the batch size, which leading to extremely large learning rate/batch sizes. 2.Larger batch causes initial learning difficult, so this paper proposes to have a warm-up period where during the initial epochs, the learning rate grows from a smaller value gradually to a larger one. But if you are not Google/Facebook/Amazon/Microsoft, the experiment setting is unrealistic to you. Best AWS instances didn't come with the 50GBits network. For now, for the others, we would still stick to at most 8 GPUs on a single node, even your soul screams distributed :/
- cs702 9y agoThe approach could prove useful in a single machine too -- for example, in cases where increasing batch size might improve the efficiency of shuffling data back and forth from main memory to GPU memory. This provides an easy recipe for trying larger batch sizes to increase GPU usage.
- jamesblonde 9y agoI disagree that this is not an architecture for others. We are a research lab, and we're building a distributed cluster. Cheap infiniband. Lots of gtx1080Tis. It's not that expensive to have a 40-GPU cluster with 10 4U servers (about 100K Euro).
- gm-conspiracy 9y agoWhat mobo/CPU are you using? How much RAM in each server? Can you provide some more detailed specs? Thanks!
- jamesblonde 9y agoCPU is not that important. A motherboard that has PCIe 3.0 and 7 PCIx16 slots is ok for at least 3 GPUs. Each GPU takes typically two slots. Then, a >1400 Watts PSU. Here is a relatively cheap box that also has 8 disk slots (for hadoop): https://exxactcorp.com/index.php/solution/solu_detail/320 https://exxactcorp.com/index.php/solution/solu_detail/320
- deleted 9y ago[deleted]
- rlv-dan 9y agoDoes anyone know if there are any "consumer grade" image training kit out there? I'm thinking a software that you can train on your own images to put into categories.
- spuz 9y agoYes, you can use Tensorflow and Google's "inception" image recognition model to do this. The model by default is trained on the Imagenet database of images/categories, but Tensorflow allows you to retrain the last layer of the model on your own images to produce your own categorisation. Since you are only retraining the last layer of the model, you can easily do it within about 20 minutes on a laptop. See the tutorial here: https://www.tensorflow.org/tutorials/image_retraining https://www.tensorflow.org/tutorials/image_retraining
- jonbaer 9y agohttps://www.clarifai.com/custom-training https://www.clarifai.com/custom-training
- EvgeniyZh 9y agoHere is another paper on large batches https://arxiv.org/abs/1705.08741 https://arxiv.org/abs/1705.08741 It is better written and has more information IMHO