3 ms·
There's a note on the site about being able to move computation to a process on a different machine, but it does mention copying the model. If I understand corr
by sudhirj 11y ago
There's a note on the site about being able to move computation to a process on a different machine, but it does mention copying the model. If I understand correctly, I think models tend to be quite small, though.
Does it actually make sense to train a model simultaneously on different machines? I can understand running the same model on lots of machines, of course.
- aconz2 11y agoI was interested in whether the license only permits a single machine.
- Sanddancer 11y agoNope, nothing in the license prohibiting it. Seems more a case that they just haven't released the glue for it to run on multiple machines, but hope to have it added one way or another.
- prajit 11y agoYes, there are two types of parallelism: model and data. Data parallelism is simply training the the model on multiple computers with different minibatches, and aggregating the gradients. Model parallelism is hosting different parts of the model on different computers. Of course, these two parallelisms can be combined. A good explanation is the One Weird Trick paper: http://arxiv.org/abs/1404.5997 http://arxiv.org/abs/1404.5997
- dr_zoidberg 11y agoIf there is randomness in some part of the training process (and most surely there will be), then it makes sense. Different trainings may end up in different local minimums/maximums, so there is value in running multiple trainings simultaneously in different machines.