4 ms·
If you are using a GPU, parallelisation is not difficult.
by milesrout 10y ago
If you are using a GPU, parallelisation is not difficult.
- jacquesm 10y agoI believe you may be missing something. Even on a GPU there are synchronization points in some algorithms and data needs to be passed from one processor to another once you spill over the boundary of what can be held in the RAM. Also, some parts of an algorithm may not be parallelizable at all. GPU parallelization is only 'not difficult' if: - all your data fits on a single GPU - your code is embarassingly parallel - for the total duration of one computation Bonus if you do can use the output of one computation as the input of the next. In all other kinds of computations the usual bottle-necks apply. https://en.wikipedia.org/wiki/Amdahl%27s_law https://en.wikipedia.org/wiki/Amdahl%27s_law
- deepnotderp 10y agoI was talking about multi-GPU parallelization. Multi GPU SGD parallelization is not easy. Single GPU parallelization is trivial of course.