5 ms·
Are multi-GPU systems worth it until you have a TON of experience building parallel models? Almost all of the reference architectures I've seen are essentially
by web007 10y ago
Are multi-GPU systems worth it until you have a TON of experience building parallel models? Almost all of the reference architectures I've seen are essentially serial, and TF doesn't have good data-parallelism built in to make use of N GPUs, N > 1.
- tripzilch 10y agoMaybe this is a stupid idea, but if you had two GPUs, doesn't that basically mean, in terms of good parallelism, that at least you can train two models/networks at the same time, roughly? Because as long as you're in the research and development phase, that'll help cut your coding/training/testing/adjusting cycle. I assume you'll be tuning your hyperparameters, perhaps on somewhat smaller test models, but they will still take half a day or so to train? That means you can try (almost) twice as many hyperparameter configurations in the same time. It still helps to spin up a second test with a different selection of parameters, even if you haven't gotten the results back from the first test, right?