3 ms·

Latency between GPUs kills performance

• sudosysgen 7y ago
  It depends on just how huge the model is. Some models take multiple seconds to run/backpropagate and might take hundreds of gigabytes of memory, in which case it could be useful.
  • strbean 7y ago
    Also seems like a problem that could be partially solved by tailoring the NN architecture. Does that make sense?
    • ganzuul 7y ago
      Do you mean like Stochastic Gradient Descent does?