4 ms·

> 100TFlops that some NVidia GPUs are putting out... Theoretically. In practice you need to make sure you're not bandwidth bottlenecked, which is heavily depen

• dragontamer 7y ago
  > large fully connected layers are bandwidth bound Hmm, well that might be useful for the Radeon VII and its 1TBps bandwidth. The #1 network of interest to me is the Leela-Zero network, which is basically a "deep" neural net with 50 layers of tiny 3x3. Something like Leela-Zero would definitely benefit from those Tensor Cores, and it'd be small enough to fit inside of L1 cache (or maybe even shared memory. I haven't looked into it too deeply though). > Besides, nothing prevents AMD from implementing its own matrix multiply OP in the next generation of GPUs. While that's true, its not really something this generation of hardware buyers can think about. I think your point about large and fully connected layers may really benefit from the Radeon VII however. Someone should probably benchmark that kind of network between Radeon VII vs RTX 2080 (both $700-class GPUs).