3 ms·

Well, it really depends on the computational intensity your algorithm needs. I've stumbled upon things of beauty porting things to GPUs, especially if you're go

Well, it really depends on the computational intensity your algorithm needs. I've stumbled upon things of beauty porting things to GPUs, especially if you're going to perform huge amount of operations based on a very small amount of data. As long as you don't have too much intermediate data, register spilling, etc. these GPU things do fly. They're also very impressive on NN-based workloads... Even something 2 or 3 gens behind can be game changing, with some optimization effort. Tensor libraries leave a lot on the floor to pick up, especially if you're not using the canned 'competition winning' networks.

cogman10 5y ago
Nature of the workload certainly matters a lot, and for a lot of work GPUs do memory bandwidth isn't always the limiting factor (though it often is, that's why consumer grade GPUs have 12GB of ram and beefier systems grade GPGPUs/TPUs have 40+GB of memory). Datastructure and locality matters a lot for GPGPU programming. While PCI-Express is really fast, it's got a lot of latency and is limited.
- mirker 5y ago
  Nit but bandwidth isn’t related to memory capacity. Bandwidth is a function of cache/access pattern and is independent of total memory size.
  - touisteur 5y ago
    The generous interpretation here would be that having huge amounts of ram that you can't access quickly enough wouldn't be that useful. I'm curious though, apart from huge NNs and raytracing scenes, what use cases call for so much RAM. I mean, apart from content lookup tables?
    - mirker 5y ago
      I believe the applications are mostly what you describe. You take a very general algorithm like a NN or raytracing and scale the model and/or data. For NN, it’s quite easy to use a huge amount of memory by making back-propagation chains huge, which is natural for deep models or a recurrent architecture. The model doesn’t have to be huge in a conventional sense; it just has to maintain state e.g., the recurrence. So, a big image model on video classification/segmentation (e.g., self-driving cars) is probably the ideal combination of memory consumption.

cogman10 5y ago

Nature of the workload certainly matters a lot, and for a lot of work GPUs do memory bandwidth isn't always the limiting factor (though it often is, that's why consumer grade GPUs have 12GB of ram and beefier systems grade GPGPUs/TPUs have 40+GB of memory). Datastructure and locality matters a lot for GPGPU programming. While PCI-Express is really fast, it's got a lot of latency and is limited.

mirker 5y ago
Nit but bandwidth isn’t related to memory capacity. Bandwidth is a function of cache/access pattern and is independent of total memory size.
- touisteur 5y ago
  The generous interpretation here would be that having huge amounts of ram that you can't access quickly enough wouldn't be that useful. I'm curious though, apart from huge NNs and raytracing scenes, what use cases call for so much RAM. I mean, apart from content lookup tables?
  - mirker 5y ago
    I believe the applications are mostly what you describe. You take a very general algorithm like a NN or raytracing and scale the model and/or data. For NN, it’s quite easy to use a huge amount of memory by making back-propagation chains huge, which is natural for deep models or a recurrent architecture. The model doesn’t have to be huge in a conventional sense; it just has to maintain state e.g., the recurrence. So, a big image model on video classification/segmentation (e.g., self-driving cars) is probably the ideal combination of memory consumption.