As a team working a _lot_ on modern GPUs in the cloud for pretty normal (non-just-deep-learning) compute, we have a different experience: 1. Big GPU memories,

As a team working a _lot_ on modern GPUs in the cloud for pretty normal (non-just-deep-learning) compute, we have a different experience: 1. Big GPU memories, sufficiently flexible compute units, breakthroughs in irregular algorithms & tooling => we found end-to-end GPU computing quite viable for most of our data processing. GOAI (link below) is becoming the next big leap here. 2. Getting data on device isn't the issue: Slow communication part is data warehouse -> device (3 GB/s), not cpu->gpu (30-300 GB/s). No significant advantage to CPU here, and for the slight CPU tax here, I expect network->GPU to become cloud-standard. 3. Multi-GPU & GPU memory is insane: The exciting thing for us is (a) all cloud providers have GPUs and are updating them regularly and (b) Nvidia is providing multi-GPU designs with a lot of fast shared memory (ex: DGX2/HGX2). It's a world where for most datasets, there is more GPU RAM than workload data, and 100K+ threads means a node can destroy a dataset in a few steps. For the bigger datasets that require streaming, GPU memory bandwidth & thread count is a monster. 1+3 => We're playing with interactive (<100ms) tasks on dataset sizes that, as interactive tool devs, we regularly get surprised by. Ex: For some of our high-level interactive visual analytics task that renders in the browser, we'll run a a GPU compute pipeline end-to-end (5+ GPU kernels), where most subtasks take 0-1ms (so many threads!), maybe repeatedly for simulations, and within 20ms, get back to the CPU, and then send down to the browser. Amdahl's Law hits us.. at the network compression & transfer layer. For a sense of where members of the GOAI (gpu open analytics initiative) community members are going, from the perspective of web devs, we wrote up some planning notes on our path to making regular nodejs apps plug into all this: https://www.graphistry.com/blog/js-gpus-ml-arrow-goai https://www.graphistry.com/blog/js-gpus-ml-arrow-goai .