4 ms·

Uh, so what happened? Were you working on a GPU project at Google before management came around to this way of thinking, and it got canned? That sucks, but th

by chubot 11y ago

Uh, so what happened? Were you working on a GPU project at Google before management came around to this way of thinking, and it got canned? That sucks, but that kind of thing happens a lot. It's not that surprising. Google wanted a homogeneous infrastructure for a long time. But a new application (neural networks) motivated a new infrastructure (GPUs in clusters). There was that Stanford paper everyone talks about which compared training a neural network on Google's 16K cores (DistBelief paper) vs a handful of GPUs with infiniband. Even Andrew Ng has seemed to subtly criticize Google for thinking "the old way" (i.e. "cloud" technology vs HPC technology, HPC being more effective for neural nets) Also, I don't quite understand your comment about brawny vs. wimpy. Yes Urs wrote a rebuttal against wimpy cores. As far as I remember, his argument is basically that there is some portion of software on the critical latency path that's not multithreaded. wimpy cores make the most sense in a world of perfect parallelism.

varelse 11y ago
I'm done ranting about Google and GPUs at last. From a technical standpoint, I believe TensorFlow vindicates my viewpoint and that's enough for me. This is closure I really needed, OK? And besides, now I can concentrate my energy on asking WTH happened to Android performance between 4.x and 5.x J/K? And from a professional standpoint, I moved on and fixed the damage I let them inflict on my career. I eventually built exactly what I was told couldn't and shouldn't be done with GPUs. But back then, the argument Google made to me was that too much of the workloads I wanted to run on GPUs were serialized (hence brawny versus wimpy). And from a classical parallel algorithms perspective, they were correct. The problem is that from a GPU programming perspective, they weren't even wrong(tm). And at the time, I had ~5 years of CUDA programming (very early adopter) and 30,000+ lines of CUDA code to back that up whereas the people I tried to convince had zero such experience so there was just no way. My argument boiled down to TLDR: O(n^2) or higher probably belongs on GPUs. Would anyone disagree these days?
- lmeyerov 11y ago
  More economic argument: any sufficiently restricted and important algorithm will fund enough (1) PhDs to pull out the parallelism and then (2) devoted engineers to map it down successively close to the metal (perf/watt), even if the legacy code is polished. CPU->GPU->FPGA->ASICS. Regex, neural nets, doesn't really matter.

by chubot 11y ago

varelse 11y ago

I'm done ranting about Google and GPUs at last. From a technical standpoint, I believe TensorFlow vindicates my viewpoint and that's enough for me. This is closure I really needed, OK? And besides, now I can concentrate my energy on asking WTH happened to Android performance between 4.x and 5.x J/K? And from a professional standpoint, I moved on and fixed the damage I let them inflict on my career. I eventually built exactly what I was told couldn't and shouldn't be done with GPUs. But back then, the argument Google made to me was that too much of the workloads I wanted to run on GPUs were serialized (hence brawny versus wimpy). And from a classical parallel algorithms perspective, they were correct. The problem is that from a GPU programming perspective, they weren't even wrong(tm). And at the time, I had ~5 years of CUDA programming (very early adopter) and 30,000+ lines of CUDA code to back that up whereas the people I tried to convince had zero such experience so there was just no way. My argument boiled down to TLDR: O(n^2) or higher probably belongs on GPUs. Would anyone disagree these days?

lmeyerov 11y ago
More economic argument: any sufficiently restricted and important algorithm will fund enough (1) PhDs to pull out the parallelism and then (2) devoted engineers to map it down successively close to the metal (perf/watt), even if the legacy code is polished. CPU->GPU->FPGA->ASICS. Regex, neural nets, doesn't really matter.