5 ms·

If it's so fast, I'd like to see it working on a smaller scale on a CPU. Every new deep learning paper that comes out, I'm disappointed that it needs... - A $

If it's so fast, I'd like to see it working on a smaller scale on a CPU. Every new deep learning paper that comes out, I'm disappointed that it needs... - A $500-$1,000 GPU - A huge proprietary NVidia driver - Some odd language or language extensions, usually CUDA - Python

rdedev 5y ago
Just curious. What is your objections wrt the last 3 points ?
- ReactiveJelly 5y ago
  I like FOSS a lot. Normal programming languages have relatively small downloads and run on normal CPUs dating about 10 years back with almost no issue. GPU workloads always want some odd driver that has a gigantic download, and they're constantly coming up with new reasons to force you to the newest APIs, which means you have to buy new chips that have the right architecture or firmware for the new APIs. So I have to buy this co-processor, and then I can't even treat it like a black box that I send commands to, I need a gigabyte-scale SDK or something to issue the commands on my behalf. I can't stand it. It's as if there was a tiny window when programming was simple, after I learned about FOSS, and before GPGPU caught on. As if the personal computer really will turn out to have been a fad.
  - astrange 5y ago
    This research uses Nvidia products because it was released by Nvidia, who have hired approximately every graphics researcher out there.
  - olliej 5y ago
    Ok, GPGPU isn’t “general purpose” in the basic sense, it means “not just graphics”. No CPU is going to be able to get performance in NNs that matches that of a GPU. A CPU simply cannot do the work. The closest a “general” CPU gets to that kind of thing are the big vector machines like the old Crays or Itanium’s packet architecture. Programming for either of those architectures is non trivial, and for normal software those architectures are slower than normal CPUs. Despite the trade offs those systems made, consumer GPUs ended up with better performance because a lot of the things and general CPU has to do interfere with performance of pure numerical computation.
jonas21 5y ago
Why? The point of research is to push the limits of what's possible, not to build something that runs on every single platform. I find it remarkable that most recent deep learning papers release the source code needed to reproduce their result -- and even more remarkable that many papers, like this one, can be reproduced on hardware that a hobbyist can afford. And if you'd like this to run on a CPU, you're welcome to port it. The code is open source after all.
hwers 5y ago
you can get really really far with a colab pro+ account these days (just $50 per month)
acchow 5y ago
Your smartphone in 10 years will have an equivalent GPU
- ReactiveJelly 5y ago
  I haven't learned Android in the last 10 years, doomsday principle says I'm probably not gonna learn it in the next 10 either.
  - sva_ 5y ago
    We'll hopefully have WebGPU by then
olliej 5y ago
The reason they run on a GPU isn’t spite. It’s because the work for neural net based ML is inherently dependent on vast amounts of independent floating point operations. CPUs tend to have very few FPUs per core, so you max out a modern systems CPUs idealised throughput at maybe 40-80 concurrent streams. On top of that the FPUs on a CPU are generally require to perform fully compliant ieee754 arithmetic at at least 32bit of precision. Modern GPUs can have that number of FPUs per hardware thread and then have a few hundred of those hardware threads. Each of those GPU FPUs are also faster as they can both elide some elements of ieee754, and operate at lower precision (fp16) to get even more performance. So you could read the paper, and implement it on a CPU and the very best that you, or anyone, could do would be literal orders of magnitude slower than the GPU implementation. That’s why you don’t see them doing it on a CPU, let alone in Python.
- adammarples 5y ago
  The reason it runs onGPU is because this research was literally done by NVIDIA!
  - olliej 5y ago
    Nvidia also makes CPUs. The reason the research is coming out of nvidia is because this kind of research is inherently GPU limited. So if it came out of AMD, Intel, Google, or Apple, it would be dependent on either GPU, or non-programmable NN specific hardware. If it came out of academia it would still be on a GPU, because none of this is remotely practical on a CPU.
astrange 5y ago
You can rent the GPU from Google Colab. (It's actually more like $15k than $1k, which makes you wonder why Colab is so cheap.) Why everything's written in Python I couldn't tell you.
- evgen 5y ago
  Everything is written in Python because early in on the process people realized that you are not doing anything special and are certainly not doing actual math: you are just wiring together libraries that you barely understand using a cookbook that someone else provided for you to generate results that you cannot explain so that an investor can tick a checkbox for a feature list that you never see. If you are just gluing together C/C++ libraries then there are worse languages that could have been selected, but once momentum gathered behind Python as the glue language it was hard to divert to another language (e.g. how hard the Julia folks are trying, and failing, to do just that...)
wokwokwok 5y ago
To be fair, almost every deep learning paper that comes out needs something like 10x GPU cloud nodes to run on. The days where you could run anything significant on a single 1k graphics card are long gone. This is, ironically, the first time that (I’m aware of) you could distill this Nerf stuff down into a size that runs on a single consumer GPU (RTX 2x or higher) …so, some of your points are fair, but hey, at least these folk are trying to bring this down from “only usable by large corporations” to “runs on your desktop”. I mean, it’s not perfect, but I think in this case you’re complaining about something abstract, when these folk are actually going in the right direction.
stjohnswarts 5y ago
I'm sorry if you're doing GPU calculations then you want a powerful video card unless your research is on improving performance of algos on less powerful hardware. There are only so many hours in a day.
PufPufPuf 5y ago
Machine learning research often scales up to solve a new problem, and then scales down the solution until it's actually usable. Object detection, for example, is now fully usable on a phone CPU.
lpapez 5y ago
By this logic you would be disappointed by any space related discovery because it needs a multimillion dollar telescope...
dr_zoidberg 5y ago
Well we can shorten that list if you're able to write your models in tensorflow 1.15 to: Windows 10 and Python 3.6+. Microsoft has done something quite interesting with tensorflow-directml [0, 1]. A friend is training convolutional networks on a Ryzen 5 3500u ultrabook, at about the same speed my old notebook with a GeForce 940mx could. I'm tempted to test it on a 4600H when I have a bit of time, it could be interesting if the iGPU is able to access a large portion of the 24GB of RAM that system has. [0] https://pypi.org/project/tensorflow-directml/ https://pypi.org/project/tensorflow-directml/ [1] https://docs.microsoft.com/en-us/windows/ai/directml/gpu-tensorflow-windows https://docs.microsoft.com/en-us/windows/ai/directml/gpu-ten...

rdedev 5y ago

Just curious. What is your objections wrt the last 3 points ?

ReactiveJelly 5y ago
I like FOSS a lot. Normal programming languages have relatively small downloads and run on normal CPUs dating about 10 years back with almost no issue. GPU workloads always want some odd driver that has a gigantic download, and they're constantly coming up with new reasons to force you to the newest APIs, which means you have to buy new chips that have the right architecture or firmware for the new APIs. So I have to buy this co-processor, and then I can't even treat it like a black box that I send commands to, I need a gigabyte-scale SDK or something to issue the commands on my behalf. I can't stand it. It's as if there was a tiny window when programming was simple, after I learned about FOSS, and before GPGPU caught on. As if the personal computer really will turn out to have been a fad.
- astrange 5y ago
  This research uses Nvidia products because it was released by Nvidia, who have hired approximately every graphics researcher out there.
- olliej 5y ago
  Ok, GPGPU isn’t “general purpose” in the basic sense, it means “not just graphics”. No CPU is going to be able to get performance in NNs that matches that of a GPU. A CPU simply cannot do the work. The closest a “general” CPU gets to that kind of thing are the big vector machines like the old Crays or Itanium’s packet architecture. Programming for either of those architectures is non trivial, and for normal software those architectures are slower than normal CPUs. Despite the trade offs those systems made, consumer GPUs ended up with better performance because a lot of the things and general CPU has to do interfere with performance of pure numerical computation.

jonas21 5y ago

Why? The point of research is to push the limits of what's possible, not to build something that runs on every single platform. I find it remarkable that most recent deep learning papers release the source code needed to reproduce their result -- and even more remarkable that many papers, like this one, can be reproduced on hardware that a hobbyist can afford. And if you'd like this to run on a CPU, you're welcome to port it. The code is open source after all.

hwers 5y ago

you can get really really far with a colab pro+ account these days (just $50 per month)

acchow 5y ago

Your smartphone in 10 years will have an equivalent GPU

ReactiveJelly 5y ago
I haven't learned Android in the last 10 years, doomsday principle says I'm probably not gonna learn it in the next 10 either.
- sva_ 5y ago
  We'll hopefully have WebGPU by then

olliej 5y ago

The reason they run on a GPU isn’t spite. It’s because the work for neural net based ML is inherently dependent on vast amounts of independent floating point operations. CPUs tend to have very few FPUs per core, so you max out a modern systems CPUs idealised throughput at maybe 40-80 concurrent streams. On top of that the FPUs on a CPU are generally require to perform fully compliant ieee754 arithmetic at at least 32bit of precision. Modern GPUs can have that number of FPUs per hardware thread and then have a few hundred of those hardware threads. Each of those GPU FPUs are also faster as they can both elide some elements of ieee754, and operate at lower precision (fp16) to get even more performance. So you could read the paper, and implement it on a CPU and the very best that you, or anyone, could do would be literal orders of magnitude slower than the GPU implementation. That’s why you don’t see them doing it on a CPU, let alone in Python.

adammarples 5y ago
The reason it runs onGPU is because this research was literally done by NVIDIA!
- olliej 5y ago
  Nvidia also makes CPUs. The reason the research is coming out of nvidia is because this kind of research is inherently GPU limited. So if it came out of AMD, Intel, Google, or Apple, it would be dependent on either GPU, or non-programmable NN specific hardware. If it came out of academia it would still be on a GPU, because none of this is remotely practical on a CPU.

astrange 5y ago

You can rent the GPU from Google Colab. (It's actually more like $15k than $1k, which makes you wonder why Colab is so cheap.) Why everything's written in Python I couldn't tell you.

evgen 5y ago
Everything is written in Python because early in on the process people realized that you are not doing anything special and are certainly not doing actual math: you are just wiring together libraries that you barely understand using a cookbook that someone else provided for you to generate results that you cannot explain so that an investor can tick a checkbox for a feature list that you never see. If you are just gluing together C/C++ libraries then there are worse languages that could have been selected, but once momentum gathered behind Python as the glue language it was hard to divert to another language (e.g. how hard the Julia folks are trying, and failing, to do just that...)

wokwokwok 5y ago

To be fair, almost every deep learning paper that comes out needs something like 10x GPU cloud nodes to run on. The days where you could run anything significant on a single 1k graphics card are long gone. This is, ironically, the first time that (I’m aware of) you could distill this Nerf stuff down into a size that runs on a single consumer GPU (RTX 2x or higher) …so, some of your points are fair, but hey, at least these folk are trying to bring this down from “only usable by large corporations” to “runs on your desktop”. I mean, it’s not perfect, but I think in this case you’re complaining about something abstract, when these folk are actually going in the right direction.

stjohnswarts 5y ago

I'm sorry if you're doing GPU calculations then you want a powerful video card unless your research is on improving performance of algos on less powerful hardware. There are only so many hours in a day.

PufPufPuf 5y ago

Machine learning research often scales up to solve a new problem, and then scales down the solution until it's actually usable. Object detection, for example, is now fully usable on a phone CPU.

lpapez 5y ago

By this logic you would be disappointed by any space related discovery because it needs a multimillion dollar telescope...

dr_zoidberg 5y ago

Well we can shorten that list if you're able to write your models in tensorflow 1.15 to: Windows 10 and Python 3.6+. Microsoft has done something quite interesting with tensorflow-directml [0, 1]. A friend is training convolutional networks on a Ryzen 5 3500u ultrabook, at about the same speed my old notebook with a GeForce 940mx could. I'm tempted to test it on a 4600H when I have a bit of time, it could be interesting if the iGPU is able to access a large portion of the 24GB of RAM that system has. [0] https://pypi.org/project/tensorflow-directml/ https://pypi.org/project/tensorflow-directml/ [1] https://docs.microsoft.com/en-us/windows/ai/directml/gpu-tensorflow-windows https://docs.microsoft.com/en-us/windows/ai/directml/gpu-ten...