5 ms·

CPU and GPU trends over time

62 points by TalGalili 16y ago

Derbasti 16y ago
Shit, I did not realize GPUs packed such a tremendous performance advantage over CPUs (FLOPs wise). They seem to be several orders of magnitude ahead--that is positively mindblowing! And yet, all GPUs and CPUs of the whole world can just barely match one human brain (http://arstechnica.com/science/news/2011/02/adding-up-the-worlds-storage-and-computation-capacities.ars http://arstechnica.com/science/news/2011/02/adding-up-the-wo...). If that's not mindblowing, then what?
- jules 16y ago
  They compare instructions per second with nerve impulses per second. That's a pretty useless comparison. I'm sure that all GPUs and CPUs could simulate a human brain more quickly than a human brain could simulate all CPUs and GPUs.
  - adrianN 16y ago
    I'd venture to say that depends strongly on the problem the brain is trying to solve and the problem the processors are trying to solve.
- abecedarius 16y ago
  In the Whole Brain Emulation Roadmap there's a table of published guesses at computing requirements: most were in the range 10^14..10^18 FLOPS. (Plus a few outliers from people who think the brain does molecular-scale or quantum computing in microtubules.) The 500th biggest computer in the latest Top500 does around 3E13 FLOPS, while the top does 3E15 -- using GPUs, incidentally. Your link's on the high end of the range, though they're all kind of guessing. http://www.philosophy.ox.ac.uk/__data/assets/pdf_file/0019/3853/brain-emulation-roadmap-report.pdf http://www.philosophy.ox.ac.uk/__data/assets/pdf_file/0019/3... http://www.top500.org/list/2010/11/100 http://www.top500.org/list/2010/11/100
- postfuturist 16y ago
  Looks like about 1 order of magnitude, not several.
cyrus_ 16y ago
There are many architectural differences between CPUs and GPUs that make it difficult to draw the kinds of conclusions suggested by these graphs. Data parallel algorithms with high computation-to-communication ratios, good data locality and low local branch divergence will see these kinds of speedups when rewritten for GPUs. This is an important subset of algorithms, but don't expect GPUs to replace CPUs for more general-purpose computation any time soon.
- manvsmachine 16y ago
  Very true. It is for the same reason that the article shows GPU progress outpacing CPU's in regards to FLOPS. The limited scope of a GPU's functionality allows most of the processor's silicon to be arithmetic units. CPU's, on the other hand, have huge sections of their real estate dedicated to handling conditional logic (something most GPU's fail miserably at), legacy instruction handling, virtualization, etc.
6ren 16y ago
If someone figured out how to write regular apps on a GPU (which would amount to parallelizing the app), it could change everything. 1. What would be a first step towards this? It seems the big problem is that most code is sequential... and the opportunity is those parts that don't need to be, but just happen to be written that way. 2. Perhaps another opportunity is to try to remove the layers that make modern software so slow. Can they be distributed across all pipelines somehow? 3. Finally, a radical approach is not to try to convert existing apps, but to think about the whole problem of an app from the very beginning (what inputs will you get, what outputs do you want), and try to write that directly on a GPU. By making it concrete, you might see new tricks, perhaps by redefining the problem (the app's features).
- Synaesthesia 16y ago
  It can't really be done trivially. A GPU is a massively parallel (hundreds of cores), very limited type of CPU, specifically made to execute vector operations.
  - arctangent 16y ago
    I agree, and I can't wait for someone (or, more likely, some group) to figure out how to parallelise arbitrary programs across GPUs and CPUs in a way that is transparent to the programmer - or as nearly so as possible. I spend a lot of my "fun time" writing agent-based systems which involve multiple objects all running the same set of code within a single environment. A person could be forgiven for thinking that computers should be able to "just run it" on all these CPU or GPU cores but it doesn't work out that way right now unfortunately. I have tried Haskell but it's quite hard to do real work with, or maybe I am just dumb :-) No doubt answers will come with time. Perhaps the "pain" of slow CPU cores will force one (or some) of us to put in some time to writing the type of solution I refer to in my first paragraph. (The related concept of "peak oil" springs to mind.)
- sedachv 16y ago
  This is what parallelizing Fortran compilers do. The space of programs that can be automatically parallelized is all computable programs (see http://en.wikipedia.org/wiki/Church–Rosser_theorem http://en.wikipedia.org/wiki/Church–Rosser_theorem). The space of programs that can be efficiently automatically parallelized is very small.
endlessvoid94 16y ago
What does "computational power" mean in this context? There's a graph but no explanation...
- Synaesthesia 16y ago
  Probably number of floating point operations per second.
- narkee 16y ago
  If you click on the graph, you can see "MIPS" (millions of instructions per second) on the dependent axis.
- narkee 16y ago
  If you click on the graph, you can see "MIPS" (millions of instructions per second) on the dependent axis.
Synaesthesia 16y ago
If this chart from nVidia is to be believed, the CPU speed increase will continue for many years to come. They're projecting almost a 100x increase in speed from Tegra 2, in 2014! http://images.anandtech.com/reviews/SoC/NVIDIA/Kal-El/roadmap_sm.jpg http://images.anandtech.com/reviews/SoC/NVIDIA/Kal-El/roadma...
- Klinky 16y ago
  How are they going to get there? They could very easily pump up the shader count to get something that is "100x" performance, while not actually being 100x as powerful for general purpose operations. The graph looks shady to say the least.
Rexxar 16y ago
I would have preferred a logarithmic scale for the last graph (http://csgillespie.files.wordpress.com/2011/01/gpu3.png http://csgillespie.files.wordpress.com/2011/01/gpu3.png).
- lwat 16y ago
  Here's one I just made using his data: http://imgur.com/TSFYB http://imgur.com/TSFYB
  - eru 16y ago
    Thanks. Looks more similar now than on the linear scale.
ChuckMcM 16y ago
Thanks for posting this, its an excellent insight. It was interesting when Intel fail to deliver on their roadmap. Here is a link from 2000 where Intel was alleged to be saying 10Ghz by 2011 http://www.geek.com/articles/chips/intel-predicts-10ghz-chips-by-2011-20000726/ http://www.geek.com/articles/chips/intel-predicts-10ghz-chip... Which of course lead to the great Multi-core debate (AMD started it, Intel followed with two physical die on a package later to integrate into a single die). Which lead to the great Memory channel bottleneck (Intel was in-order single channel, AMD was multi-channel) and resolution (everyone now is multi-channel). GPUs on the other hand have always had memory bandwidth issues so they went wide, the GPU equivalent of multi-channel, from the start. They also benefit from a feature of graphics which was 'do this same algorithm on all of these texels' which is straight forward to parallelize. And as nuclear weapons designers, and genomic researchers will tell you, GPUs are also useful in doing MIMD simulations. Its less clear what the parallel architecture breakthrough is for OLAP type problems (or parallel file systems as an even easier example) These have lots of operations in parallel but they also have high connectivity of data synchronicity across those parallel threads which, at the moment, brings everything down to earth. Fun times!
Xpirate 16y ago
Some real world benchmarks for GPU vs CPU vs CPUs from helix/biowulf at NIH: http://biowulf.nih.gov/apps/bioinf-gpu.html http://biowulf.nih.gov/apps/bioinf-gpu.html From the summary : While there is a definite speedup from running on 1 cpu only to running on 1 cpu + 1 gpu, the multi-threaded versions generally have better overall performance. It is likely that once the CUDA versions are multi-threaded, running on the GPU nodes will confer additional advantages, but at present the codes are not yet mature.
GICodeWarrior 16y ago
This seems like an apples to oranges comparison with different scales and units...

62 points by TalGalili 16y ago

Shit, I did not realize GPUs packed such a tremendous performance advantage over CPUs (FLOPs wise). They seem to be several orders of magnitude ahead--that is positively mindblowing! And yet, all GPUs and CPUs of the whole world can just barely match one human brain (http://arstechnica.com/science/news/2011/02/adding-up-the-worlds-storage-and-computation-capacities.ars http://arstechnica.com/science/news/2011/02/adding-up-the-wo...). If that's not mindblowing, then what?

jules 16y ago
They compare instructions per second with nerve impulses per second. That's a pretty useless comparison. I'm sure that all GPUs and CPUs could simulate a human brain more quickly than a human brain could simulate all CPUs and GPUs.
- adrianN 16y ago
  I'd venture to say that depends strongly on the problem the brain is trying to solve and the problem the processors are trying to solve.
abecedarius 16y ago
In the Whole Brain Emulation Roadmap there's a table of published guesses at computing requirements: most were in the range 10^14..10^18 FLOPS. (Plus a few outliers from people who think the brain does molecular-scale or quantum computing in microtubules.) The 500th biggest computer in the latest Top500 does around 3E13 FLOPS, while the top does 3E15 -- using GPUs, incidentally. Your link's on the high end of the range, though they're all kind of guessing. http://www.philosophy.ox.ac.uk/__data/assets/pdf_file/0019/3853/brain-emulation-roadmap-report.pdf http://www.philosophy.ox.ac.uk/__data/assets/pdf_file/0019/3... http://www.top500.org/list/2010/11/100 http://www.top500.org/list/2010/11/100
postfuturist 16y ago
Looks like about 1 order of magnitude, not several.

cyrus_ 16y ago

There are many architectural differences between CPUs and GPUs that make it difficult to draw the kinds of conclusions suggested by these graphs. Data parallel algorithms with high computation-to-communication ratios, good data locality and low local branch divergence will see these kinds of speedups when rewritten for GPUs. This is an important subset of algorithms, but don't expect GPUs to replace CPUs for more general-purpose computation any time soon.

manvsmachine 16y ago
Very true. It is for the same reason that the article shows GPU progress outpacing CPU's in regards to FLOPS. The limited scope of a GPU's functionality allows most of the processor's silicon to be arithmetic units. CPU's, on the other hand, have huge sections of their real estate dedicated to handling conditional logic (something most GPU's fail miserably at), legacy instruction handling, virtualization, etc.

6ren 16y ago

If someone figured out how to write regular apps on a GPU (which would amount to parallelizing the app), it could change everything. 1. What would be a first step towards this? It seems the big problem is that most code is sequential... and the opportunity is those parts that don't need to be, but just happen to be written that way. 2. Perhaps another opportunity is to try to remove the layers that make modern software so slow. Can they be distributed across all pipelines somehow? 3. Finally, a radical approach is not to try to convert existing apps, but to think about the whole problem of an app from the very beginning (what inputs will you get, what outputs do you want), and try to write that directly on a GPU. By making it concrete, you might see new tricks, perhaps by redefining the problem (the app's features).

Synaesthesia 16y ago
It can't really be done trivially. A GPU is a massively parallel (hundreds of cores), very limited type of CPU, specifically made to execute vector operations.
- arctangent 16y ago
  I agree, and I can't wait for someone (or, more likely, some group) to figure out how to parallelise arbitrary programs across GPUs and CPUs in a way that is transparent to the programmer - or as nearly so as possible. I spend a lot of my "fun time" writing agent-based systems which involve multiple objects all running the same set of code within a single environment. A person could be forgiven for thinking that computers should be able to "just run it" on all these CPU or GPU cores but it doesn't work out that way right now unfortunately. I have tried Haskell but it's quite hard to do real work with, or maybe I am just dumb :-) No doubt answers will come with time. Perhaps the "pain" of slow CPU cores will force one (or some) of us to put in some time to writing the type of solution I refer to in my first paragraph. (The related concept of "peak oil" springs to mind.)
sedachv 16y ago
This is what parallelizing Fortran compilers do. The space of programs that can be automatically parallelized is all computable programs (see http://en.wikipedia.org/wiki/Church–Rosser_theorem http://en.wikipedia.org/wiki/Church–Rosser_theorem). The space of programs that can be efficiently automatically parallelized is very small.

endlessvoid94 16y ago

What does "computational power" mean in this context? There's a graph but no explanation...

Synaesthesia 16y ago
Probably number of floating point operations per second.
narkee 16y ago
If you click on the graph, you can see "MIPS" (millions of instructions per second) on the dependent axis.
narkee 16y ago
If you click on the graph, you can see "MIPS" (millions of instructions per second) on the dependent axis.

Synaesthesia 16y ago

If this chart from nVidia is to be believed, the CPU speed increase will continue for many years to come. They're projecting almost a 100x increase in speed from Tegra 2, in 2014! http://images.anandtech.com/reviews/SoC/NVIDIA/Kal-El/roadmap_sm.jpg http://images.anandtech.com/reviews/SoC/NVIDIA/Kal-El/roadma...

Klinky 16y ago
How are they going to get there? They could very easily pump up the shader count to get something that is "100x" performance, while not actually being 100x as powerful for general purpose operations. The graph looks shady to say the least.

Rexxar 16y ago

I would have preferred a logarithmic scale for the last graph (http://csgillespie.files.wordpress.com/2011/01/gpu3.png http://csgillespie.files.wordpress.com/2011/01/gpu3.png).

lwat 16y ago
Here's one I just made using his data: http://imgur.com/TSFYB http://imgur.com/TSFYB
- eru 16y ago
  Thanks. Looks more similar now than on the linear scale.

ChuckMcM 16y ago

Thanks for posting this, its an excellent insight. It was interesting when Intel fail to deliver on their roadmap. Here is a link from 2000 where Intel was alleged to be saying 10Ghz by 2011 http://www.geek.com/articles/chips/intel-predicts-10ghz-chips-by-2011-20000726/ http://www.geek.com/articles/chips/intel-predicts-10ghz-chip... Which of course lead to the great Multi-core debate (AMD started it, Intel followed with two physical die on a package later to integrate into a single die). Which lead to the great Memory channel bottleneck (Intel was in-order single channel, AMD was multi-channel) and resolution (everyone now is multi-channel). GPUs on the other hand have always had memory bandwidth issues so they went wide, the GPU equivalent of multi-channel, from the start. They also benefit from a feature of graphics which was 'do this same algorithm on all of these texels' which is straight forward to parallelize. And as nuclear weapons designers, and genomic researchers will tell you, GPUs are also useful in doing MIMD simulations. Its less clear what the parallel architecture breakthrough is for OLAP type problems (or parallel file systems as an even easier example) These have lots of operations in parallel but they also have high connectivity of data synchronicity across those parallel threads which, at the moment, brings everything down to earth. Fun times!

Xpirate 16y ago

Some real world benchmarks for GPU vs CPU vs CPUs from helix/biowulf at NIH: http://biowulf.nih.gov/apps/bioinf-gpu.html http://biowulf.nih.gov/apps/bioinf-gpu.html From the summary : While there is a definite speedup from running on 1 cpu only to running on 1 cpu + 1 gpu, the multi-threaded versions generally have better overall performance. It is likely that once the CUDA versions are multi-threaded, running on the GPU nodes will confer additional advantages, but at present the codes are not yet mature.

GICodeWarrior 16y ago

This seems like an apples to oranges comparison with different scales and units...