5 ms·

What practical problems can 4000 CPUs solve that 16 CPUs can't? You get less and less been benefit for each additional CPU you add to the problem, unless the C

What practical problems can 4000 CPUs solve that 16 CPUs can't? You get less and less been benefit for each additional CPU you add to the problem, unless the CPU is the main bottleneck. Also if something requires 4000 CPUs, it is going to start getting expensive if you need to double the output. These types of problems don't scale well.

pradn 8y ago
One obvious answer is parallel builds. It's an immense waste of developer productivity to force builds on developer machines.
- vbezhenar 8y ago
  Not just builds but running tests as well. Actually I think I've heard that Google does exactly that (and caches compilation results, so they could just return them if they are ready).
  - smueller1234 8y ago
    Google's build and test execution infrastructure is both huge in size and absolutely amazing. One of the biggest positive surprises for me when I had my technical onboarding period.
    - adrianhel 8y ago
      Feel free to tell us more about it! :)
      - mochomocha 8y ago
        Recent article: https://queue.acm.org/detail.cfm?id=3287302 https://queue.acm.org/detail.cfm?id=3287302
        smueller1234 8y ago
        Thank you, I hadn't realized that was out there!
- danbolt 8y ago
  In a lot of C++ game development this is a huge factor for productivity, especially since a lot of translation units may need to be touched when changing APIs. Being able to utilize an entire idle office for each .cpp file can make a big difference in build times.
  - mikepurvis 8y ago
    Most places doing any non-trivial C++ development will have a centralized distcc and ccache cluster, surely?
    - danbolt 8y ago
      Most C++ shops I've worked at use MSVC and Incredibuild, so yes! I think MSVC is the norm in game development, especially for PC games, but I could be wrong on that.
- chrisseaton 8y ago
  I think the bottleneck there is actually memory. When building LLVM I run out of system memory before I run out of cores.
- Macha 8y ago
  At sufficient speed too. A build on our build farm takes 2 hours, my local mac does it in 50 minutes.
  - creatornator 8y ago
    Am I reading this right that a build farm is slower than your local development computer? Shouldn't that launch a re-evaluation of the build farm?
astrodust 8y ago
Rendering.
EliRivers 8y ago
We've spent decades developing a huge software/hardware edifice that we all stand atop of, thinking in terms of a single thread. The majority of programmers still have to actively push themselves to think in more than one thread (and why would they - the majority of problems programmers come across are single-threaded). I don't know if there a whole other edifice of computing out there, built atop of decades of thinking in terms of multiple threads, but I have sympathy to the idea that if it's out there, we'd have an awful lot of trouble conceptualising it, and an awful awful lot of trouble conceptualising it after decades of development. I don't know what kind of practical problems 4000 CPUs will solve that 16CPUs can't, but I give weight to the argument that the way we think, the problems we've created for ourselves and subsequently solved, could have blinded us to them.
- rbanffy 8y ago
  There is much that can be done that isn't easily done because of the way software and hardware evolved together. We think single threaded because our languages express single threads and because our languages express single threads the processors we use have to do outrageous things to reorder instructions trying to extract some meager parallelism across a couple execution units and do all this behind our backs. And, because they do that well enough, we don't bother inventing many new languages for doing that explicitly. You can't easily express "do these two independent things as you can and, when finished, do this other thing" in C (or Python, or Java) and it's up to brave compiler writers to figure out (sometimes erroneously) what can be done with the independent execution flows.
- JadeNB 8y ago
  > the majority of problems programmers come across are single-threaded Surely it's solutions (or rather programs), not problems, that are single-threaded? A problem can probably be solved in many ways, and the fact that many programmers will first reach for a single-threaded program to solve it doesn't mean that's the only, or even the best, way to solve it.
chaosbutters 8y ago
computational fluid dynamics, FEA, physics based simulations and the like
raverbashing 8y ago
> What practical problems can 4000 CPUs solve that 16 CPUs can't? I am going to be the funny person and say that 4k CPUs can solve a scheduling problem on time so that jobs for 10Mi Idle CPUs can be assigned on time. But yeah, there are problems where ~ 200x CPU power can make a lot of difference, especially if you're time bound (that's roughly solving in 2 days what 16 CPUs would solve in 1 year)
dragontamer 8y ago
> You get less and less been benefit for each additional CPU you add to the problem, unless the CPU is the main bottleneck. I disagree. Each CPU you add to a problem comes with 4x to 8x memory channels of DDR4 (2x on consumer systems, 4x on Threadripper, 6x on Skylake-X). So each CPU increases your memory bandwidth in a very predictable manner. > Also if something requires 4000 CPUs, it is going to start getting expensive if you need to double the output. These types of problems don't scale well. Finishing the problem in 1/4000th of the time is often good enough reason. That turns a problem that takes 10-years to finish into a problem that takes 1-day to finish. You only get good scaling when all the data fits in memory and the problem scales well, but that happens often enough that its worth studying these cases.

pradn 8y ago

One obvious answer is parallel builds. It's an immense waste of developer productivity to force builds on developer machines.

vbezhenar 8y ago
Not just builds but running tests as well. Actually I think I've heard that Google does exactly that (and caches compilation results, so they could just return them if they are ready).
- smueller1234 8y ago
  Google's build and test execution infrastructure is both huge in size and absolutely amazing. One of the biggest positive surprises for me when I had my technical onboarding period.
  - adrianhel 8y ago
    Feel free to tell us more about it! :)
    - mochomocha 8y ago
      Recent article: https://queue.acm.org/detail.cfm?id=3287302 https://queue.acm.org/detail.cfm?id=3287302
      - smueller1234 8y ago
        Thank you, I hadn't realized that was out there!
danbolt 8y ago
In a lot of C++ game development this is a huge factor for productivity, especially since a lot of translation units may need to be touched when changing APIs. Being able to utilize an entire idle office for each .cpp file can make a big difference in build times.
- mikepurvis 8y ago
  Most places doing any non-trivial C++ development will have a centralized distcc and ccache cluster, surely?
  - danbolt 8y ago
    Most C++ shops I've worked at use MSVC and Incredibuild, so yes! I think MSVC is the norm in game development, especially for PC games, but I could be wrong on that.
chrisseaton 8y ago
I think the bottleneck there is actually memory. When building LLVM I run out of system memory before I run out of cores.
Macha 8y ago
At sufficient speed too. A build on our build farm takes 2 hours, my local mac does it in 50 minutes.
- creatornator 8y ago
  Am I reading this right that a build farm is slower than your local development computer? Shouldn't that launch a re-evaluation of the build farm?

astrodust 8y ago

Rendering.

EliRivers 8y ago

We've spent decades developing a huge software/hardware edifice that we all stand atop of, thinking in terms of a single thread. The majority of programmers still have to actively push themselves to think in more than one thread (and why would they - the majority of problems programmers come across are single-threaded). I don't know if there a whole other edifice of computing out there, built atop of decades of thinking in terms of multiple threads, but I have sympathy to the idea that if it's out there, we'd have an awful lot of trouble conceptualising it, and an awful awful lot of trouble conceptualising it after decades of development. I don't know what kind of practical problems 4000 CPUs will solve that 16CPUs can't, but I give weight to the argument that the way we think, the problems we've created for ourselves and subsequently solved, could have blinded us to them.

rbanffy 8y ago
There is much that can be done that isn't easily done because of the way software and hardware evolved together. We think single threaded because our languages express single threads and because our languages express single threads the processors we use have to do outrageous things to reorder instructions trying to extract some meager parallelism across a couple execution units and do all this behind our backs. And, because they do that well enough, we don't bother inventing many new languages for doing that explicitly. You can't easily express "do these two independent things as you can and, when finished, do this other thing" in C (or Python, or Java) and it's up to brave compiler writers to figure out (sometimes erroneously) what can be done with the independent execution flows.
JadeNB 8y ago
> the majority of problems programmers come across are single-threaded Surely it's solutions (or rather programs), not problems, that are single-threaded? A problem can probably be solved in many ways, and the fact that many programmers will first reach for a single-threaded program to solve it doesn't mean that's the only, or even the best, way to solve it.

chaosbutters 8y ago

computational fluid dynamics, FEA, physics based simulations and the like

raverbashing 8y ago

> What practical problems can 4000 CPUs solve that 16 CPUs can't? I am going to be the funny person and say that 4k CPUs can solve a scheduling problem on time so that jobs for 10Mi Idle CPUs can be assigned on time. But yeah, there are problems where ~ 200x CPU power can make a lot of difference, especially if you're time bound (that's roughly solving in 2 days what 16 CPUs would solve in 1 year)

dragontamer 8y ago

> You get less and less been benefit for each additional CPU you add to the problem, unless the CPU is the main bottleneck. I disagree. Each CPU you add to a problem comes with 4x to 8x memory channels of DDR4 (2x on consumer systems, 4x on Threadripper, 6x on Skylake-X). So each CPU increases your memory bandwidth in a very predictable manner. > Also if something requires 4000 CPUs, it is going to start getting expensive if you need to double the output. These types of problems don't scale well. Finishing the problem in 1/4000th of the time is often good enough reason. That turns a problem that takes 10-years to finish into a problem that takes 1-day to finish. You only get good scaling when all the data fits in memory and the problem scales well, but that happens often enough that its worth studying these cases.