5 ms·

Having worked for a company that made a "hundreds of small CPUs on a single chip", I can tell you now that they're all going to fail because the programming mod

Having worked for a company that made a "hundreds of small CPUs on a single chip", I can tell you now that they're all going to fail because the programming model is too weird, and nobody will write software for them. Whatever comes next will be a GPU with extra capabilities, not a totally new architecture. Probably an nVidia GPU.

bryanlarsen 2y ago
While acknowledging that it's theoretically possible other approaches might succeed, it seems quite clear the author agrees with you.
convolvatron 2y ago
my take from reading this is more about programming abstractions than any particular hardware instantiation. the part of the Connection Machine that remains interesting is not building machines with CPUS with transistor counts in the hundreds running off a globally synchronous clock, but that there were a whole family of SIMD languages and let you do general purpose programming in parallel. And that those language were still relevant when the architecture changed to a MIMD machine with a bunch of vector units behind each CPU.
snovymgodym 2y ago
Reminds me of Itanium
- CyberDildonics 2y ago
  How is that at all like Itanium except for the superficial headline level where people say they are hard to program?
  - snovymgodym 2y ago
    Because the main feature that made Itanium hard to program for was its explicit instruction-level parallelism.
    - CyberDildonics 2y ago
      They weren't talking about instruction level parallelism.
      - snovymgodym 2y ago
        Similarities between two things don't require them to be identical.
        CyberDildonics 2y ago
        They aren't similar, they couldn't be more different. One is about lots of small threads of execution communicating with each other and synchronizing, one is about a few instructions being able to be run in parallel because implicitly within the CPU there are different pipelines. They aren't just different, they are at completely opposite ends of the programming spectrum. There are literally the two extremes of trying to make throughput faster.
turtletontine 2y ago
Could you elaborate on this? How does many-small-CPUs make for a weirder programming model than a GPU? Im no expert, but I’ve done my fair share of parallel HPC stuff using MPI, and a little bit of Cuda. And to me the GPU programming model is far far “weirder” and harder to code for than the many-CPUs model. (Granted, I’m assuming you’re describing a different regime?)
- dist-epoch 2y ago
  In CUDA you don't really manage the individual compute units, you start a kernel, and the drivers take care of distributing that to the compute cores and managing the data flows between them. When programming CPUs however you are controlling and managing the individual threads. Of course, there are libraries which can do that for you, but fundamentally it's a different model.
  - adrian_b 2y ago
    What you say has nothing to do with CPU vs. GPU, or with CUDA, which is basically equivalent with the older OpenMP. When you have a set of concurrent threads, each thread may run a different program. There are many applications where this is necessary, but such applications are hard to scale to very high levels of concurrency, because each thread must be handled individually by the programmer. Another case is when all the threads run the same program, but on different data. This is equivalent with a concurrent execution of a "for" loop, which is always possible when the iterations are independent. The execution of such a set of threads that execute the same program has been named "parallel DO instruction" by Melvin E. Conway in 1963, "array of processes" by C. A. R. Hoare in 1978, "replicated parallel" in the Occam programming language in 1985, SPMD around the same time, "PARALLEL DO" in the OpenMP Fortran language extension in 1997, "parallel for" in the OpenMP C/C++ language extension in 1998, and "kernel execution" in CUDA, which has also introduced the superfluous acronym SIMT to describe it. When a problem can be solved by a set of concurrent threads that run the same program, then it is much simpler to scale the parallelism to extremely high levels and the parallel execution can usually be scheduled by a compiler or by a hardware controller without the programmer having to be concerned with the details. There is no inherent difficulty in making a compiler that provides exactly the same programming model as CUDA, but which creates a program for a CPU, not for a GPU. Such compilers exist, e.g. ispc, which is mentioned in the parent article. The difference between GPUs and CPUs is that the former appear to have some extra hardware support for what you describe as "distributing that to the compute cores and managing the data flows between them", but nobody is able to tell exactly what is done by this extra hardware support and whether it really matters, because it is a part of the GPUs that has never been documented publicly by the GPU vendors. From the point of view of the programmer, this possible hardware advantage of the GPUs does not really matter, because there are plenty of programming language extensions for parallelism and libraries that can take care of the details of thread spawning and work distribution over SIMD lanes, regardless if the target is a CPU or a GPU. Whenever you write a program equivalent with a "parallel for", which is the same as writing for CUDA, you do not manage individual threads, because what you write, the "kernel" in CUDA lingo, can be executed by thousands of threads, also on a CPU, not only on a GPU. A desktop CPU like Ryzen 9 9950X has the same product of threads by SIMD lanes like a big integrated GPU (obviously, discrete GPUs can be many times bigger).
mikewarot 2y ago
The key transformation required to make any parallel architecture work is going to be taking a program that humans can understand, and translating it into a directed acyclic graph of logical Boolean operations. This type of intermediate representation could then be broken up into little chunks for all those small CPUS. It could be executed very slowly using just a few logic gates and enough ram to hold the state, or it could run at FPGA speeds or better on a generic sea of LUTs.
- KerrAvon 2y ago
  Isn't that the Connection Machine architecture?
  - Grosvenor 2y ago
    I would not complain about getting my own personal Connection Machine. So long as Tamiko Thiel does the design.
  - convolvatron 2y ago
    the CM architecture or programming model wasn't really a DAG. It was more like tensors of arbitrary rank with power of two sizes. Tensor operations themselves were serialized, but each of them ran in parallel. It was however much nicer than coding vectors today - it included Blelloch scans, generalizied scatter-gather, and systolic-esque nearest neighbor operations (shift this tensor in the positive direction along this axis). I would love to see a language like this that runs on modern GPUs, but its really not sufficiently general to get good performance there I think.
  - mikewarot 2y ago
    Most practical parallel computing hardware had queues to handle the mismatch in compute speed for various CPUs to run different algorithms on part of the data. Eliminating the CPU bound compute, and running everything truly in parallel eliminates the need for the queues and all the related hardware/software complexity. Imagine a sea of LUTs (look up tables), that are all clocked and only connected locally to their neighbors. The programming for this, even as virtual machine, allows for exploration of a virtually infinite design space of hardware with various tradeoffs for speed, size, cost, reliability, security, etc. The same graph could be refactored to run on anything in that design space.
deviantbit 2y ago
Yep, transputers failed miserably. I wrote a ton a code for them. Everything had to be solved in a serial bus, which defeated the purpose of the transputer.
- pdimitar 2y ago
  Quite fascinating. Did you write about your experiences in that area? Would love to read it!
  - deviantbit 2y ago
    Not in those terms, but an autobiography is coming, and bits and pieces are being explained. I expect about 10 people to buy the book, as all of the socialists will want it for free. I am negotiating with a publisher as we speak on the terms.
- __d 2y ago
  Could you elaborate on the “serial bus” bit?
audiofish 2y ago
Picochip?
petermcneeley 2y ago
Im guessing you just dont have the computational power to compete with a real GPU. It would be relatively easy for a top end graphics programmer to write the front end graphics API for your chip. Im guessing that if they did this you would just end up with a very poor performing GPU.

bryanlarsen 2y ago

While acknowledging that it's theoretically possible other approaches might succeed, it seems quite clear the author agrees with you.

convolvatron 2y ago

my take from reading this is more about programming abstractions than any particular hardware instantiation. the part of the Connection Machine that remains interesting is not building machines with CPUS with transistor counts in the hundreds running off a globally synchronous clock, but that there were a whole family of SIMD languages and let you do general purpose programming in parallel. And that those language were still relevant when the architecture changed to a MIMD machine with a bunch of vector units behind each CPU.

snovymgodym 2y ago

Reminds me of Itanium

CyberDildonics 2y ago
How is that at all like Itanium except for the superficial headline level where people say they are hard to program?
- snovymgodym 2y ago
  Because the main feature that made Itanium hard to program for was its explicit instruction-level parallelism.
  - CyberDildonics 2y ago
    They weren't talking about instruction level parallelism.
    - snovymgodym 2y ago
      Similarities between two things don't require them to be identical.
      - CyberDildonics 2y ago
        They aren't similar, they couldn't be more different. One is about lots of small threads of execution communicating with each other and synchronizing, one is about a few instructions being able to be run in parallel because implicitly within the CPU there are different pipelines. They aren't just different, they are at completely opposite ends of the programming spectrum. There are literally the two extremes of trying to make throughput faster.

turtletontine 2y ago

Could you elaborate on this? How does many-small-CPUs make for a weirder programming model than a GPU? Im no expert, but I’ve done my fair share of parallel HPC stuff using MPI, and a little bit of Cuda. And to me the GPU programming model is far far “weirder” and harder to code for than the many-CPUs model. (Granted, I’m assuming you’re describing a different regime?)

dist-epoch 2y ago
In CUDA you don't really manage the individual compute units, you start a kernel, and the drivers take care of distributing that to the compute cores and managing the data flows between them. When programming CPUs however you are controlling and managing the individual threads. Of course, there are libraries which can do that for you, but fundamentally it's a different model.
- adrian_b 2y ago
  What you say has nothing to do with CPU vs. GPU, or with CUDA, which is basically equivalent with the older OpenMP. When you have a set of concurrent threads, each thread may run a different program. There are many applications where this is necessary, but such applications are hard to scale to very high levels of concurrency, because each thread must be handled individually by the programmer. Another case is when all the threads run the same program, but on different data. This is equivalent with a concurrent execution of a "for" loop, which is always possible when the iterations are independent. The execution of such a set of threads that execute the same program has been named "parallel DO instruction" by Melvin E. Conway in 1963, "array of processes" by C. A. R. Hoare in 1978, "replicated parallel" in the Occam programming language in 1985, SPMD around the same time, "PARALLEL DO" in the OpenMP Fortran language extension in 1997, "parallel for" in the OpenMP C/C++ language extension in 1998, and "kernel execution" in CUDA, which has also introduced the superfluous acronym SIMT to describe it. When a problem can be solved by a set of concurrent threads that run the same program, then it is much simpler to scale the parallelism to extremely high levels and the parallel execution can usually be scheduled by a compiler or by a hardware controller without the programmer having to be concerned with the details. There is no inherent difficulty in making a compiler that provides exactly the same programming model as CUDA, but which creates a program for a CPU, not for a GPU. Such compilers exist, e.g. ispc, which is mentioned in the parent article. The difference between GPUs and CPUs is that the former appear to have some extra hardware support for what you describe as "distributing that to the compute cores and managing the data flows between them", but nobody is able to tell exactly what is done by this extra hardware support and whether it really matters, because it is a part of the GPUs that has never been documented publicly by the GPU vendors. From the point of view of the programmer, this possible hardware advantage of the GPUs does not really matter, because there are plenty of programming language extensions for parallelism and libraries that can take care of the details of thread spawning and work distribution over SIMD lanes, regardless if the target is a CPU or a GPU. Whenever you write a program equivalent with a "parallel for", which is the same as writing for CUDA, you do not manage individual threads, because what you write, the "kernel" in CUDA lingo, can be executed by thousands of threads, also on a CPU, not only on a GPU. A desktop CPU like Ryzen 9 9950X has the same product of threads by SIMD lanes like a big integrated GPU (obviously, discrete GPUs can be many times bigger).

mikewarot 2y ago

The key transformation required to make any parallel architecture work is going to be taking a program that humans can understand, and translating it into a directed acyclic graph of logical Boolean operations. This type of intermediate representation could then be broken up into little chunks for all those small CPUS. It could be executed very slowly using just a few logic gates and enough ram to hold the state, or it could run at FPGA speeds or better on a generic sea of LUTs.

KerrAvon 2y ago
Isn't that the Connection Machine architecture?
- Grosvenor 2y ago
  I would not complain about getting my own personal Connection Machine. So long as Tamiko Thiel does the design.
- convolvatron 2y ago
  the CM architecture or programming model wasn't really a DAG. It was more like tensors of arbitrary rank with power of two sizes. Tensor operations themselves were serialized, but each of them ran in parallel. It was however much nicer than coding vectors today - it included Blelloch scans, generalizied scatter-gather, and systolic-esque nearest neighbor operations (shift this tensor in the positive direction along this axis). I would love to see a language like this that runs on modern GPUs, but its really not sufficiently general to get good performance there I think.
- mikewarot 2y ago
  Most practical parallel computing hardware had queues to handle the mismatch in compute speed for various CPUs to run different algorithms on part of the data. Eliminating the CPU bound compute, and running everything truly in parallel eliminates the need for the queues and all the related hardware/software complexity. Imagine a sea of LUTs (look up tables), that are all clocked and only connected locally to their neighbors. The programming for this, even as virtual machine, allows for exploration of a virtually infinite design space of hardware with various tradeoffs for speed, size, cost, reliability, security, etc. The same graph could be refactored to run on anything in that design space.

deviantbit 2y ago

Yep, transputers failed miserably. I wrote a ton a code for them. Everything had to be solved in a serial bus, which defeated the purpose of the transputer.

pdimitar 2y ago
Quite fascinating. Did you write about your experiences in that area? Would love to read it!
- deviantbit 2y ago
  Not in those terms, but an autobiography is coming, and bits and pieces are being explained. I expect about 10 people to buy the book, as all of the socialists will want it for free. I am negotiating with a publisher as we speak on the terms.
__d 2y ago
Could you elaborate on the “serial bus” bit?

audiofish 2y ago

Picochip?

petermcneeley 2y ago

Im guessing you just dont have the computational power to compete with a real GPU. It would be relatively easy for a top end graphics programmer to write the front end graphics API for your chip. Im guessing that if they did this you would just end up with a very poor performing GPU.