6 ms·

So how do you program such a beast? What progress is being made on that front? Cache coherency seems really hard to give up on, and even CPU-GPU cache coheren

So how do you program such a beast? What progress is being made on that front? Cache coherency seems really hard to give up on, and even CPU-GPU cache coherency is becoming the expected norm, with even ARM delivering it.

ci5er 10y ago
These people have some grand claims in that regard: - http://texasmulticore.com/ I keep failing to get plans to match up to talk to their CTO on how this would stack up against what a rebuilt (differently optimized) Erlang VM could do... And, yes, I know: Not open source. No one needs to make that clever observation...
RachelF 10y ago
It's very hard. In the mid 1980s there was a CPU called a "Transputer" [1] made some of the people who moved to ARM. These CPUs could be connected together in huge networks and directly talk to each other. The network of CPU's could auto-discover its topology, but coding for so many CPU's was difficult. Some specific algorithms scaled well with the number of CPUs, but most did not. [1] https://en.wikipedia.org/wiki/Transputer https://en.wikipedia.org/wiki/Transputer
- hexane360 10y ago
  Wow, that reminds me of this[1] game. Lots of tiny processors messaging between each other. I didn't realize there were actual implementations. [1]http://store.steampowered.com/app/370360/ http://store.steampowered.com/app/370360/
- jacquesm 10y ago
  Occam did it quite nicely. I think the reason transputers didn't 'make it' is not because they were super hard to program (it was only a little bit harder than programming a regular computer), but because the price premium you paid for a transputer set-up was too high and x86 got faster very rapidly. This is right around the time when the first 386 machines were launched and in a very short time we went from 12-20 MHz 286 boxes (and some 68K machines for the lucky ones) to 33 MHz 386 machines with a ton of RAM. So the advantage that transputers had was eroded very quickly and I don't think INMOS was ready to match pace. Now that we've reached the end of the line for that kind of speed increase we are seeing a renewed interest in multi-cpu fabric architectures, of which the transputer was an instance.
  - bluedino 10y ago
    >> x86 got faster very rapidly. Is ARM ( or something else) going to surpass x86 (amd64 whatever you want to call it) in the near future?
    - digi_owl 10y ago
      Just a layman, but i get the impression as of late that things are running into thermal and feature size issues. And that these will affect all ISA equally.
  - stcredzero 10y ago
    I think the reason transputers didn't 'make it' is not because they were super hard to program You could replace "transputers" with [insert-architecture] and wind up with a nice overarching theory to explain the history of computer hardware.
  - gnufx 10y ago
    Yes, but in the areas I saw at the time, transputers lost out in particular to i860 for floating point systems and to m68k for things like our mostly-integer "streaming map-reduce" -- it seems! -- system. (Being able to house 68ks in a blade system, i.e. VME/VXI crates, helped.)
- TickleSteve 10y ago
  I was involved with a system porting it from an old transputer based system to a PowerPC&C/C++ based one. Transputers never really scaled into this sort of league, they were more 8/16/32 type scenarios.. For the interested, there was also the Atari Transputer Workstation (ATW). ...Although interesting, they were expensive and didn't perform compared to the equivalent sequential machine. Given development tho, who knows where it could have gone.
- atrn 10y ago
  transputers had four links and comms beyond that required routing in s/w. I wrote such a thing for a transputer machine. Also, the auto-discovery wasn't really auto... That was the boot code probing for CPUs on other ends of links and propagating itself to connected CPUs, building a map of the network in the process (which is kind of cool).
rwmj 10y ago
This is "the" question. I suppose one could look at the Connection Machine: https://en.wikipedia.org/wiki/Connection_Machine https://en.wikipedia.org/wiki/Connection_Machine The CM-2a had 4K or 8K processors and was reputedly very difficult to program. Looked great though: http://www.corestore.org/cm2a.htm http://www.corestore.org/cm2a.htm
- panic 10y ago
  The CM-200 looks even cooler (at least in the brochure…): http://www.corestore.org/cm200.htm http://www.corestore.org/cm200.htm
- sedachv 10y ago
  The Connection Machine was a SIMD design and the languages available for it (StarLisp: https://omohundro.files.wordpress.com/2009/03/omohundro86_the_essential_starlisp_manual.pdf https://omohundro.files.wordpress.com/2009/03/omohundro86_th... and C*: http://people.csail.mit.edu/bradley/cm5docs/CStarProgrammingGuide.pdf http://people.csail.mit.edu/bradley/cm5docs/CStarProgramming...) were actually pretty good compared to OpenCL. The Connection Machine Lisp programming language described in Daniel Hillis' PhD dissertation was essentially going to be Lisp with parallel map/reduce but AFAIK was never done being implemented. I think the big problem was that most of the SIMD algorithms were yet to be discovered at the time. For example this paper by Hillis and Steele was a very big deal but looks kind of basic today: http://uenics.evansville.edu/~mr56/ece757/DataParallelAlgorithms.pdf http://uenics.evansville.edu/~mr56/ece757/DataParallelAlgori... Guy Blelloch did a lot of work on Connection Machines and basically wrote the book on SIMD programming: http://www.cs.cmu.edu/~blelloch/papers/Ble90.pdf http://www.cs.cmu.edu/~blelloch/papers/Ble90.pdf He also made a very nice programming language for parallel computing (NESL), as did Gary Sabot who worked at Connection Machines (Paralations). When you compare those to Hadoop or OpenCL it really is a wonder where we went wrong and what the designers of the latter were thinking (or not).
deleted 10y ago
[deleted]
rbanffy 10y ago
I've read the first paragraphs, so take my comment with a grain of salt. There are 12 memory modules on the chip, so I assume each is shared by less than 100 cores, which is close to a beefy multithreaded server chip. Each core also has a small local program memory that's not shared between them. It'd be not unlike programming a networked cluster of processing nodes each with a separate chunk of memory, except that the cores could be assigned to neighboring workloads. Sounds fun!
pjc50 10y ago
Effectively you have to design for message-passing between microservices as a replacement for function calls. Not only do you have to step back from cache consistency but you have to step back from uniform memory access altogether and head towards a streaming architecture. It's very alien, but that's the only way to really use a system which has such a high ratio of processing power to external memory bandwidth/latency.
jacquesm 10y ago
You program a machine like this much like you work with the unix command line when you string together a series of commands using 'pipes'. Only now the pipes are the comms links between the CPUs and the programs are the pieces of software executing on each of the CPUs, rather than that they all run on the same CPU. So it's more of a data flow model than a function-call model, and every node will transform its input rather than that it will return results to callers. This means that not every problem is easily mapped to such an architecture, but when there is a good correspondence between the topology of the network of CPUs and the problem then the throughput can be very good. After all, there is no bottleneck where all of these have to access a shared resource (RAM) so when it pays off it pays off big.
- digi_owl 10y ago
  This layman finds himself reminded of the Cell CPU used in the Playstation 3.
qaq 10y ago
Erlang/Elixir anything running on the BEAM
gnufx 10y ago
High performance computing relies on message passing -- if not pure and simple -- normally with the MPI standard. There are applications which scale at least past petaflops and 1000 cores is unexceptional. The developers of the PETSc system wrote a position paper eschewing the typical hybrid (MPI+threading) techniques for trundling towards "exascale" -- if with MPI's shared memory support, which doesn't require cache coherence. (Apart from remote memory access, MPI departs from actor-/CSP-like systems in supporting things like collective communication (and i/o), and dynamic process management.)

ci5er 10y ago

These people have some grand claims in that regard: - http://texasmulticore.com/ I keep failing to get plans to match up to talk to their CTO on how this would stack up against what a rebuilt (differently optimized) Erlang VM could do... And, yes, I know: Not open source. No one needs to make that clever observation...

RachelF 10y ago

It's very hard. In the mid 1980s there was a CPU called a "Transputer" [1] made some of the people who moved to ARM. These CPUs could be connected together in huge networks and directly talk to each other. The network of CPU's could auto-discover its topology, but coding for so many CPU's was difficult. Some specific algorithms scaled well with the number of CPUs, but most did not. [1] https://en.wikipedia.org/wiki/Transputer https://en.wikipedia.org/wiki/Transputer

hexane360 10y ago
Wow, that reminds me of this[1] game. Lots of tiny processors messaging between each other. I didn't realize there were actual implementations. [1]http://store.steampowered.com/app/370360/ http://store.steampowered.com/app/370360/
jacquesm 10y ago
Occam did it quite nicely. I think the reason transputers didn't 'make it' is not because they were super hard to program (it was only a little bit harder than programming a regular computer), but because the price premium you paid for a transputer set-up was too high and x86 got faster very rapidly. This is right around the time when the first 386 machines were launched and in a very short time we went from 12-20 MHz 286 boxes (and some 68K machines for the lucky ones) to 33 MHz 386 machines with a ton of RAM. So the advantage that transputers had was eroded very quickly and I don't think INMOS was ready to match pace. Now that we've reached the end of the line for that kind of speed increase we are seeing a renewed interest in multi-cpu fabric architectures, of which the transputer was an instance.
- bluedino 10y ago
  >> x86 got faster very rapidly. Is ARM ( or something else) going to surpass x86 (amd64 whatever you want to call it) in the near future?
  - digi_owl 10y ago
    Just a layman, but i get the impression as of late that things are running into thermal and feature size issues. And that these will affect all ISA equally.
- stcredzero 10y ago
  I think the reason transputers didn't 'make it' is not because they were super hard to program You could replace "transputers" with [insert-architecture] and wind up with a nice overarching theory to explain the history of computer hardware.
- gnufx 10y ago
  Yes, but in the areas I saw at the time, transputers lost out in particular to i860 for floating point systems and to m68k for things like our mostly-integer "streaming map-reduce" -- it seems! -- system. (Being able to house 68ks in a blade system, i.e. VME/VXI crates, helped.)
TickleSteve 10y ago
I was involved with a system porting it from an old transputer based system to a PowerPC&C/C++ based one. Transputers never really scaled into this sort of league, they were more 8/16/32 type scenarios.. For the interested, there was also the Atari Transputer Workstation (ATW). ...Although interesting, they were expensive and didn't perform compared to the equivalent sequential machine. Given development tho, who knows where it could have gone.
atrn 10y ago
transputers had four links and comms beyond that required routing in s/w. I wrote such a thing for a transputer machine. Also, the auto-discovery wasn't really auto... That was the boot code probing for CPUs on other ends of links and propagating itself to connected CPUs, building a map of the network in the process (which is kind of cool).

rwmj 10y ago

This is "the" question. I suppose one could look at the Connection Machine: https://en.wikipedia.org/wiki/Connection_Machine https://en.wikipedia.org/wiki/Connection_Machine The CM-2a had 4K or 8K processors and was reputedly very difficult to program. Looked great though: http://www.corestore.org/cm2a.htm http://www.corestore.org/cm2a.htm

panic 10y ago
The CM-200 looks even cooler (at least in the brochure…): http://www.corestore.org/cm200.htm http://www.corestore.org/cm200.htm
sedachv 10y ago
The Connection Machine was a SIMD design and the languages available for it (StarLisp: https://omohundro.files.wordpress.com/2009/03/omohundro86_the_essential_starlisp_manual.pdf https://omohundro.files.wordpress.com/2009/03/omohundro86_th... and C*: http://people.csail.mit.edu/bradley/cm5docs/CStarProgrammingGuide.pdf http://people.csail.mit.edu/bradley/cm5docs/CStarProgramming...) were actually pretty good compared to OpenCL. The Connection Machine Lisp programming language described in Daniel Hillis' PhD dissertation was essentially going to be Lisp with parallel map/reduce but AFAIK was never done being implemented. I think the big problem was that most of the SIMD algorithms were yet to be discovered at the time. For example this paper by Hillis and Steele was a very big deal but looks kind of basic today: http://uenics.evansville.edu/~mr56/ece757/DataParallelAlgorithms.pdf http://uenics.evansville.edu/~mr56/ece757/DataParallelAlgori... Guy Blelloch did a lot of work on Connection Machines and basically wrote the book on SIMD programming: http://www.cs.cmu.edu/~blelloch/papers/Ble90.pdf http://www.cs.cmu.edu/~blelloch/papers/Ble90.pdf He also made a very nice programming language for parallel computing (NESL), as did Gary Sabot who worked at Connection Machines (Paralations). When you compare those to Hadoop or OpenCL it really is a wonder where we went wrong and what the designers of the latter were thinking (or not).

deleted 10y ago

[deleted]

rbanffy 10y ago

I've read the first paragraphs, so take my comment with a grain of salt. There are 12 memory modules on the chip, so I assume each is shared by less than 100 cores, which is close to a beefy multithreaded server chip. Each core also has a small local program memory that's not shared between them. It'd be not unlike programming a networked cluster of processing nodes each with a separate chunk of memory, except that the cores could be assigned to neighboring workloads. Sounds fun!

pjc50 10y ago

Effectively you have to design for message-passing between microservices as a replacement for function calls. Not only do you have to step back from cache consistency but you have to step back from uniform memory access altogether and head towards a streaming architecture. It's very alien, but that's the only way to really use a system which has such a high ratio of processing power to external memory bandwidth/latency.

jacquesm 10y ago

You program a machine like this much like you work with the unix command line when you string together a series of commands using 'pipes'. Only now the pipes are the comms links between the CPUs and the programs are the pieces of software executing on each of the CPUs, rather than that they all run on the same CPU. So it's more of a data flow model than a function-call model, and every node will transform its input rather than that it will return results to callers. This means that not every problem is easily mapped to such an architecture, but when there is a good correspondence between the topology of the network of CPUs and the problem then the throughput can be very good. After all, there is no bottleneck where all of these have to access a shared resource (RAM) so when it pays off it pays off big.

digi_owl 10y ago
This layman finds himself reminded of the Cell CPU used in the Playstation 3.

qaq 10y ago

Erlang/Elixir anything running on the BEAM

gnufx 10y ago

High performance computing relies on message passing -- if not pure and simple -- normally with the MPI standard. There are applications which scale at least past petaflops and 1000 cores is unexceptional. The developers of the PETSc system wrote a position paper eschewing the typical hybrid (MPI+threading) techniques for trundling towards "exascale" -- if with MPI's shared memory support, which doesn't require cache coherence. (Apart from remote memory access, MPI departs from actor-/CSP-like systems in supporting things like collective communication (and i/o), and dynamic process management.)