6 ms·

I worked in Cray's compiler department for seven years. If we couldn't dramatically parallelize someone's code, we couldn't sell a vector supercomputer. Period.

I worked in Cray's compiler department for seven years. If we couldn't dramatically parallelize someone's code, we couldn't sell a vector supercomputer. Period. Automatic parallelization is very possible. The problem is tends to be less efficient. A decent developer can often do a better job than the compiler by performing manual code restructuring. The compiler cannot always determine which changes are safe without pragmas to guide it. With that said, our top compiler devs did some amazing work adding automatic parallelization to some awful code. We inevitably sold our supercomputers because we had application experts who would manually restructure the most mission-critical code to fit cache lines and fill the vectors. Most other problems would perform quite adequately with the automatically-generated code. What this article lacks is a description of why Erlang is more uniquely suited to writing parallel code than all the other natively parallel languages like Unified Parallel C, Fortran2008, Chapel, Golang, etc. There are so many choices and many have been around for a long, long time.

larsberg 14y ago
I completely agree. As someone who works on a parallel functional language, it's very hard to sell a parallel language that isn't as fast as parallel fortran or hand-tuned C code that uses pthreads and the fastest parallel implementation of BLAS and other libraries. The people who really care about performance are using those. The ones who don't are honestly mostly still writing code that has large constant factors of _sequential_ performance available as low-hanging fruit. Sure, they'd take free performance, but the rewrite/porting/debug costs (even in automatic parallel compilers for the same language) are at least as high as just firing up a profiler. I'm increasingly of the opinion that if you can't win a top spot in the supercomputer performance fight, you have to have a unique application domain. Erlang's seems to be reliability. I suspect that a parallel variant of javascript that runs in every browser will end up being the next compelling parallel language, as opposed to all of us who are either inventing or attempting to resurrect languages that target x86 or GPUs.
- necrodome 14y ago
  Does/can Manticore have a such unique application domain?
  - larsberg 14y ago
    We believe so! Our project leader is focused on how we can parallelize general-purpose applications easily. With more and more people writing in static functional languages that have relatively poor parallel scalability without massive program transformations (Haskell, OCaml, F#, etc.), we think there is an opportunity there. That said, we're at a "go big or go home point." We either need to ramp up from the 1.5 grad students + 2 undergrads / year significantly or wrap things up. Getting to a point where we can be used in general-purpose projects requires a lot of work, none of which results in papers. If I had to guess, the most probable impact is what you would expect from PL research - integration of lessons learned in other systems down the road: - CilkPlus has a nice first pass at a work stealing algorithm, but we showed how to do it without static tuning by the programmer with lower overheads, to boot. - Vectorization to take advantage of wider vector hardware requires transformation of data structures (e.g., array of structs to struct of arrays). We showed how to do that automatically and reason about changes in program performance. - We have boatloads of papers - at both workshops and conferences - on what has to be done to the compiler and runtime to run efficiently on NUMA multicore systems. Right now, most fp systems do not run into these problems because they cannot scale past their own parallel overheads. Once past that bottleneck, the next one will be memory traffic, at least in our experience. I don't say any of that to fling mud at other systems; we started our project after them all, at the start of the multicore era (2006), with the goal of investigating these specific issues without carrying along the baggage of a pre-existing sequential implementation. There's also still a lot more to learn. I personally don't buy that that deterministic and total chaos are the two only points in the design space of program reasoning. There have to be some interesting midpoints (e.g., histories that are linearizable) that are worth investigating.
rdtsc 14y ago
As the complexity increases one thing that starts to show is not speedup in palatalization but fault tolerance. Debugging a non-concurrent program can be difficult, now throw in threads, shared memory, pointers and it quickly becomes a nightmare. The system could be fast, but if it crashes every week, is it useful. Often the answer is yes. But in some cases the answer it no. There is no free lunch. Shared-nothing architecture doesn't come for free. You pay a toll in _sequential_ performance. It might or not matter to you. > Automatic parallelization is very possible. For numerical algorithms many and for small function scope scale I can see that. Numerical code. But the problem is (and what Joe was pointing out) is that applications and algorithm design has to be build concurrently to start with. A compile will not re-factor your code to not access a single database and acquire a lock from 100k clients into using some eventually consistent or event-sourcing data store. It is something that has to be built from ground up. Same thing with fault tolerance, it has to built in from ground up. Adding it later is not easy.
seanmcdirmid 14y ago
Erlang is not designed for parallel programming; it is designed for concurrent programming. These are two very different programming domains with different problems. Every time someone conflates parallelism with concurrency...everyone gets very confused.
- unoti 14y ago
  Isn't it really fair to say that it's designed for both? The way it uses immutable state and something-similar-to-s-expressions to express data make it very straightforward (or even transparent) to distribute work between multiple processes and separate computers, in addition to how it makes it practical and simple to break work into small chunks that can be interleaved easily within the same thread. It's really designed for doing both very well, wouldn't you say?
  - seanmcdirmid 14y ago
    Not at all. Erlang isn't useful for modern parallel computing as we know it, which is usually done as some kind of SIMD program; say MapReduce or GPGPU using something like CUDA. The benefit doesn't just come from operating on data all at once, but these systems (or the programmer) also do a lot of work to optimize the I/O and cache characteristics of the computation. Actor architectures are only useful for task parallelism which no one really knows how to get much out of; definitely not the close-to-linear performance benefits we can get from data parallelism. Task parallelism is much better for when you have to do multiple things at once (more efficient concurrency), not for when you want to make a sequential task faster. Maybe this will help http://jlouisramblings.blogspot.com/2011/07/erlangs-parallelism-is-not-parallelism.html http://jlouisramblings.blogspot.com/2011/07/erlangs-parallel... and https://news.ycombinator.com/item?id=2726661 https://news.ycombinator.com/item?id=2726661
    - peerst 14y ago
      "modern parallel computing" ... well not everything that can run parallel on multi core CPU's can run very well on a GPU. I'm using Erlang and GPU programming each for its area of expertise. FWIW I even use both together via https://github.com/tonyrog/cl https://github.com/tonyrog/cl Erlang is great at asynchronous concurrency which happens to be able to run in parallel well because of how the VM is built. GPU's solve totally different problems
SoftwareMaven 14y ago
If the code isn't efficient, you'll run into Ahmdal's Law much more quickly. In fact, I think your comment aligns with what Joe was saying: automated paralellization is not going to happen. You will have to go through and find all your contention points, just like your application experts did. I completely agree with your last sentence. For those of us who have dived in a way, the advantages become clear, but TFA was really just preaching to the choir.
reeses 14y ago
Fortran (especially ancient, wheezy Fortran) lent itself to supervised automatic parallelization because of its lack of dynamic arrays. It was "easy" to vectorize code at compile time when you had so much information about the runtime expectations. We can do some of this now in most languages with hot-spot profiling, basic block analysis, selective inlining, and other innovations. However, you really can't beat low-level languages that explicitly "hint" at their execution paths. By the same token, Cray's applications were...so...slow if you were foolish enough to run them on the expensive hardware and not the FEPs.

larsberg 14y ago

I completely agree. As someone who works on a parallel functional language, it's very hard to sell a parallel language that isn't as fast as parallel fortran or hand-tuned C code that uses pthreads and the fastest parallel implementation of BLAS and other libraries. The people who really care about performance are using those. The ones who don't are honestly mostly still writing code that has large constant factors of _sequential_ performance available as low-hanging fruit. Sure, they'd take free performance, but the rewrite/porting/debug costs (even in automatic parallel compilers for the same language) are at least as high as just firing up a profiler. I'm increasingly of the opinion that if you can't win a top spot in the supercomputer performance fight, you have to have a unique application domain. Erlang's seems to be reliability. I suspect that a parallel variant of javascript that runs in every browser will end up being the next compelling parallel language, as opposed to all of us who are either inventing or attempting to resurrect languages that target x86 or GPUs.

necrodome 14y ago
Does/can Manticore have a such unique application domain?
- larsberg 14y ago
  We believe so! Our project leader is focused on how we can parallelize general-purpose applications easily. With more and more people writing in static functional languages that have relatively poor parallel scalability without massive program transformations (Haskell, OCaml, F#, etc.), we think there is an opportunity there. That said, we're at a "go big or go home point." We either need to ramp up from the 1.5 grad students + 2 undergrads / year significantly or wrap things up. Getting to a point where we can be used in general-purpose projects requires a lot of work, none of which results in papers. If I had to guess, the most probable impact is what you would expect from PL research - integration of lessons learned in other systems down the road: - CilkPlus has a nice first pass at a work stealing algorithm, but we showed how to do it without static tuning by the programmer with lower overheads, to boot. - Vectorization to take advantage of wider vector hardware requires transformation of data structures (e.g., array of structs to struct of arrays). We showed how to do that automatically and reason about changes in program performance. - We have boatloads of papers - at both workshops and conferences - on what has to be done to the compiler and runtime to run efficiently on NUMA multicore systems. Right now, most fp systems do not run into these problems because they cannot scale past their own parallel overheads. Once past that bottleneck, the next one will be memory traffic, at least in our experience. I don't say any of that to fling mud at other systems; we started our project after them all, at the start of the multicore era (2006), with the goal of investigating these specific issues without carrying along the baggage of a pre-existing sequential implementation. There's also still a lot more to learn. I personally don't buy that that deterministic and total chaos are the two only points in the design space of program reasoning. There have to be some interesting midpoints (e.g., histories that are linearizable) that are worth investigating.

rdtsc 14y ago

As the complexity increases one thing that starts to show is not speedup in palatalization but fault tolerance. Debugging a non-concurrent program can be difficult, now throw in threads, shared memory, pointers and it quickly becomes a nightmare. The system could be fast, but if it crashes every week, is it useful. Often the answer is yes. But in some cases the answer it no. There is no free lunch. Shared-nothing architecture doesn't come for free. You pay a toll in _sequential_ performance. It might or not matter to you. > Automatic parallelization is very possible. For numerical algorithms many and for small function scope scale I can see that. Numerical code. But the problem is (and what Joe was pointing out) is that applications and algorithm design has to be build concurrently to start with. A compile will not re-factor your code to not access a single database and acquire a lock from 100k clients into using some eventually consistent or event-sourcing data store. It is something that has to be built from ground up. Same thing with fault tolerance, it has to built in from ground up. Adding it later is not easy.

seanmcdirmid 14y ago

Erlang is not designed for parallel programming; it is designed for concurrent programming. These are two very different programming domains with different problems. Every time someone conflates parallelism with concurrency...everyone gets very confused.

unoti 14y ago
Isn't it really fair to say that it's designed for both? The way it uses immutable state and something-similar-to-s-expressions to express data make it very straightforward (or even transparent) to distribute work between multiple processes and separate computers, in addition to how it makes it practical and simple to break work into small chunks that can be interleaved easily within the same thread. It's really designed for doing both very well, wouldn't you say?
- seanmcdirmid 14y ago
  Not at all. Erlang isn't useful for modern parallel computing as we know it, which is usually done as some kind of SIMD program; say MapReduce or GPGPU using something like CUDA. The benefit doesn't just come from operating on data all at once, but these systems (or the programmer) also do a lot of work to optimize the I/O and cache characteristics of the computation. Actor architectures are only useful for task parallelism which no one really knows how to get much out of; definitely not the close-to-linear performance benefits we can get from data parallelism. Task parallelism is much better for when you have to do multiple things at once (more efficient concurrency), not for when you want to make a sequential task faster. Maybe this will help http://jlouisramblings.blogspot.com/2011/07/erlangs-parallelism-is-not-parallelism.html http://jlouisramblings.blogspot.com/2011/07/erlangs-parallel... and https://news.ycombinator.com/item?id=2726661 https://news.ycombinator.com/item?id=2726661
  - peerst 14y ago
    "modern parallel computing" ... well not everything that can run parallel on multi core CPU's can run very well on a GPU. I'm using Erlang and GPU programming each for its area of expertise. FWIW I even use both together via https://github.com/tonyrog/cl https://github.com/tonyrog/cl Erlang is great at asynchronous concurrency which happens to be able to run in parallel well because of how the VM is built. GPU's solve totally different problems

SoftwareMaven 14y ago

If the code isn't efficient, you'll run into Ahmdal's Law much more quickly. In fact, I think your comment aligns with what Joe was saying: automated paralellization is not going to happen. You will have to go through and find all your contention points, just like your application experts did. I completely agree with your last sentence. For those of us who have dived in a way, the advantages become clear, but TFA was really just preaching to the choir.

reeses 14y ago

Fortran (especially ancient, wheezy Fortran) lent itself to supervised automatic parallelization because of its lack of dynamic arrays. It was "easy" to vectorize code at compile time when you had so much information about the runtime expectations. We can do some of this now in most languages with hot-spot profiling, basic block analysis, selective inlining, and other innovations. However, you really can't beat low-level languages that explicitly "hint" at their execution paths. By the same token, Cray's applications were...so...slow if you were foolish enough to run them on the expensive hardware and not the FEPs.