7 ms·

Now that Moore's Law has effectively ended on a single chip (ended around 2007), FP is front-and-center again, but mostly because of the emphasis on immutabilit

Now that Moore's Law has effectively ended on a single chip (ended around 2007), FP is front-and-center again, but mostly because of the emphasis on immutability, which is a prerequisite for parallelization. Moore's Law has been replaced with Amdahl's Law.

adwn 10y ago
> Now that Moore's Law has effectively ended on a single chip (ended around 2007), FP is front-and-center again Ah yes, the decades-old claim that FP will make parallel programming much simpler. I wonder when that notion will go the way of "Java will be faster than C because of the JIT compiler" or "Compilers become better and better, so dynamic scripting languages will be quite fast soon." There are two main problems with this sentiment: 1) Immutability does not automatically result in parallelity. If your algorithm is sequential, there's nothing a functional, immutable language can do to help. Oh, and a lot of algorithms are hard or impossible to parallelize efficiently, and even if you can parallelize it in theory, you still have work with limited latency and throughput to other processing elements. 2) Mutable arrays of compact, pointer-less data elements map incredibly well to transistor arrays (SRAM for caches) and transistor/capacitor arrays (DRAM for main memory). > the emphasis on immutability, which is a prerequisite for parallelization This is blatantly false. Basically all parallel algorithms in high-performance-computing make extensive use of mutable memory.
- nickbauman 10y ago
  What I should have said immutability allows for code that is both amenable to parallelization and be easier to reason about by the people that write it.
  - p1esk 10y ago
    So if Moore's law is dead since 2007, why do I still use C for my CUDA code, and not some FP language?
    - pjlegato 10y ago
      Inertia, mostly. All the common tools are built around C, all the existing code is written in C, all the jobs to maintain and extend that code are all in C. Like most "Why does everyone only ever use language X to do Y" questions, considerations of actual technical merit / best technical fit are a very minor factor in the equation.
      - p1esk 10y ago
        Sure, I understand the inertia, but can you show me any demo where a FP language outperforms C for parallel tasks? People in the deep learning field care a lot about speed, and if someone demonstrated a factor of x speedup with FP language, they would use it.
        oldmanjay 10y ago
        You want proof of a claim that only you have made. That doesn't seem like a terribly fair criticism of the OP.
        p1esk 10y ago
        Well, this discussion started when someone suggested the speed increases fueled by Moore's law held FP back. This is what I've been questioning. If it's not about speed, then what does it have to do with Moore's law?
        nickpsecurity 10y ago
        "but can you show me any demo where a FP language outperforms C for parallel tasks?" That's not a fair comparison. C proponents should ask, "Is there a demo where (alternative) outperforms C for (application domain) when C has (alternative)'s safety-checks enabled, too?" Memory and dataflow safety in C, using checks instead of exotic stuff, easily adds 20-500% overhead depending on application. Whereas, these safer or more functional languages are usually within 200% with some Common LISP's and Schemes matching or outperforming C in a few benchmarks. Concurrent Haskell wouldn't be giving me a performance reduction: I'd be getting more capabilities in development and reliability with a performance sacrifice. One that goes down every year for at least one implementation of each language or style.
- marcosdumay 10y ago
  Just a few weeks ago, there was a question on /r/haskell that was basically "Hey, I wrote this code, and it is taking all my cores. Why is it doing so if it is sequential?" And, of course, the answer was that, no, his code was not sequential, he just failed to understand how the compiler parallelized it. My point being, until you've tried what we have today, and understood how powerful it is, you don't get to make a good point about it. Anyway, yes, Amdahl's law is true. And no, it does not impose huge constrains on most practical problems, at least not for the next decade. And by the way, yes, specialized hardware can be a great speed-up. And it's been designed with FP-like constraints since forever, because if programming parallel general purpose computers is hard, specializing hardware for parallel operation is orders of magnitude worse. Our high level FP languages aren't up to the task, that's true. But our high level imperative ones can't even theoretically get there.
  - adwn 10y ago
    > a question on /r/haskell Can you post a link? > And, of course, the answer was that, no, his code was not sequential, he just failed to understand how the compiler parallelized it. Like I wrote before, it's not enough to find independent operations if scheduling and communication overhead eat up most of the gains. > And by the way, yes, specialized hardware can be a great speed-up. And it's been designed with FP-like constraints since forever What specialized hardware (for speed gains) has been programmed using functional languages "since forever"? Name two examples, please.
- pron 10y ago
  While I agree with your main argument, your examples are a bit unfortunate because Java is faster than (or certainly as fast as) C already for a similar development effort (with one caveat[1]) and dynamic languages are quite fast (see V8). Today, projects like Truffle/Graal can run pretty much any dynamic language within 3-5x slower than C. So I'd say that those are two predictions that have come to pass. They do, however, require greater computational resources (RAM and energy) which makes them less suitable in some constrained environments, so whether you want to call that mission accomplished or not is up to you. [1]: Today a simple Java program running on HotSpot compiles pretty much to the same machine instructions as a comparable C program, and a large Java program would likely be faster than a large C program give the same development effort. The major hindrance to Java performance these days is the simple lack of value types, which creates many unnecessary cache faults and has nothing to do with the JIT. And there's the GC which normally trades better throughput for worse latency, but, again -- nothing to do with the JIT. BTW, value types are the top feature being worked on by the HotSpot team.
  - adwn 10y ago
    > Java is faster than (or certainly as fast as) C already for a similar development effort Last I heard, this only applies if you write C in Java (i.e., no object composition through references, basic arrays instead of ArrayList, avoid GC wherever possible). I remember a project some years back where a git client (?) was to be written in Java, and even with these techniques they only got to within 2x C performance. > for a similar development effort Well, then that defeats the main reason for using Java, doesn't it? The gist of that claim was that you could write idiomatic Java (with GC, small objects, etc.) and the JIT would use the information gathered during runtime to optimize the program on-the-fly to be faster than C. > Today, projects like Truffle/Graal can run pretty much any dynamic language within 3-5x slower than C. Do you have a source for that? Not doubting you, geniunely curious.
    - pron 10y ago
      The JIT can and does do all that, and I am talking about idiomatic Java (I'm speaking mainly from the experience of migrating a >5MLOC soft realtime defense application from C++ to Java). Similar performance can be achieved for much less effort (although C would always beat Java given significantly greater effort), and this effect is more pronounced the larger and more concurrent the application is. The cost -- for large application -- is just significantly greater RAM consumption. For small/constrained applications there are other costs. As I said, the main barrier (which is currently being addressed) is memory layout of arrays-of-structs, whose fixing requires additions to the language semantics (which are being done). Perhaps that particular issue was a failed promise, but it's mostly a result of hardware changes that occurred 10 years after Java's introduction. As to Graal, this very recent talk has some nice demos: https://www.youtube.com/watch?v=N__8TBcaDTs https://www.youtube.com/watch?v=N__8TBcaDTs You can find more material here: https://wiki.openjdk.java.net/display/Graal/Publications+and+Presentations https://wiki.openjdk.java.net/display/Graal/Publications+and...

adwn 10y ago

> Now that Moore's Law has effectively ended on a single chip (ended around 2007), FP is front-and-center again Ah yes, the decades-old claim that FP will make parallel programming much simpler. I wonder when that notion will go the way of "Java will be faster than C because of the JIT compiler" or "Compilers become better and better, so dynamic scripting languages will be quite fast soon." There are two main problems with this sentiment: 1) Immutability does not automatically result in parallelity. If your algorithm is sequential, there's nothing a functional, immutable language can do to help. Oh, and a lot of algorithms are hard or impossible to parallelize efficiently, and even if you can parallelize it in theory, you still have work with limited latency and throughput to other processing elements. 2) Mutable arrays of compact, pointer-less data elements map incredibly well to transistor arrays (SRAM for caches) and transistor/capacitor arrays (DRAM for main memory). > the emphasis on immutability, which is a prerequisite for parallelization This is blatantly false. Basically all parallel algorithms in high-performance-computing make extensive use of mutable memory.

nickbauman 10y ago
What I should have said immutability allows for code that is both amenable to parallelization and be easier to reason about by the people that write it.
- p1esk 10y ago
  So if Moore's law is dead since 2007, why do I still use C for my CUDA code, and not some FP language?
  - pjlegato 10y ago
    Inertia, mostly. All the common tools are built around C, all the existing code is written in C, all the jobs to maintain and extend that code are all in C. Like most "Why does everyone only ever use language X to do Y" questions, considerations of actual technical merit / best technical fit are a very minor factor in the equation.
    - p1esk 10y ago
      Sure, I understand the inertia, but can you show me any demo where a FP language outperforms C for parallel tasks? People in the deep learning field care a lot about speed, and if someone demonstrated a factor of x speedup with FP language, they would use it.
      - oldmanjay 10y ago
        You want proof of a claim that only you have made. That doesn't seem like a terribly fair criticism of the OP.
        p1esk 10y ago
        Well, this discussion started when someone suggested the speed increases fueled by Moore's law held FP back. This is what I've been questioning. If it's not about speed, then what does it have to do with Moore's law?
      - nickpsecurity 10y ago
        "but can you show me any demo where a FP language outperforms C for parallel tasks?" That's not a fair comparison. C proponents should ask, "Is there a demo where (alternative) outperforms C for (application domain) when C has (alternative)'s safety-checks enabled, too?" Memory and dataflow safety in C, using checks instead of exotic stuff, easily adds 20-500% overhead depending on application. Whereas, these safer or more functional languages are usually within 200% with some Common LISP's and Schemes matching or outperforming C in a few benchmarks. Concurrent Haskell wouldn't be giving me a performance reduction: I'd be getting more capabilities in development and reliability with a performance sacrifice. One that goes down every year for at least one implementation of each language or style.
marcosdumay 10y ago
Just a few weeks ago, there was a question on /r/haskell that was basically "Hey, I wrote this code, and it is taking all my cores. Why is it doing so if it is sequential?" And, of course, the answer was that, no, his code was not sequential, he just failed to understand how the compiler parallelized it. My point being, until you've tried what we have today, and understood how powerful it is, you don't get to make a good point about it. Anyway, yes, Amdahl's law is true. And no, it does not impose huge constrains on most practical problems, at least not for the next decade. And by the way, yes, specialized hardware can be a great speed-up. And it's been designed with FP-like constraints since forever, because if programming parallel general purpose computers is hard, specializing hardware for parallel operation is orders of magnitude worse. Our high level FP languages aren't up to the task, that's true. But our high level imperative ones can't even theoretically get there.
- adwn 10y ago
  > a question on /r/haskell Can you post a link? > And, of course, the answer was that, no, his code was not sequential, he just failed to understand how the compiler parallelized it. Like I wrote before, it's not enough to find independent operations if scheduling and communication overhead eat up most of the gains. > And by the way, yes, specialized hardware can be a great speed-up. And it's been designed with FP-like constraints since forever What specialized hardware (for speed gains) has been programmed using functional languages "since forever"? Name two examples, please.
pron 10y ago
While I agree with your main argument, your examples are a bit unfortunate because Java is faster than (or certainly as fast as) C already for a similar development effort (with one caveat[1]) and dynamic languages are quite fast (see V8). Today, projects like Truffle/Graal can run pretty much any dynamic language within 3-5x slower than C. So I'd say that those are two predictions that have come to pass. They do, however, require greater computational resources (RAM and energy) which makes them less suitable in some constrained environments, so whether you want to call that mission accomplished or not is up to you. [1]: Today a simple Java program running on HotSpot compiles pretty much to the same machine instructions as a comparable C program, and a large Java program would likely be faster than a large C program give the same development effort. The major hindrance to Java performance these days is the simple lack of value types, which creates many unnecessary cache faults and has nothing to do with the JIT. And there's the GC which normally trades better throughput for worse latency, but, again -- nothing to do with the JIT. BTW, value types are the top feature being worked on by the HotSpot team.
- adwn 10y ago
  > Java is faster than (or certainly as fast as) C already for a similar development effort Last I heard, this only applies if you write C in Java (i.e., no object composition through references, basic arrays instead of ArrayList, avoid GC wherever possible). I remember a project some years back where a git client (?) was to be written in Java, and even with these techniques they only got to within 2x C performance. > for a similar development effort Well, then that defeats the main reason for using Java, doesn't it? The gist of that claim was that you could write idiomatic Java (with GC, small objects, etc.) and the JIT would use the information gathered during runtime to optimize the program on-the-fly to be faster than C. > Today, projects like Truffle/Graal can run pretty much any dynamic language within 3-5x slower than C. Do you have a source for that? Not doubting you, geniunely curious.
  - pron 10y ago
    The JIT can and does do all that, and I am talking about idiomatic Java (I'm speaking mainly from the experience of migrating a >5MLOC soft realtime defense application from C++ to Java). Similar performance can be achieved for much less effort (although C would always beat Java given significantly greater effort), and this effect is more pronounced the larger and more concurrent the application is. The cost -- for large application -- is just significantly greater RAM consumption. For small/constrained applications there are other costs. As I said, the main barrier (which is currently being addressed) is memory layout of arrays-of-structs, whose fixing requires additions to the language semantics (which are being done). Perhaps that particular issue was a failed promise, but it's mostly a result of hardware changes that occurred 10 years after Java's introduction. As to Graal, this very recent talk has some nice demos: https://www.youtube.com/watch?v=N__8TBcaDTs https://www.youtube.com/watch?v=N__8TBcaDTs You can find more material here: https://wiki.openjdk.java.net/display/Graal/Publications+and+Presentations https://wiki.openjdk.java.net/display/Graal/Publications+and...