4 ms·

I predicted that because some computations are hard or impossible to parallelize, attention would shift to solving real-world problems with computations that ca

I predicted that because some computations are hard or impossible to parallelize, attention would shift to solving real-world problems with computations that can be parallelized. Maybe this is one factor in ML, deep learning neural nets vs. symbolic? However, engineers tend not to develop techniques without actual hardware [1]. And CPU core-count has remained much lower than I expected [2]. Therefore, my prediction has not come to pass, yet. But it totally will! 1. In myself, I can imagine a maybe-possible situation one or two steps away from present reality, but not see the consequences... but the instant I learn it is definitely possible, I can see consequences. 2. Yes, GPU core-count is sky-high, but is limited by memory access.

verdverm 6y ago
"Gordon Freeman" (as he is affectionately referred to) would argue otherwise, fervently! - https://www.cs.binghamton.edu/~pmadden/pubs/dispelling-ieeedt-2013.pdf https://www.cs.binghamton.edu/~pmadden/pubs/dispelling-ieeed... - https://dl.acm.org/doi/abs/10.1145/1324499.1324502 https://dl.acm.org/doi/abs/10.1145/1324499.1324502 - https://community.cadence.com/cadence_blogs_8/b/ii/posts/eda-workshop-debate-erupts-over-parallel-programming?CommentId=504ea082-91ee-455b-aede-fdad45e9a76d&pifragment-1576=310 https://community.cadence.com/cadence_blogs_8/b/ii/posts/eda... - https://www.cs.binghamton.edu/~pmadden/ https://www.cs.binghamton.edu/~pmadden/
- hyperpallium2 6y ago
  I'm saying that techniques that are easy to parallelize will become more popular. Of course he's right, that if you can make a serial technique faster, you should do that. I see a way to support his position: in past decades we got free speedups from silicon - then we hit a kind of "Peak Silicon". Just as for Peak Oil, it makes sense to try techniques that weren't worth it before. Personally, I think the first step should be to give up all the horrific layers of bloat-on-bloat; but instead we got (e.g.) faster JVMs and JIT JS compilation. But what about the step after that? He thinks we should mine the tailings (and he's surely right there's a lot to be found there). I think multi-core seems the only extendable solution (til I guess computers are the size of houses again). But because it's so difficult to parallelize code, we'll instead change our approach, and favour techniques that are easy to parallelize. Arguably, this is currently happening with DL. I've only skimmed the links you gave, so I don't have a deep appreciation of his position.
  - thyrsus 6y ago
    Many data centers are larger than houses. I expect them to get larger as cores get cheaper, and code gets smarter about NUMA.
  - carlmr 6y ago
    >Personally, I think the first step should be to give up all the horrific layers of bloat-on-bloat; but instead we got (e.g.) faster JVMs and JIT JS compilation. I think this is already happening. The trending languages I see in HN are Rust, nim, zig right now. We're going back to native, and in Rust's case also with fearless concurrency as a paradigm.
    - yellowapple 6y ago
      Right, but how does that compare to the incumbents? The JVM and .NET CLR don't seem to be going anywhere. In this day and age I don't think the abstraction layer is the issue. Modern bytecode VMs are already able to be within an order of magnitude of native code, and I'd imagine there's still plenty of room there for improvement (WASM comes to mind). Unless you're on constrained hardware, or developing directly against hardware (rather than using an operating system, which is itself a horrific layer of bloat-on-bloat), a bytecode VM is still perfectly reasonable even for soft-real-time applications (let alone for applications that are throughput-sensitive rather than latency-sensitive).
      - carlmr 6y ago
        >Modern bytecode VMs are already able to be within an order of magnitude of native code Looking at the slowing of hardware advances this could be a 10 year lag in performance. I think that's quite significant. I think byte code VMs have their place, and for many applications that may be enough, but if you have high performance needs it may be worth checking if the price is one you're willing to pay.
        yellowapple 6y ago
        The significance of a "10 year lag in performance" has wildly different meanings between the present day and, say, 10 years ago. 2010 (serial) performance on 2020 hardware would be hardly perceptible. 2000 performance on 2010 would be much more noticeable. 1990 performance on 2000 hardware would be catastrophic. And yet, Java, C#, Erlang, Perl, Python, Ruby, Tcl... all of these languages and their VMs are decades old now. And yeah, the languages toward the end of that list ain't exactly speed demons, but the ones toward the front are already commonly used for high-performance applications with pretty dang good success, and even the ones toward the back of the pack are there for other reasons beyond just the VM (global interpreter locks, parsing overhead, things like that). The VM, that is to say, ain't the issue. Hell, things like SPIR-V demonstrate that VMs are perfectly fine even for GPU computation. There's a price, sure, but eliminating that price is about as premature of an optimization at it gets for all but the most constrained of environments. There are almost certainly worse bottlenecks - and those bottlenecks are probably around I/O and memory, knowing most applications (and a bytecode VM often helps here, since the opcodes can be tuned for minimum size, often giving even RISC CPU opcodes a run for their money).
jsolson 6y ago
... how high did you expect CPU core count to be at this point with close to uniform memory access costs, and what changes did you expect to enable that degree of concurrent bus access without catastrophic stalling due to contention?
- hyperpallium2 6y ago
  I was thinking of GPU cores having random write access, but yes that has concurrent access problems. BTW how do GPUs solve the concurrent bus access you mention, with many cores are writing pixels? Also, is this GPU-style processing, where each core writes a separate result (a texture/display pixel or transformed vector) but can read randomly (from textures), the only approach to parallelization that works? It's a form of scatter-gather.
  - monocasa 6y ago
    > BTW how do GPUs solve the concurrent bus access you mention, with many cores are writing pixels? Batching and buffering with spatial locality so that you can stream out relatively large bursts of pixels.

verdverm 6y ago

"Gordon Freeman" (as he is affectionately referred to) would argue otherwise, fervently! - https://www.cs.binghamton.edu/~pmadden/pubs/dispelling-ieeedt-2013.pdf https://www.cs.binghamton.edu/~pmadden/pubs/dispelling-ieeed... - https://dl.acm.org/doi/abs/10.1145/1324499.1324502 https://dl.acm.org/doi/abs/10.1145/1324499.1324502 - https://community.cadence.com/cadence_blogs_8/b/ii/posts/eda-workshop-debate-erupts-over-parallel-programming?CommentId=504ea082-91ee-455b-aede-fdad45e9a76d&pifragment-1576=310 https://community.cadence.com/cadence_blogs_8/b/ii/posts/eda... - https://www.cs.binghamton.edu/~pmadden/ https://www.cs.binghamton.edu/~pmadden/

hyperpallium2 6y ago
I'm saying that techniques that are easy to parallelize will become more popular. Of course he's right, that if you can make a serial technique faster, you should do that. I see a way to support his position: in past decades we got free speedups from silicon - then we hit a kind of "Peak Silicon". Just as for Peak Oil, it makes sense to try techniques that weren't worth it before. Personally, I think the first step should be to give up all the horrific layers of bloat-on-bloat; but instead we got (e.g.) faster JVMs and JIT JS compilation. But what about the step after that? He thinks we should mine the tailings (and he's surely right there's a lot to be found there). I think multi-core seems the only extendable solution (til I guess computers are the size of houses again). But because it's so difficult to parallelize code, we'll instead change our approach, and favour techniques that are easy to parallelize. Arguably, this is currently happening with DL. I've only skimmed the links you gave, so I don't have a deep appreciation of his position.
- thyrsus 6y ago
  Many data centers are larger than houses. I expect them to get larger as cores get cheaper, and code gets smarter about NUMA.
- carlmr 6y ago
  >Personally, I think the first step should be to give up all the horrific layers of bloat-on-bloat; but instead we got (e.g.) faster JVMs and JIT JS compilation. I think this is already happening. The trending languages I see in HN are Rust, nim, zig right now. We're going back to native, and in Rust's case also with fearless concurrency as a paradigm.
  - yellowapple 6y ago
    Right, but how does that compare to the incumbents? The JVM and .NET CLR don't seem to be going anywhere. In this day and age I don't think the abstraction layer is the issue. Modern bytecode VMs are already able to be within an order of magnitude of native code, and I'd imagine there's still plenty of room there for improvement (WASM comes to mind). Unless you're on constrained hardware, or developing directly against hardware (rather than using an operating system, which is itself a horrific layer of bloat-on-bloat), a bytecode VM is still perfectly reasonable even for soft-real-time applications (let alone for applications that are throughput-sensitive rather than latency-sensitive).
    - carlmr 6y ago
      >Modern bytecode VMs are already able to be within an order of magnitude of native code Looking at the slowing of hardware advances this could be a 10 year lag in performance. I think that's quite significant. I think byte code VMs have their place, and for many applications that may be enough, but if you have high performance needs it may be worth checking if the price is one you're willing to pay.
      - yellowapple 6y ago
        The significance of a "10 year lag in performance" has wildly different meanings between the present day and, say, 10 years ago. 2010 (serial) performance on 2020 hardware would be hardly perceptible. 2000 performance on 2010 would be much more noticeable. 1990 performance on 2000 hardware would be catastrophic. And yet, Java, C#, Erlang, Perl, Python, Ruby, Tcl... all of these languages and their VMs are decades old now. And yeah, the languages toward the end of that list ain't exactly speed demons, but the ones toward the front are already commonly used for high-performance applications with pretty dang good success, and even the ones toward the back of the pack are there for other reasons beyond just the VM (global interpreter locks, parsing overhead, things like that). The VM, that is to say, ain't the issue. Hell, things like SPIR-V demonstrate that VMs are perfectly fine even for GPU computation. There's a price, sure, but eliminating that price is about as premature of an optimization at it gets for all but the most constrained of environments. There are almost certainly worse bottlenecks - and those bottlenecks are probably around I/O and memory, knowing most applications (and a bytecode VM often helps here, since the opcodes can be tuned for minimum size, often giving even RISC CPU opcodes a run for their money).

jsolson 6y ago

... how high did you expect CPU core count to be at this point with close to uniform memory access costs, and what changes did you expect to enable that degree of concurrent bus access without catastrophic stalling due to contention?

hyperpallium2 6y ago
I was thinking of GPU cores having random write access, but yes that has concurrent access problems. BTW how do GPUs solve the concurrent bus access you mention, with many cores are writing pixels? Also, is this GPU-style processing, where each core writes a separate result (a texture/display pixel or transformed vector) but can read randomly (from textures), the only approach to parallelization that works? It's a form of scatter-gather.
- monocasa 6y ago
  > BTW how do GPUs solve the concurrent bus access you mention, with many cores are writing pixels? Batching and buffering with spatial locality so that you can stream out relatively large bursts of pixels.