12 ms·

Is parallel programming hard, and, if so, what can you do about it?

135 points by matt_d 7y ago

classified 7y ago
Direct jump to the PDF download page, avoiding the terrible hosting and Russian ads: https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html https://mirrors.edge.kernel.org/pub/linux/kernel/people/paul...
dang 7y ago
A thread from 2015: https://news.ycombinator.com/item?id=9315152 https://news.ycombinator.com/item?id=9315152 2014: https://news.ycombinator.com/item?id=7381877 https://news.ycombinator.com/item?id=7381877 2011: https://news.ycombinator.com/item?id=2784515 https://news.ycombinator.com/item?id=2784515
- matt_d 7y ago
  Thanks! FWIW, I've used the original post title ("Parallel Programming: December 2019 Update") to focus on the updates (and partially driven by the fear of comments made right after reading just the book title, "Is Parallel Programming Hard, And, If So, What Can You Do About It?", and focusing solely on answering the question while completely ignoring the content... :-]).
Koshkin 7y ago
Parallel programming is not hard per se, it's just that there's so much more to learn (compared to "sequential programming"): threads, processes, IPC, messaging, actors, queues, synchronization primitives (mutexes, semaphores, monitors, condition variables, critical sections, atomics... you name it) along with their gotchas, concurrent containers with their limitations, transactions, distributed computing with its performance implications, new design patterns (e.g. async/await, promises and futures). But hey, programming has never been thought of as easy, right? Yet, speaking of having it hard, it may be enlightening to compare software development with having to design a piece of (digital) hardware, where not only much has always been happening "in parallel" but one also must keep taking into account physical characteristics of various parts of the circuit.
- gopalv 7y ago
  > Parallel programming is not hard per se, it's just that there's so much more to learn Parallel programming isn't the hardest part (it is hard, but good teachers help with good abstractions over problems), it is the parallel debugging that is - or at least the second pass over the same codebase. Often being able to write decent parallel code is easier than debugging a system which is behaving inconsistently without any ability to set up the timing bugs the right way around. The difference between "I fixed a bug" and "I massively lowered the probability of it" is hard to tell. I spent 7 months chasing an issue that happened once on average of a billion requests (or rather once every 2-3 days across a web farm). The fix was easy, but debugging it drove me insane slowly over a whole summer. And after that, there's the loss of efficiency from parallelization which needs enough parallel tracks to actually be faster - more threads is often slower at low thread counts. As an anecdote, there were enough servers at Yahoo which had SMT turned off because the extra locking introduced by 2 threads on a core ate up the extra CPU those threads freed up.
  - agumonkey 7y ago
    This gives me two questions: - did you have a method/tool to debug (from book, or chatting with others) - are there still cpu designs dedicated to parallel workloads (I assume the usual desktop intel/amd, no matter how brilliant, may not be without hiccups in high parallelism) oh one last thing, do you use dedicated compilers for parallelization ? or is it something "mainstream" like openmp
    - namibj 7y ago
      I can greatly recommended mozilla's rr for hunting concurrency bugs, as it allows efficient replay and provides a scheduler that is designed to increase the probability of triggering concurrency bugs. The issue with x86 is simple: the memory model pretty much requires loads of broadcasts on the SMP interconnect. IIRC Power9 is nice due to (1) a memory model that doesn't require this (much) broadcasting on the interconnect, and (2) remote atomics like "fetch current 64bit value pointed to by register1 and atomically add register2 after", which prevent the cacheline from bouncing between cores or even sockets (Power9 scales to about 5~10kW TDP @8~20W TDP/core and 4 threads/core (there are versions that combine core pairs to have 8 threads using the resources of (almost) 2 cores (overhead / diminishing returns))). I can't really recommend OpenMP. If the architectural restrictions aren't a problem, you might want to check out Intel's TBB, which uses C++ templates instead of compiler pragmas. Unfortunately you'd ideally need multithreading-aware polyhedral optimization, and _that_ mostly stalled AFAIK. You'd likely write C++ or now maybe Rust with low-level primitives / intrinsics to build libraries. I don't know of any actually good "automatic" frameworks for general shared-memory parallel programming (There's Chapel, but it seems fairly niche. Still going strong though, by the looks of it.). If you can efficiently express the problem with loop-dataflow, you might want to try timely-dataflow (Rust, does thread/process/node parallelism), as it nicely integrates with normal Rust code.
zwaps 7y ago
Parallel programming in Python is hard. On Windows, it's hell. In Matlab it's a breeze. Just put "par" in front of the thing. Done. If you wanna shell out money to enable that functionality, that is.
- honestoHeminway 7y ago
  Software is able to reason about software? Do you notice something in that sentence? If it would work - it would make the programers useless. But it doesent. Once a pointer points at something in your parallel task, software is unable to reason about it, erring on the side of caution and thus, parallelism wont happen there. Also these parallelisms, usually work for trivial parallel structures like for loops. Easy to unroll into multiple cores, but also with little gain, as communication times eats the gained advantages.
jhfdbkofdcho 7y ago
Someone wiser than me observed, “the thing about parallel programming is that it always ends in tears.”
carapace 7y ago
It's hard or easy depending on your model (of parallelism) and what you want to do with it. From a certain POV computers are always doing things in parallel (8-, 16-, 32-, 64-bits at a time, and so on, eh?) Has anyone mentioned Pi Calculus yet? https://en.wikipedia.org/wiki/Pi-calculus https://en.wikipedia.org/wiki/Pi-calculus
- kvakil 7y ago
  Anecdote: for me the actor model has been the most understandable and useful concurrency primitive I've used. Pi-calculus, which was inspired by the actor model, is similarly elegant.
Ericson2314 7y ago
One version of the parallel vs concurrency distinction is parallelism is the easy part and concurrency is the hard part. I would say that for concurrency, FRP (or various incremental programming technics) is essential. The problem is the possible control flow / traces space is just too large to characterize by hand. Another problem is control flow can be bidirectional. If you join together two streams and wish to be push driven, when you reach the merge you need to be able to pull from the other side. Virtually all of Futures/Tokio, hit JS thing du hour, etc etc fail at this. Even forks are a bit hard to do manual CPS as you only have one continuation. Blocking calls fail because the inflexibility of linear 1 input 1 output, but this is better known.
- throwaway17_17 7y ago
  Do you have any links handy discussing the need and implementation of the push/pull dynamic you’re talking about. I’ve seen passing references before, but I’ve never come across any substantial discussion of it outside of knowing Conal Elliott has a paper on push/pull FPR.
  - Ericson2314 7y ago
    I don't know of other papers, sadly. I am at the company behind reflex-frp.org/ so it's code + oral tradition for us.
deleted 7y ago
[deleted]
war1025 7y ago
The thing I finally realized after being big on parallel algorithms and such since first learning about them in university is that, honestly, most things don't need to be cleverly parallel. The majority of tasks are "embarrassingly parallel" as they say. Which is just to mean, there are a lot of standalone tasks that need to be done. For example, any web server is embarrassingly parallel. You don't need to be clever, you just need to have the processing power available to handle them.
- WrtCdEvrydy 7y ago
  Yeah, to be honest, parallelism is great for number crunching. I used GPU.js for some frontend crunching and it turns out even a builtin GPU (integrated graphics) can get a small upgrade out of moving some data into a more parallel format.
  - onion2k 7y ago
    I've only looked at the readme for gpu.js so far but from that I guess it's doing some sort of transformation from the subset of JS it supports in to GLSL, and then running that as a shader in a WebGL context. Presumably there's a not-significant overhead in both starting it up and getting data back from the gpu.js function. Is there a way to "compile" a gpu.js function to remove the translation step? That could be really useful.
    - axel180 7y ago
      Hi! I maintain GPU.js. >I guess it's doing some sort of transformation That is exactly correct. From javascript to an abstract syntax tree, and then to whatever target language we need. In this case GLSL. >Presumably there's a not-significant overhead in both starting it up That has been the goal, we're really careful to measure it with releases. >and getting data back from the gpu.js function That is correct as well, it is about as costly as it would be without GPU.js. However you can enable pipeline on kernels, so they can communicate via textures, which is significantly faster, as we don't need to transfer back to the CPU. >Is there a way to "compile" a gpu.js function to remove the translation step? That could be really useful. It is built into every kernel. You simply call `kernel.toString(...args)`.
Syzygies 7y ago
Haskell. I'm at almost fifty years of programming (APL was an early highlight; punched cards weren't) in many languages. Haskell isn't the be all / end all of languages, but when I want to add ten lines to a program and have it use every core, I don't know a better choice than Haskell. Nothing else comes close.
- Polylactic_acid 7y ago
  I used Haskell for a few months. It seemed like a really well designed language. I didn't look too much about the multi threading stuff (Does it just work automatically for all haskell code?) The main issue I found was the language and libraries are really hard to pick up as a beginner. A library for ruby or python would start off with a few examples showing how the most common use case works and then full api docs for the library. Haskell libraries usually only provide a function name and type definition and if you are lucky you get a one liner telling you what the function does but not how to use it. This seems to be not an issue for haskell experts because "PathMultiPiece ps => (ps -> Writer LiteApp ()) -> Writer LiteApp ()" is all the info needed but I found it much harder to use because of that. I also found that many of the libraries were totally inaccessible unless you had an understanding abstract mathematical constructs. I wanted to do some basic xml parsing in haskell and the prerequisite was reading some huge book on the theory of some math behind how the library works. I then switched to ruby and nokogiri had on the website basically the exact code snipit I needed.
  - k_bx 7y ago
    This is a totally valid point. There are now initiatives like https://tech.fpcomplete.com/haskell https://tech.fpcomplete.com/haskell that gather tutorials for essential stuff, but I think we're far from being beginner-friendly. I don't think there's any other option except for improving libraries little by little, adding well-written tutorials.
    - verttii 7y ago
      Haskell absolutely lacks the soft documentation needed to attract new people. Just about every library could use a "mini tutorial" in their README, at the very least, to allow even novices to quickly bootstrap them and start getting productive without having to understand all the underlying concepts.
ww520 7y ago
Parallel programming is not hard inherently, just structure your data parallel friendly, i.e. pay attention to the dependency among data.
- username90 7y ago
  Parallel programming is not hard if you have already solved all the hard parts.
shaklee3 7y ago
I feel like every comment here simply read the title, and not even skimmed the book. This is a book about extreme low-level optimizations in concurrency. It's a poor title for this kind of book since it seems like it's a beginner book, but it's quite the opposite. It's arguing it's hard to implement the structures and methods used by concurrency APIs, not that it's hard to use them.
- joe_the_user 7y ago
  It doesn't help that the link goes to a page that links to another page that links to the book and the first page doesn't make it clear that that it's talking about a book.
  - gpderetta 7y ago
    is there an HTML version?
    - shaklee3 7y ago
      I believe there's only a PDF version, and the author encourages you to print it. It's a monumental effort to make that kind of book, so it's pretty amazing he released it free.
- platz 7y ago
  It's hard to find problems that benefit from parallelism, because most work has sequential elements. Due to amdhal's law, it's harder to realize significant parallel gains unless the problem is of a special or constrained type.
  - stingraycharles 7y ago
    This seems like unnecessarily black and white: in the real world, most parallel solutions have a part shared state / sequential operations, and a part parallel. Just because a solution has some sequential elements, doesn’t mean it doesn’t benefit from parallelism.
rubyn00bie 7y ago
Oh parallelism, the one thing I used to worry about more, and probably should (as far as using GPUs goes)... but also one I don't really sweat so much. Most of the reason, for me, was finding Elixir/Erlang AND their virtual machine BEAM. Until Elixir and BEAM for me, parallel programming had been difficult because the competing mental models in other languages pretty much make for a textbook fantasy. I'm sure some folks just hit rage land, "Pardon me, you contemptuously ignorant and vile buffoon, what did you say?" but if you'd be so kind as to let me qualify it... Parallelism on computers has more or less evolved from things being asynchronous which gave us the effect of parallelism (yay, for concurrency) to real parallelism with the proliferation of multi-core computers. This jump from emulated parallelism to real parallelism hasn't really been too thoughtfully designed into most languages, and for those starting out, depending on your language and runtime/vm, you might be in for quite an odyssey. We live in a world where almost every language you use has NOT been designed with parallelism as a first class problem. I'm going to try to give an abridged version of my odyssey, Please note I love anything I seem hate on in this tirade. First was Ruby, pure and simple, it has (system) processes (forking), threads, fibers, and libraries implementing actors and event-loops :naive-heart-eyes: but _none_ of them offer real parallelism inside the runtime itself. When one writes code with one of those mental models it falls apart at the metal (hey, GIL) and eventually you're left scratching your head, disillusioned with the world, and realize "why am I even doing this if my work is CPU bound? It won't matter." Next up, I knew the JVM had "REAL THREADS!" I needed to get experience with another language and thought "oh hell, yeah, baby, I'm gonna rock the shit outta these 1.. 2... 4 CORES" watch out CPU bound tasks. Then came reality creeping in like Santa Claus in July... "Creating threads is expensive? Oh. No, shit?" and "oh wow, so like, I now have between 1 and N threads, that I must to schedule, and now I lock my system somehow." I was told apply locks until the burning goes away, but it never does.. And in that a drunken haze of threading hell one night I saw what I thought was a beacon of hope, a JVM based Actor pattern named Akka. "I get the actor model, how could this go wrong?" So. Many. Ways. First and foremost is choking this new fangled "thread pool," Akka had given me with my CPU bound tasks. How does one then gloriously obliterate CPU bound tasks? Oh shit... One goes right back to where they started... in a separate thread. Or one can do as I did, and GALAXY BRAIN the solution: create a separate pool for my CPU bound tasks. This also introduces a new gang of friends "back pressure," "priority," and my favorite "contention." Eventually, I kind of accepted fate, knowing that some mental model even if totally contrived was better than no mental model-- right? Still I believed there had to be something better, and I kept reading about how awesome Erlang was for this sort of thing [1] at the time and Elixir had just come out. Peanut butter meet pickles. This shit was good. Erlang the language, and it's runtime/vm BEAM, were designed to work together to solve a problem space: asynchronous, fault tolerant, and distributed systems. At that point of realization, I blacked out from astonishment that people could write asynchronous code in a language and run-time designed to do so. Turns out its mental model was for me, the end all be all, because they (lang/vm) were, and this my big point, DESIGNED TOGETHER TO SOLVE THE PROBLEM. On BEAM a "process" can be pre-empted (queue choir of angels) and thus _everything_ is non-blocking. The actor pattern was finally able to hit the metal in my head and my beard gained substantial majesty. I write Elixir nearly all day long right now, and while I love it, I wish there were other options (I should probably check out Pony again) for me to use, but honestly, I don't know of any. I'm beyond thankful for Erlang and BEAM since it really helped out my software development game and understanding how parallelism works. P.S. Anyone know if is there a language designed for running on GPUs designed to make it easier to facilitate programming even if they're slightly slower to execute, e.g. the Erlang/OTP/Beam of GPUs? [1] BEAM is actually not great a CPU bound tasks, but that's a different story.
fulafel 7y ago
To answer the headline, a worthy option is to not play the game. It's another kind of optimization and often it's better to spend the effort on making your program more understandable and accessible, more correct, simpler, have more features, more iterations with user feedback, easier to change, cost less engineering-hours, etc.
imtringued 7y ago
Understand the fundamental problems/principles. Almost all problems and solutions stem from one limitation: only one writer at any given time. Immutable means one writer. Locking means one writer. Splitting a list and processing each half means one writer. The actor model means one writer. All are valid solutions to your problems. A piece of personal advice: Avoid acquiring more than one lock at the same time if possible. The solutions are usually incredibly messy. The dining philosopher problem is a pretty good example. Every single problem involving multiple "per object" locks requires a hand crafted solution to the problem. Changing the order in which forks are picked up does not generalize well to other problems. If you have global locks you may get away with always acquiring them in the same order but if you make a single mistake you're opening yourself up to deadlocks.
- cryptica 7y ago
  These are good tips but I would also add that if you can avoid shared memory altogether, then you should avoid it. If you can shard and distribute the data neatly and evenly so that each process handles a completely separate subset, then that is the most reliable long term solution. With shared memory, it's difficult to control who has access to what parts and when - and that can make it difficult to detect and analyse bottlenecks.
- dreamcompiler 7y ago
  > Avoid acquiring more than one lock at the same time if possible. If you know you're going to need 2 locks and you acquire both of them at the same time atomically, deadlock is impossible. The trouble happens when you acquire one lock, wait a bit, then acquire a second one. Maybe that's what you meant and I just misread it.
- allovernow 7y ago
  What about sequential locking/unlocking with consistent locking order for all resources? I was under the impression that this solved the deadlock problem but I haven't yet tried it in practice.
- JoeAltmaier 7y ago
  Queues are my advice. The locking is all in the queue operations. Everybody else just takes and posts work items to the queue. Can even use lockless queues.
novaleaf 7y ago
If you are a C# developer: I'm writing an open-source async Job System, very similar to that found in Unity and other modern game engines. You can track development here: https://github.com/jasonswearingen/godot-csharp-tech https://github.com/jasonswearingen/godot-csharp-tech I'm just starting the implementation, but I've done a similar system before so pretty sure I'm not in over my head :)

135 points by matt_d 7y ago

Direct jump to the PDF download page, avoiding the terrible hosting and Russian ads: https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html https://mirrors.edge.kernel.org/pub/linux/kernel/people/paul...

dang 7y ago

A thread from 2015: https://news.ycombinator.com/item?id=9315152 https://news.ycombinator.com/item?id=9315152 2014: https://news.ycombinator.com/item?id=7381877 https://news.ycombinator.com/item?id=7381877 2011: https://news.ycombinator.com/item?id=2784515 https://news.ycombinator.com/item?id=2784515

matt_d 7y ago
Thanks! FWIW, I've used the original post title ("Parallel Programming: December 2019 Update") to focus on the updates (and partially driven by the fear of comments made right after reading just the book title, "Is Parallel Programming Hard, And, If So, What Can You Do About It?", and focusing solely on answering the question while completely ignoring the content... :-]).

Koshkin 7y ago

Parallel programming is not hard per se, it's just that there's so much more to learn (compared to "sequential programming"): threads, processes, IPC, messaging, actors, queues, synchronization primitives (mutexes, semaphores, monitors, condition variables, critical sections, atomics... you name it) along with their gotchas, concurrent containers with their limitations, transactions, distributed computing with its performance implications, new design patterns (e.g. async/await, promises and futures). But hey, programming has never been thought of as easy, right? Yet, speaking of having it hard, it may be enlightening to compare software development with having to design a piece of (digital) hardware, where not only much has always been happening "in parallel" but one also must keep taking into account physical characteristics of various parts of the circuit.

gopalv 7y ago
> Parallel programming is not hard per se, it's just that there's so much more to learn Parallel programming isn't the hardest part (it is hard, but good teachers help with good abstractions over problems), it is the parallel debugging that is - or at least the second pass over the same codebase. Often being able to write decent parallel code is easier than debugging a system which is behaving inconsistently without any ability to set up the timing bugs the right way around. The difference between "I fixed a bug" and "I massively lowered the probability of it" is hard to tell. I spent 7 months chasing an issue that happened once on average of a billion requests (or rather once every 2-3 days across a web farm). The fix was easy, but debugging it drove me insane slowly over a whole summer. And after that, there's the loss of efficiency from parallelization which needs enough parallel tracks to actually be faster - more threads is often slower at low thread counts. As an anecdote, there were enough servers at Yahoo which had SMT turned off because the extra locking introduced by 2 threads on a core ate up the extra CPU those threads freed up.
- agumonkey 7y ago
  This gives me two questions: - did you have a method/tool to debug (from book, or chatting with others) - are there still cpu designs dedicated to parallel workloads (I assume the usual desktop intel/amd, no matter how brilliant, may not be without hiccups in high parallelism) oh one last thing, do you use dedicated compilers for parallelization ? or is it something "mainstream" like openmp
  - namibj 7y ago
    I can greatly recommended mozilla's rr for hunting concurrency bugs, as it allows efficient replay and provides a scheduler that is designed to increase the probability of triggering concurrency bugs. The issue with x86 is simple: the memory model pretty much requires loads of broadcasts on the SMP interconnect. IIRC Power9 is nice due to (1) a memory model that doesn't require this (much) broadcasting on the interconnect, and (2) remote atomics like "fetch current 64bit value pointed to by register1 and atomically add register2 after", which prevent the cacheline from bouncing between cores or even sockets (Power9 scales to about 5~10kW TDP @8~20W TDP/core and 4 threads/core (there are versions that combine core pairs to have 8 threads using the resources of (almost) 2 cores (overhead / diminishing returns))). I can't really recommend OpenMP. If the architectural restrictions aren't a problem, you might want to check out Intel's TBB, which uses C++ templates instead of compiler pragmas. Unfortunately you'd ideally need multithreading-aware polyhedral optimization, and _that_ mostly stalled AFAIK. You'd likely write C++ or now maybe Rust with low-level primitives / intrinsics to build libraries. I don't know of any actually good "automatic" frameworks for general shared-memory parallel programming (There's Chapel, but it seems fairly niche. Still going strong though, by the looks of it.). If you can efficiently express the problem with loop-dataflow, you might want to try timely-dataflow (Rust, does thread/process/node parallelism), as it nicely integrates with normal Rust code.

zwaps 7y ago

Parallel programming in Python is hard. On Windows, it's hell. In Matlab it's a breeze. Just put "par" in front of the thing. Done. If you wanna shell out money to enable that functionality, that is.

honestoHeminway 7y ago
Software is able to reason about software? Do you notice something in that sentence? If it would work - it would make the programers useless. But it doesent. Once a pointer points at something in your parallel task, software is unable to reason about it, erring on the side of caution and thus, parallelism wont happen there. Also these parallelisms, usually work for trivial parallel structures like for loops. Easy to unroll into multiple cores, but also with little gain, as communication times eats the gained advantages.

jhfdbkofdcho 7y ago

Someone wiser than me observed, “the thing about parallel programming is that it always ends in tears.”

carapace 7y ago

It's hard or easy depending on your model (of parallelism) and what you want to do with it. From a certain POV computers are always doing things in parallel (8-, 16-, 32-, 64-bits at a time, and so on, eh?) Has anyone mentioned Pi Calculus yet? https://en.wikipedia.org/wiki/Pi-calculus https://en.wikipedia.org/wiki/Pi-calculus

kvakil 7y ago
Anecdote: for me the actor model has been the most understandable and useful concurrency primitive I've used. Pi-calculus, which was inspired by the actor model, is similarly elegant.

Ericson2314 7y ago

One version of the parallel vs concurrency distinction is parallelism is the easy part and concurrency is the hard part. I would say that for concurrency, FRP (or various incremental programming technics) is essential. The problem is the possible control flow / traces space is just too large to characterize by hand. Another problem is control flow can be bidirectional. If you join together two streams and wish to be push driven, when you reach the merge you need to be able to pull from the other side. Virtually all of Futures/Tokio, hit JS thing du hour, etc etc fail at this. Even forks are a bit hard to do manual CPS as you only have one continuation. Blocking calls fail because the inflexibility of linear 1 input 1 output, but this is better known.

throwaway17_17 7y ago
Do you have any links handy discussing the need and implementation of the push/pull dynamic you’re talking about. I’ve seen passing references before, but I’ve never come across any substantial discussion of it outside of knowing Conal Elliott has a paper on push/pull FPR.
- Ericson2314 7y ago
  I don't know of other papers, sadly. I am at the company behind reflex-frp.org/ so it's code + oral tradition for us.

deleted 7y ago

[deleted]

war1025 7y ago

The thing I finally realized after being big on parallel algorithms and such since first learning about them in university is that, honestly, most things don't need to be cleverly parallel. The majority of tasks are "embarrassingly parallel" as they say. Which is just to mean, there are a lot of standalone tasks that need to be done. For example, any web server is embarrassingly parallel. You don't need to be clever, you just need to have the processing power available to handle them.

WrtCdEvrydy 7y ago
Yeah, to be honest, parallelism is great for number crunching. I used GPU.js for some frontend crunching and it turns out even a builtin GPU (integrated graphics) can get a small upgrade out of moving some data into a more parallel format.
- onion2k 7y ago
  I've only looked at the readme for gpu.js so far but from that I guess it's doing some sort of transformation from the subset of JS it supports in to GLSL, and then running that as a shader in a WebGL context. Presumably there's a not-significant overhead in both starting it up and getting data back from the gpu.js function. Is there a way to "compile" a gpu.js function to remove the translation step? That could be really useful.
  - axel180 7y ago
    Hi! I maintain GPU.js. >I guess it's doing some sort of transformation That is exactly correct. From javascript to an abstract syntax tree, and then to whatever target language we need. In this case GLSL. >Presumably there's a not-significant overhead in both starting it up That has been the goal, we're really careful to measure it with releases. >and getting data back from the gpu.js function That is correct as well, it is about as costly as it would be without GPU.js. However you can enable pipeline on kernels, so they can communicate via textures, which is significantly faster, as we don't need to transfer back to the CPU. >Is there a way to "compile" a gpu.js function to remove the translation step? That could be really useful. It is built into every kernel. You simply call `kernel.toString(...args)`.

Syzygies 7y ago

Haskell. I'm at almost fifty years of programming (APL was an early highlight; punched cards weren't) in many languages. Haskell isn't the be all / end all of languages, but when I want to add ten lines to a program and have it use every core, I don't know a better choice than Haskell. Nothing else comes close.

Polylactic_acid 7y ago
I used Haskell for a few months. It seemed like a really well designed language. I didn't look too much about the multi threading stuff (Does it just work automatically for all haskell code?) The main issue I found was the language and libraries are really hard to pick up as a beginner. A library for ruby or python would start off with a few examples showing how the most common use case works and then full api docs for the library. Haskell libraries usually only provide a function name and type definition and if you are lucky you get a one liner telling you what the function does but not how to use it. This seems to be not an issue for haskell experts because "PathMultiPiece ps => (ps -> Writer LiteApp ()) -> Writer LiteApp ()" is all the info needed but I found it much harder to use because of that. I also found that many of the libraries were totally inaccessible unless you had an understanding abstract mathematical constructs. I wanted to do some basic xml parsing in haskell and the prerequisite was reading some huge book on the theory of some math behind how the library works. I then switched to ruby and nokogiri had on the website basically the exact code snipit I needed.
- k_bx 7y ago
  This is a totally valid point. There are now initiatives like https://tech.fpcomplete.com/haskell https://tech.fpcomplete.com/haskell that gather tutorials for essential stuff, but I think we're far from being beginner-friendly. I don't think there's any other option except for improving libraries little by little, adding well-written tutorials.
  - verttii 7y ago
    Haskell absolutely lacks the soft documentation needed to attract new people. Just about every library could use a "mini tutorial" in their README, at the very least, to allow even novices to quickly bootstrap them and start getting productive without having to understand all the underlying concepts.

ww520 7y ago

Parallel programming is not hard inherently, just structure your data parallel friendly, i.e. pay attention to the dependency among data.

username90 7y ago
Parallel programming is not hard if you have already solved all the hard parts.

shaklee3 7y ago

I feel like every comment here simply read the title, and not even skimmed the book. This is a book about extreme low-level optimizations in concurrency. It's a poor title for this kind of book since it seems like it's a beginner book, but it's quite the opposite. It's arguing it's hard to implement the structures and methods used by concurrency APIs, not that it's hard to use them.

joe_the_user 7y ago
It doesn't help that the link goes to a page that links to another page that links to the book and the first page doesn't make it clear that that it's talking about a book.
- gpderetta 7y ago
  is there an HTML version?
  - shaklee3 7y ago
    I believe there's only a PDF version, and the author encourages you to print it. It's a monumental effort to make that kind of book, so it's pretty amazing he released it free.
platz 7y ago
It's hard to find problems that benefit from parallelism, because most work has sequential elements. Due to amdhal's law, it's harder to realize significant parallel gains unless the problem is of a special or constrained type.
- stingraycharles 7y ago
  This seems like unnecessarily black and white: in the real world, most parallel solutions have a part shared state / sequential operations, and a part parallel. Just because a solution has some sequential elements, doesn’t mean it doesn’t benefit from parallelism.

rubyn00bie 7y ago

Oh parallelism, the one thing I used to worry about more, and probably should (as far as using GPUs goes)... but also one I don't really sweat so much. Most of the reason, for me, was finding Elixir/Erlang AND their virtual machine BEAM. Until Elixir and BEAM for me, parallel programming had been difficult because the competing mental models in other languages pretty much make for a textbook fantasy. I'm sure some folks just hit rage land, "Pardon me, you contemptuously ignorant and vile buffoon, what did you say?" but if you'd be so kind as to let me qualify it... Parallelism on computers has more or less evolved from things being asynchronous which gave us the effect of parallelism (yay, for concurrency) to real parallelism with the proliferation of multi-core computers. This jump from emulated parallelism to real parallelism hasn't really been too thoughtfully designed into most languages, and for those starting out, depending on your language and runtime/vm, you might be in for quite an odyssey. We live in a world where almost every language you use has NOT been designed with parallelism as a first class problem. I'm going to try to give an abridged version of my odyssey, Please note I love anything I seem hate on in this tirade. First was Ruby, pure and simple, it has (system) processes (forking), threads, fibers, and libraries implementing actors and event-loops :naive-heart-eyes: but _none_ of them offer real parallelism inside the runtime itself. When one writes code with one of those mental models it falls apart at the metal (hey, GIL) and eventually you're left scratching your head, disillusioned with the world, and realize "why am I even doing this if my work is CPU bound? It won't matter." Next up, I knew the JVM had "REAL THREADS!" I needed to get experience with another language and thought "oh hell, yeah, baby, I'm gonna rock the shit outta these 1.. 2... 4 CORES" watch out CPU bound tasks. Then came reality creeping in like Santa Claus in July... "Creating threads is expensive? Oh. No, shit?" and "oh wow, so like, I now have between 1 and N threads, that I must to schedule, and now I lock my system somehow." I was told apply locks until the burning goes away, but it never does.. And in that a drunken haze of threading hell one night I saw what I thought was a beacon of hope, a JVM based Actor pattern named Akka. "I get the actor model, how could this go wrong?" So. Many. Ways. First and foremost is choking this new fangled "thread pool," Akka had given me with my CPU bound tasks. How does one then gloriously obliterate CPU bound tasks? Oh shit... One goes right back to where they started... in a separate thread. Or one can do as I did, and GALAXY BRAIN the solution: create a separate pool for my CPU bound tasks. This also introduces a new gang of friends "back pressure," "priority," and my favorite "contention." Eventually, I kind of accepted fate, knowing that some mental model even if totally contrived was better than no mental model-- right? Still I believed there had to be something better, and I kept reading about how awesome Erlang was for this sort of thing [1] at the time and Elixir had just come out. Peanut butter meet pickles. This shit was good. Erlang the language, and it's runtime/vm BEAM, were designed to work together to solve a problem space: asynchronous, fault tolerant, and distributed systems. At that point of realization, I blacked out from astonishment that people could write asynchronous code in a language and run-time designed to do so. Turns out its mental model was for me, the end all be all, because they (lang/vm) were, and this my big point, DESIGNED TOGETHER TO SOLVE THE PROBLEM. On BEAM a "process" can be pre-empted (queue choir of angels) and thus _everything_ is non-blocking. The actor pattern was finally able to hit the metal in my head and my beard gained substantial majesty. I write Elixir nearly all day long right now, and while I love it, I wish there were other options (I should probably check out Pony again) for me to use, but honestly, I don't know of any. I'm beyond thankful for Erlang and BEAM since it really helped out my software development game and understanding how parallelism works. P.S. Anyone know if is there a language designed for running on GPUs designed to make it easier to facilitate programming even if they're slightly slower to execute, e.g. the Erlang/OTP/Beam of GPUs? [1] BEAM is actually not great a CPU bound tasks, but that's a different story.

fulafel 7y ago

To answer the headline, a worthy option is to not play the game. It's another kind of optimization and often it's better to spend the effort on making your program more understandable and accessible, more correct, simpler, have more features, more iterations with user feedback, easier to change, cost less engineering-hours, etc.

imtringued 7y ago

Understand the fundamental problems/principles. Almost all problems and solutions stem from one limitation: only one writer at any given time. Immutable means one writer. Locking means one writer. Splitting a list and processing each half means one writer. The actor model means one writer. All are valid solutions to your problems. A piece of personal advice: Avoid acquiring more than one lock at the same time if possible. The solutions are usually incredibly messy. The dining philosopher problem is a pretty good example. Every single problem involving multiple "per object" locks requires a hand crafted solution to the problem. Changing the order in which forks are picked up does not generalize well to other problems. If you have global locks you may get away with always acquiring them in the same order but if you make a single mistake you're opening yourself up to deadlocks.

cryptica 7y ago
These are good tips but I would also add that if you can avoid shared memory altogether, then you should avoid it. If you can shard and distribute the data neatly and evenly so that each process handles a completely separate subset, then that is the most reliable long term solution. With shared memory, it's difficult to control who has access to what parts and when - and that can make it difficult to detect and analyse bottlenecks.
dreamcompiler 7y ago
> Avoid acquiring more than one lock at the same time if possible. If you know you're going to need 2 locks and you acquire both of them at the same time atomically, deadlock is impossible. The trouble happens when you acquire one lock, wait a bit, then acquire a second one. Maybe that's what you meant and I just misread it.
allovernow 7y ago
What about sequential locking/unlocking with consistent locking order for all resources? I was under the impression that this solved the deadlock problem but I haven't yet tried it in practice.
JoeAltmaier 7y ago
Queues are my advice. The locking is all in the queue operations. Everybody else just takes and posts work items to the queue. Can even use lockless queues.

novaleaf 7y ago

If you are a C# developer: I'm writing an open-source async Job System, very similar to that found in Unity and other modern game engines. You can track development here: https://github.com/jasonswearingen/godot-csharp-tech https://github.com/jasonswearingen/godot-csharp-tech I'm just starting the implementation, but I've done a similar system before so pretty sure I'm not in over my head :)