11 ms·
Is parallel programming hard, and, if so, what can you do about it? [pdf]
- binary0x01 4y agoTry using libthread on plan9, no locks.
- dang 4y agoRelated: “Is Parallel Programming Hard, and, If So, What Can You Do About It?” v2 Is Out - https://news.ycombinator.com/item?id=26537298 https://news.ycombinator.com/item?id=26537298 - March 2021 (75 comments) Is parallel programming hard, and, if so, what can you do about it? - https://news.ycombinator.com/item?id=22030928 https://news.ycombinator.com/item?id=22030928 - Jan 2020 (85 comments) Is Parallel Programming Hard, and, If So, What Can You Do About It? [pdf] - https://news.ycombinator.com/item?id=9315152 https://news.ycombinator.com/item?id=9315152 - April 2015 (31 comments) Is Parallel Programming Hard, And, If So, What Can You Do About It? - https://news.ycombinator.com/item?id=7381877 https://news.ycombinator.com/item?id=7381877 - March 2014 (26 comments) Is Parallel Programming Hard, And, If So, What Can You Do About It? - https://news.ycombinator.com/item?id=2784515 https://news.ycombinator.com/item?id=2784515 - July 2011 (39 comments)
- dragontamer 4y ago1. Try process level I/O, such pipes, sockets, and the like. Have Linux deal with the concurrency problem, not you. (Note: the BASH & background job works in so many cases it ain't funny). Also try fork/join parallelism models like OpenMP. These are all far easier than dipping down to a lower level. 2. Try a mutex 3. If that doesn't work, try adding a condition variable. 4. If that still doesn't work, try an atomic in default sequentially consistent mode or equivalent (ex: Java volatile, InterlockedAdd, and the like). Warning: atomics are very subtle. Definitely have a review with an expert if you are here. 5. If that still doesn't work, consider lock free paradigms. That is, combinations of atomics and memory barriers. 6. If that still doesn't work, publish a paper on your problem lol. --------- #1 is my most important piece of advice. There was a Blender render I was doing, like 2.6 or something old a few years ago. Blenders parallelism wasn't too good and only utilized 25% of my computer. So I ran 4 instances of headless Blender. Bam, 100% utilization. Done. Don't overthink parallelism. It's stupid easy sometimes, as easy as a & on the end of your shell command.
- chasil 4y agoThe Oracle database has adopted process-level parallelism, utilizing System V IPC. Threading is used on Windows for performance reasons, but each client gets its own server pid by default on UNIX. This architecture expresses the original design intentions of "Columbus UNIX." "CB UNIX was developed to address deficiencies inherent in Research Unix, notably the lack of interprocess communication (IPC) and file locking, considered essential for a database management system... The interprocess communication features developed for CB UNIX were message queues, semaphores and shared memory support. These eventually appeared in mainstream Unix systems starting with System V in 1983, and are now collectively known as System V IPC." This approach has realized some degree of success. https://en.m.wikipedia.org/wiki/CB_UNIX https://en.m.wikipedia.org/wiki/CB_UNIX
- anarazel 4y agoPostgres also uses a multi process architecture. But I think that turned out to be a mistake for something like a database, on modern systems. There are other reasons, but the biggest problem is that inter process context switches are considerably more expensive than intra process ones. Far less efficient use of the TLB being a big part of that. It used to be worse before things like process context identifiers, but even with them you're wasting a large portion of the TLB by storing redundant information.
- chasil 4y agoOracle still managed to be the TPC-C performance leader from 12/2010 until Oceanbase took the crown in 2019 (I don't think that Oracle cares anymore). https://www.tpc.org/tpcc/results/tpcc_results5.asp?print=false&orderby=tpm&sortby=desc https://www.tpc.org/tpcc/results/tpcc_results5.asp?print=fal... https://www.alibabacloud.com/blog/oceanbase-breaks-tpc-c-record-a-dialogue-with-ant-financial-experts_596271 https://www.alibabacloud.com/blog/oceanbase-breaks-tpc-c-rec... They did this with an SGA (Shared Global Area) that (I'm assuming) pollutes the Translation Lookaside Buffer (TLB) with different addresses for this shared memory in every process.
- 4y ago
- gavinhoward 4y agoFunny. I've been reading this for the past six months and just finished today. Life is weird.
- JUNGLEISMASSIVE 4y ago[dead]
- credit_guy 4y agoAnd, did you like it? I guess you did, otherwise you wouldn't waste six months on it. But can you share in a few words what you think of this book?
- gavinhoward 4y agoSilly me for thinking that people wouldn't care about my opinion. It's a great book. Some things could be better, of course, but it's a free book, so the quality to price ratio is off the charts, and not just because it's free. I would have happily paid $75 for it, maybe more. That said, it's a book that requires you to care about the Linux kernel, where the author's experience is. If that's a problem, then you won't get as much out of it. The book is best used as a reference after reading through it once. I would do a medium-deep read the first time. This will tell you the concepts you need to look up later, what techniques exist, etc. This is because the book delves into detail about various techniques to get concurrency. The philosophy is to do the easiest thing that works, which is great, but it does mean it talks about details. Thus, it's best as a reference later after absorbing the surface level. However, that medium-deep read is still necessary for you to know what you need to look for later when you need details. I hope that helps.
- bullen 4y agoI use OS threads + non-blocking IO with concurrent package for shared data in Java. The performance is incredible. If I wanted to get a little more performance per watt I would probably rewrite it in C with arrays of atomic variables. But you need a VM with GC to be able to be productive during the day and sleep at night, so probably not...
- zelphirkalt 4y agoNow, if you could reduce the overhead of OS threads by using lightweight processes instead ... Wait a moment, isn't that what Erlang does? Well, project Lumen might help you out on the JVM at some point.
- bullen 4y agoWith NIO there is no OS overhead like context switches because you only need one thread per core as long as everything is async. Erlang is single threaded unless you copy memory between threads. The only step left for Java to implement (after io_uring file stuff) is user-space networking.
- zelphirkalt 4y agoThe Beam VM handles memory copying, afaik. It is very much multi-process. It has very lightweight processes, of which you can spawn thousands. Those are then run in as many OS threads as makes sense and are available to the virtual machine. It is unclear what you mean by Erlang (the language) being single threaded. The VM easily runs on as many cores as you give it, probably by using as many or more OS threads.
- college_physics 4y agoIts interesting that while Moore's law saturated many years ago there is still no parallel programming style that hits some sweet spot between productivity and performance for multicore cpus (and thus gets more adopted for mainstream development) Its not clear if this means there is not such "optimum" or simply it is not something anybody cares about people focused a lot on gpus but thats not easy either
- toast0 4y agoI think it's because shared memory concurrency is easy to start on in lots of popular languages. But it doesn't take long until you're in a tricky mess of locks. Actor style based on explicit communication and no shared memory is a lot easier to work with (IMHO), but it's not as easy to get started on because it's not as simple as pthread_create and go.
- apelapan 4y agoNo shared memory will also mean that actor style is slower-than-single-thread-slow, when the "message-size to processing-per-actor ratio" is not right. Actors and message passing is great for problems where it fits and worse than useless where it doesn't
- jcranmer 4y agoBut there is a dominant parallel programming style. In fact, it actually boils down to one of two styles: * Here's a list of things. Run the same bit on code for every item in the list of things. (Slight adjustment is necessary if you need to something like a reduction tree). * Here's a graph of tasks, with dependencies expressed as edges. Run as much as you can in parallel. What makes parallel programming difficult is two main things. First, the way to achieve parallelism is highly dependent on the size of the tasks, with designs for one scale being horribly bad ideas at different scales. Second, there's a pretty severe penalty when communication between tasks is involved (and, notably, two tasks both wanting to read the same data can cause pain, not just read/write or write/write conflicts).
- 4y ago
- userbinator 4y agoSection 2.3.3 is definitely worth reading carefully. I've both increased efficiency and removed bugs by rewriting a system that someone thought the only way to make faster was to add more threads, when optimising algorithms and data layout resulted in much more gains.
- didgetmaster 4y agoThere are three ways to 'scale' a computer program. The first is optimizing the algorithms and data structures which is discussed in the section you noted. Make your code run as fast as possible using a single thread even when the amount of data you are processing gets extremely large. The second is taking advantage of better hardware (more cores, bigger L2 and L3 caches, SSDs, etc.) which includes spinning off threads to do single tasks in parallel. The third is a 'scale out' architecture where the application is spread out across multiple machines. With all the cloud infrastructure available today, too many programmers focus on the third option (which can mask inefficiencies in the first two) when they could get as good or better performance at a cheaper price by focusing on the other options instead.
- mhsdef 4y agoUse an actor model language -- by far the sanest way. Message passing is intuitive to human experience. 1. Elixir (Erlang) 2. Scala/Akka 3. Pony
- travisgriggs 4y agoCame here to say just this. In particular, _immutable_ message passing.
- arunc 4y ago4. D
- eslaught 4y agoI am biased because this is my research area, but I have to respectfully disagree. Actor models are awful, and the only reason it's not obvious is because everything else is even more awful. But if you look at e.g., the recent work on task-based models, you'll see that you can have literally sequential programs that parallelize automatically. No message passing, no synchronization, no data races, no deadlocks. Read your programs as if they're sequential, and you immediately understand their semantics. Some of these systems are able to scale to thousands of nodes. An interesting example of this is cuNumeric, which allows you to take sequential Python programs that use NumPy, and by changing one line (the import statement), run automatically on clusters of GPUs. It is 100% pure awesomeness. https://github.com/nv-legate/cunumeric https://github.com/nv-legate/cunumeric (I don't work on cuNumeric, but I do work on the runtime framework that cuNumeric uses.)
- rramadass 4y ago>the recent work on task-based models, you'll see that you can have literally sequential programs that parallelize automatically Can you provide some details please? I am not quite clear what you mean.
- eslaught 4y agoIf you really want to dig into it you can read up on the tutorials and/or papers from the Legion project: https://legion.stanford.edu/ https://legion.stanford.edu/ But briefly, these task-based programs preserve sequential semantics. That means (whatever the system actually does when running your program), as long as you follow the rules, the parallelism should be invisible to the execution of the program.
- didgetmaster 4y agoMulti-threaded programming has been of particular interest to me for decades (since my early years programming for OS/2). Whenever I write code, I look for ways to do things in parallel. My new data management system is highly parallel. I am always finding tasks that take minutes to complete and getting them down to just seconds (when running on multi-core CPUs) by getting multiple threads working together on the same problem. Just yesterday, I found a task that was taking over 12 minutes to finish (inserting 125 million key/value pairs into a data store) and was able to get it to do the same task in just 37 seconds (running on my 16 core/32 thread CPU) by spinning off multiple threads.
- jasfi 4y agoI use channels to send messages between threads in Nim. It works quite well.
- threeseed 4y agoI use ZIO (http://zio.dev http://zio.dev) for Scala which makes parallel programming trivial. Wraps different styles of asynchronicity e.g. callbacks, futures, fibers into one coherent model. And has excellent resource management so you can be sure that when you are forking a task that it will always clean up after itself. Have yet to see anything that comes close whilst still being practical i.e. you can leverage the very large ecosystem of Java libraries.
- delta_p_delta_x 4y agoMy solution is to only solve problems that are embarrassingly parallel, like graphics (one pixel = one thread) or physics simulations (one object = one thread), and escape the pain of synchronisation.
- patrulek 4y agoIts isnt hard but you need to change your programming standpoint. To write parallel code you need to think more about data alignment, dependency and flow. Its quite different than typical object/behaviour oriented programming.