7 ms·

What every systems programmer should know about concurrency (2020) [pdf]

• anacrolix 2y ago
  Relaxed atomic operations
  • mwkaufma 2y ago
    That's section 10.2 of the linked article, yes.
• tobyhinloopen 2y ago
  Every time I dip my toes in multithreading in C++, I just get into a world of random crashes and segfaults. It seems unreasonably hard, hah.
  • saagarjha 2y ago
    Concurrency is hard, unfortunately. C++ just makes it even harder than that.
    • jazzypants 2y ago
      I hear you have to be pretty tall to be able to do it right. [1] [1] https://bholley.net/blog/2015/must-be-this-tall-to-write-multi-threaded-code.html https://bholley.net/blog/2015/must-be-this-tall-to-write-mul...
    • menaerus 2y ago
      CPU design makes it inherently hard. C or C++ is just a thin layer above it making no tradeoffs. If you can live with the tradeoff then Rust land or VM-language land is more appropriate.
      • tialaramex 2y ago
        What is it that you imagine C++ is doing that Rust has traded off?
        • menaerus 2y ago
          In theory none but in practice the codebase ends up littered with hidden shared state mostly disguised through one or another shared pointer implementation. And this happens because that's what the Rust compiler is pessimistically enforcing upon you by default. For heavy workloads, this approach doesn't scale particularly very well.
          • tialaramex 2y ago
            It sounds to me as though what you're saying is that when you write Rust programs which don't scale very well they don't scale very well, whereas when you write C++ programs you don't do that, I suggest learning not to do it in Rust either.
            • menaerus 2y ago
              Easier to be said than done since that's one of the core tradeoffs of the Rust language. Language is forcing these semantics upon you and while it is possible to get around it with unsafe blocks it turns out to be much more difficult in the practice. So, by default Rust code will almost certainly in majority of cases going to be designed around shared ownership.
              • tialaramex 2y ago
                If you actually have shared ownership but in C++ you're getting away with pretending you don't, chances are that'll bite you really hard. Maybe it's already biting and you didn't notice, so Rust actually do you a massive favour. If there is no shared ownership then inventing it so as to make your Rust slower is just a problem with you, not with Rust.
                • 2y ago
    • sumtechguy 2y ago
      We had multi CPU stuff in the 90s. C/C++ was dead set on ignoring it for a long time. Every OS had its own way of handling it and all with subtle weird bits that did not act like other OS's. You could not even trust the stdlib or crt to get it right with their globals that could change underneath you. So it was left to the developer. It is much better now but for so long the problem was ignored and now we have decades of 'not sure if I can touch that code'. Also by default C/C++ are fairly open about sharing memory. So it is very easy create race conditions on memory. It would be nice if the base language had a concept of 'locked to a thread' or 'i want to share this with other threads' then the compiler can flag where I have wandered into the weeds for a class so we could catch the race conditions at compile time, at least some sort of warning. Sharing semantics were awful for a long time. stdlib has done some very good things to help clean that up but it is still very easy to share between threads and cause yourself a headache.
      • tialaramex 2y ago
        C++ itself doesn't have the owning mutex, but there is one in Boost for example. The problem with an owning mutex in such a language is that you can (on purpose or by mistake) keep accessing the thing it owned after you've released the protecting mutex. Rust's Mutex<T> owns the T but it has the behaviour you want where if you tried to keep the access to T but give back the mutex that doesn't compile. "I tried to unlock the mutex but I still need it, d'oh". And the same problem applies broadly, you should not share write access to things without an ordering guarantee, but it's hard to ensure that guarantee is followed in C++.
        • sumtechguy 2y ago
          Exactly. This stuff has been known about for a long time. It was just kind of ignored and you kinda hoped your library might have something to deal with it (boost, win32, pthread, etc). Then each one acted differently on different platforms or with each other. Some of the std lib is starting to have things we need. But now I have to deal with things in the crt and stdlib that actively break multi threading. Mutexes, semaphores, flags, pulsing, etc is not exactly new patterns. Real mess and you have to understand it too deeply for it to be meaningful to more people. It is why things like promise/async/await are very popular with javascript and their libraries. As it looks like multithreaded programming with a decently clear interface as to what it is going on.
  • pionaryon 2y ago
    You can get pretty far with keeping sharing to an absolute minimum and when you do need to share data, slap a lock free ringbuffer between them to communicate. Pretty simple to get right.
    • anal_reactor 2y ago
      Step 1: use concurrency on a very high level. For example, write an app, and then run 4 instances of it, each working on a quarter of your data. Step 2: when you absolutely need concurrency within one app, try using some library that already solved the issue. For example, there are lots of databases with pretty strong concurrency contracts while still being efficient. Step 3: if you absolutely need custom solution, use a library/language that provides reasonable tools out of the box and keep your logic to minimum. Following the first two steps will solve 95% of your concurrency issues, if you include step 3 it goes to 99%.
      • tobyhinloopen 2y ago
        Thanks anal reactor!
    • adrian_b 2y ago
      You are right, but what you call "lock free" is not the same as many other things that are called "lock free", even if indeed a ringbuffer needs no locks, so this may be confusing for newbies. I strongly dislike the term "lock free", which is really just a marketing term invented by people trying to promote the idea that some algorithms are better than those "lock-based", when in fact those "lock-free" algorithms were only choosing a different trade-off in performance, which can be better or worse, depending on the application. Even worse is that after the term "lock free" has become fashionable, it has also been applied to unrelated algorithms, so now it has become ambiguous, so you cannot know for sure what is meant by it, unless more details are provided. When accessing shared data structures, the accesses are most frequently done in one of three ways. The first is to use mutual exclusion, when the shared data structure is accessed within critical sections and only one thread can execute the critical section at a given time. This method is usually called as lock-based access. The second is to use optimistic access, when the shared data structure is accessed concurrently by many threads, but they are able to detect interference from the other concurrent accesses and they retry their accesses in such cases. This is what is most frequently referred as "lock free" access. Compared to mutual exclusion, this access method may be faster in the best cases, but it is much slower in the worst cases, so whether this is a good choice depends on the application. The third method happens when it is possible to partition the shared resource between the threads that access it concurrently, so their concurrent accesses can proceed without fear of interference. This partitioning is usually possible for arrays and for buffers a.k.a. FIFO memories a.k.a. message queues (including one-to-one, many-to-one, one-to-many and many-to-many message queues). So your "lock free ringbuffer" refers to the third method from above, which is very different from the "lock free" algorithms of the second kind from above. Whenever concurrent access to partitioned shared resources is possible, it is much better than accesses with mutual exclusion or optimistic accesses, which require either waiting or retrying, both of which are wasting CPU time. Therefore using correctly-implemented message queues or other kinds of shared buffers is usually the best method to achieve high levels of concurrency, in comparison with other kinds of shared data structures, because it avoids the bottlenecks caused by mutual exclusion or optimistic accesses.
  • bitbasher 2y ago
    If C++ makes it possible to blow your leg off, multithreading with C++ makes it possible to blow yourself up, along with half your neighborhood and some homes halfway around the country, unexplicitly.
    • commandersaki 2y ago
      Really? I don't share either sentiment. I found multithreading in C++ to be pretty mundane, and in fact a lot easier than it probably once was with lambdas. You definitely need to be on top of the multithreading primitives such as atomics, mutex, condition variables, shared_ptr, etc. But otherwise it's pretty straightforward.
      • crabbone 2y ago
        One of the problems I'm usually facing with this is the need to call library functions from C++ code. Especially TLS-related stuff. It doesn't matter what primitives C++ has to offer, as long as you are using library code things will break.
  • mattgreenrocks 2y ago
    The standard library concurrency primitives are still too low-level for a lot of general purpose concurrency needs. IMO, the minimum level of abstraction that you should start with for many apps is a thread-safe queue that dispatches messages with immutable data to one or more worker threads (e.g. actor-like model). You can go lower-level, but it needs to be deliberately chosen.
    • Jtsummers 2y ago
      People keep reinventing those and thread pools over and over in C++. I've been researching one of our older systems (slated for decommissioning in 2021, as is typical that did not happen). In trying to understand it I have found many areas of concern around how they deal with concurrency, in particular they created their own queue and thread pool. Based on past experience, there's 50/50 chance for each that they were created correctly (with proper concurrency controls), and less than that that the tasks submitted to the thread pool themselves make use of proper concurrency controls rather than assuming that they can read/write whatever they want as if the system were single-threaded.
  • rramadass 2y ago
    That is just not true and you are being unnecessarily hyperbolic. Even when i was learning/doing concurrency/multi-threading (using pthreads) long ago, i never got "into a world of random crashes and segfaults". It was of course challenging but not too difficult. You structure your application following standard usage patterns given in a good book (eg. Butenhof's) and then plug in your app logic in the thread routines. With some experience things get clearer over time and you begin to have an intuitive feel for structuring multi-threaded code. The key is to stay at a high enough level of abstraction appropriate for your usecase (eg. mutexes/semaphores/condition variables) before diving into compiler and hardware level intrinsics/atomics/etc. A good book to study and get a handle on all aspects of Concurrent Programming is Foundations of Multithreaded, Parallel, and Distributed Programming by Gregory Andrews - https://www2.cs.arizona.edu/~greg/mpdbook/ https://www2.cs.arizona.edu/~greg/mpdbook/
    • nocman 2y ago
      In fairness the person you were responding to was referring to their own personal experience. They certainly are not the first person to conclude that doing non-trivial concurrent programming is too difficult for them. I agree that it is achievable with an appropriate level of care and experience, but I know there are many very smart people that conclude that multithreaded programming in C++ is too difficult for their taste. Even Rich Hickey, when discussing concurrency in Java/C#/C++ said "I am tired of trying to get right, because it is simply far too difficult." > In particular, talk about shared state, how we do it today, how you do it in C#, Java, or C++, what happens when we get into a multi-threaded context, and specifically what are some of the current solutions for that in those spaces. Locking, in particular. That is something I have done a lot of, over a long period of time, and I am tired of trying to get right, because it is simply far too difficult. ( https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/ClojureConcurrency.md https://github.com/matthiasn/talk-transcripts/blob/master/Hi... )
      • neonsunset 2y ago
        > Java/C#/C++ These three have different memory models, concurrency primitives and adopted practices...it's odd that they are lobbed together.
• itronitron 2y ago
  what is a systems programmer?
  • asgeir 2y ago
    It often refers to programmers that develop things that other programmers use to construct applications. A systems programmer may work on compilers, operating systems, framework libraries, etc. They will also often work with lower-level abstractions, such as threads, so that the higher-level developers need not think about them in their own designs.
    • covofeee 2y ago
      Fuck ... according to that I am a systems programmer. Not the OS/Compiler kind!
      • sethammons 2y ago
        Realized I was a systems developer some 15 years ago and underpaid. Negotiated a 33% raise
  • montalbano 2y ago
    My definition: A person who writes software that is closely bound to the operating system interface and/or hardware for reasons of performance and/or functionality.
    • lacedeconstruct 2y ago
      I think its more of a person who builds "systems" for other programmers to use and work within
      • montalbano 2y ago
        That's a valid broader definition but in practice I find it too broad to be very useful.
• woile 2y ago
  Is there a maybe ebook friendly version? I'd love to read this in the kindle
  • BrentOzar 2y ago
    Can’t you send PDFs to a Kindle? Been a decade since I used one, but I vaguely remember that being a thing.
    • woile 2y ago
      Yes you can, but it's not the best format to read on, and a PDF with 2 columns is kind of even harder to read.
      • utopcell 2y ago
        Pdfs are perfect on the kindle scribe, so long as they are not color. This is my primary use for the device. On smaller kindles, pdfs are probably a pain to read though.
  • bananatype 2y ago
    The ebook has a Github repo, here: https://github.com/mrkline/concurrency-primer https://github.com/mrkline/concurrency-primer You can fork/clone/download the repo and then convert the ebook into Kindle or epub with Pandoc, like so (this is what I run): pandoc -o concurrency-primer.epub concurrency-primer.tex (I can share my converted file, but I don't know which file-sharing service is permitted on HN. If anyone can give me suggestions on this, I will be happy to upload it and share my epub file.)
    • woile 2y ago
      Thanks, yes, I ended up doing the same!
• spintin 2y ago
  [dead]
• lincpa 2y ago
  [dead]
• msanlop 2y ago
  Spent 4h yesterday looking for a bug in my atomics, and probably 4 more today :^)
  • david-gpu 2y ago
    Been there. I don't have advice, only commiseration.
  • sebtron 2y ago
    Last time this happened to me it turned out I made a logic error, the algorithm required a lock but I erroneously thought atomics would do.
  • tialaramex 2y ago
    Several other people have commiserated, so rather than do the same maybe we can help. Are you quite sure you understand the atomic primitives you're working with? Mara's book https://marabos.nl/atomics/ https://marabos.nl/atomics/ is pretty good, it's written in Rust but like, if you thought you needed a Memory Ordering that doesn't cover you're boned because your tools don't support that order (and probably in an appendix or footnote somewhere they admit that) either. Reach for tooling to diagnose ordering problems. Loom and Shuttle are where I'd start in Rust, hopefully you have equivalents in any language you're using. Those are: A thorough tool which tries every possible correct interleaving and tells you what happens, and, a probabilistic approach which just picks at random and figures on average it should find the problem cases (but we can't prove it does). When in doubt, steal: There is often a published, correct, algorithm for whatever you're doing. It might take an hour to convince yourself that algorithm is correct, but hey, you're planning to spend four hours on your algorithm which isn't correct, that's a bad trade. Rubber Ducks are invaluable. If you feel awkward addressing an inanimate object I don't recommend using pets or livestock as both have no reason to be attentive, try a human who likes you. The important property of the rubber duck is not that they understand what you mean, after all a plastic bath toy can't understand anything, but that you will feel the need to make whatever you're saying make sense and in doing so may uncover a place where your algorithm is inconsistent.
• zelphirkalt 2y ago
  Hm. Seems very C++ oriented. Talks about specific keywords of C++ and their semantics. Not even mentioned "actor" in the whole document. "immutable" not found. Seems rather like an very incomplete view of a C++ programmer, wanting to raise it to "systems programmer".
  • ajross 2y ago
    To be fair most of the consensus-imposed[1] interface to memory ordering semantics was developed as part of the C++ memory model standardization process. GCC/clang have imported them as extensions for C code, etc... [1] Which is not to say "best". My standard complaint in these threads is that the C++ ordering semantics are a mess designed to make things easier for compiler writers to target architectures with variant ordering guarantees and not for algorithm developers to create correct code. But lockless code is much harder to write than compilers! It's a bad trade. ARM had this right decades ago with simple read/write barriers, which are vastly easier to understand when reading code.
    • packetlost 2y ago
      Rust also adopted the C/C++ memory ordering semantics, for better or worse.
      • ajross 2y ago
        Everyone has, even the hardware vendors are treating these rules as the golden standard to which they can optimize and conform. So it's too late. But it's still a shame. The acquire/release semantics make perfect sense for locks, because that's what it was designed for. Designing anything else with them is extremely difficult, and in practice such work tends to lean heavily on the full-barrier "sequentially consistent" primitive instead. The older read/write barrier instructions were much simpler. (And x86 is simpler still, with just one reordering rule to worry about and a single idea of serializing instructions to regulate it).
        • gpderetta 2y ago
• 9999_points 2y ago
  Speaking of concurrency, whatever happened to that Parallella "super computer" project?
  • codetrotter 2y ago
    > whatever happened to that Parallella "super computer" project? I had a couple of those. (Well, still do – but not using them for anything now. Mine are the ones with 16 cores, that I got for backing them on Kickstarter when they did their original crowd funding) And I was wondering the same a while ago. This guy that was part of Adapteva, makers of Parallella, is working on some ASIC stuff now it looks like. https://github.com/aolofsson https://github.com/aolofsson https://scholar.google.com/citations?user=qdDUg48AAAAJ&hl=en https://scholar.google.com/citations?user=qdDUg48AAAAJ&hl=en And since Adapteva website also says: “Adapteva is now Zero ASIC”, I guess maybe some other people that were originally doing the Parallella thing are now doing ASIC things with that guy too. https://www.adapteva.com/ https://www.adapteva.com/ The Parallella website is still up. https://parallella.org/ https://parallella.org/ And you can download the old Ubuntu images for the Parallela SBC from the GitHub releases. But they stopped adding anything new to it years ago. https://github.com/parallella/parabuntu/releases https://github.com/parallella/parabuntu/releases
• wslh 2y ago
  Having worked with some of the concurrency issues mentioned and observed both failures and successes, I’d like to share a perspective: developers who struggle most with concurrency often lack a formal Computer Science background. This isn’t to say that a CS degree is the only path to competence, but it provides structured exposure to concepts like concurrency through coursework, exams, and practical projects. These academic challenges can help solidify an understanding that’s harder to acquire in a fragmentary way
  • _kb 2y ago
    I think that applies in general. A formal education, in any domain, forces you to explore areas of knowledge that may not be directly applicable to immediate 'real world' problems. It does however give you the background to identify and explore those areas when you approach something that does overlap with work that people have spent literal lifetimes considering ahead of you. I say this as someone without a formal CS education but has been working with computers for some time. I've spent countless hours, days, and months playing catch-up on that knowledge when I stumble into those blindspots.
  • rramadass 2y ago
    While a formal Computer Science background is helpful, i don't think it is necessary for learning Concurrency. The reason people struggle with Concurrency is because they are not taught properly and also of course many are not willing to study and try and figure it out for themselves. Once you start thinking about a program as a sequence of loads/stores (i.e. reads/writes to shared memory) and note that Pipelining/OOO/Superscalar are techniques to parallelize these (and other) instructions for a single thread of control you start getting an idea of how sequential order can be preserved though the actual execution is not quite so. Now add in another thread of control on another core and you get the idea of Memory Consistency problems and can be introduced to various "Consistency Models" (https://en.wikipedia.org/wiki/Consistency_model https://en.wikipedia.org/wiki/Consistency_model). Next introduce caches in between and you get "Cache Coherence" problems (https://en.wikipedia.org/wiki/Cache_coherence https://en.wikipedia.org/wiki/Cache_coherence) Walk through a simple example like "c=a+b" in thread1 and "z=x+c" in thread2 and observe the interleavings and possible problems. Now move on to "Atomic Instructions" (https://en.wikipedia.org/wiki/Linearizability#Primitive_atomic_instructions https://en.wikipedia.org/wiki/Linearizability#Primitive_atom...) and how they can be used to implement "Locking" (https://en.wikipedia.org/wiki/Lock_(computer_science) https://en.wikipedia.org/wiki/Lock_(computer_science) ). Finally show "Memory Barriers" (https://en.wikipedia.org/wiki/Memory_barrier https://en.wikipedia.org/wiki/Memory_barrier) as a means of enforcing a checkpoint on memory operations. All these together should make an understanding of Concurrency clearer. One can then jump back up into the language and study the language primitives for concurrency management and their common usage patterns.
• w10-1 2y ago
  This 2020 review carefully summarizes the overly complex and error-prone outcome of programming language memory model changes in the early 2010's. It's tempting to master such tricky details, but doing so almost inevitably leads to the costliest bugs ever: rare data corruption that's impossible to detect and reproduce. Russ Cox in 2021 offered a higher-level insightful history of evolving hardware and software support for concurrency, discussed just yesterday https://news.ycombinator.com/item?id=42397737 His take (as I take it): systems programmers should only use sequentially-consistent atomics (and only via well-tested concurrent data structures). Easy peasy! ;p
  • tialaramex 2y ago
    I am inclined to agree with the opinion that Sequentially Consistent Atomics are another of the long list of wrong defaults chosen by WG21 (the C++ committee). The argument goes roughly like this: If you don't offer a default here, the programmer who supposes that the answer to their question is atomics will be obliged next to figure out which Order they wanted. Or they could give up and say "I need a grown-up". Because C++ chooses to offer Sequentially Consistent as the default that programmer will presume this must mean it now does what they naively anticipated. Russ' overview might suggest that's true too. Here's the problem: It may not do what they wanted, it may be that no ordering rule would achieve their actual goals, but since they weren't asked to go read the ordering rules and pick one, they never confronted this awful problem, and are instead blissfully unaware. They are strolling into traffic knowing that they've picked the best possible road crossing rule and not aware that in fact no possible rule could make crossing this Interstate on foot safe to do. Here's another bad problem: If it does achieve what they wanted, it's probably overkill. The only reason programmers should reach for atomics is that the provided facilities which were safe for them to use didn't deliver the performance they need. But the Sequentially Consistent ordering may hurt their performance, injecting barriers they don't need, using slower instructions or even stalling when that was unnecessary. If they'd read the rules they might for example have realised they only needed a Relaxed ordering - much better, or that with a slight tweak they can use Acquire/Release.