19 ms·
“Is Parallel Programming Hard, and, If So, What Can You Do About It?” v2 Is Out
- person_of_color 6y agoDisappointing that the author joined FB.
- capableweb 6y agoIndeed. It's hard to imagine people who could work basically anywhere in the world chooses to work for the company that has demonstrated they only care about money, don't care about its users and actively do more harm than good. I could understand people who started working for Facebook 10 years ago, before we knew the beast. But now, just a year ago start working for Facebook? I really wish we (engineers working in the software industry) could be more ethical inclined and stop caring about money so much.
- gumby 6y ago> actively do more harm than good I don’t believe this statement is axiomatic. There are literally billions of people who are happy to use it and get happiness from it. Yes it is also, fundamentally, spyware. I think the answer is more nuanced and I struggle to untease it. Consider that everyone who owns an internal combustion vehicle is knowingly implicit in poisoning the atmosphere. Most of them only care about money and don’t care about the atmosphere. Are they actively doing more harm than good? (FB is a large corp and doesn’t need defense from little old me. I’m pushing back though at the absolutism of your judgement)
- dvfjsdhgfv 6y agoWe don't know the circumstances. Maybe he didn't have that many options, and needed money badly? It's easy to criticize people for their choices, but I always wonder what the critics would actually do in exactly the same circumstances. I personally am not so sure I can always justify my life choices as 100% ethically pure.
- rangoon626 6y agoVery easy to write this if you’ve already made it. Plenty of people haven’t, however. And while Facebook might not be the nicest place doing the best stuff for the world, we should not be disparaging developers for trying to better their situation unless you’re gonna go ahead and disparage every developer who ever worked at Facebook too.
- slibhb 6y agoYou're begging the question. Not everyone shares your "facebook is evil" perspective yet your post assumes that we all agree about that.
- capableweb 6y agoYeah, sorry, I do assume people have a base level of ethics and moral code :) If you're not part of the group, I understand my comment seems foreign.
- fsociety 6y agoSo instead of having a complex ethical and moral framework, you turn to dogma.
- slibhb 6y agoPeople do not agree on ethical and moral questions. The assertion that everyone who disagrees with you is objectively wrong is childish and you should work on overcoming that :)
- matz1 6y agoWhat would be reason to assume people have a the same ethics and moral code ?
- bawolff 6y agoIs your base moral code to do what is popular? Because lets be real, this thread would not appear for any of the other companies that do the exact same thing but aren't currently in vouge to hate on.
- kmonsen 6y agoDo you eat meat? Do you were clothes created in sweatshop? Do you use any cobalt mined in Congo? I believe all of those are far more evil than Facebook. Any TV you buy, any webpage you go to (not literally any, but more or less) sells your data as well. I understand that you, or most, might disagree with me. I do believe that is a matter of opinion.
- mdpopescu 6y agoYeah, that's how I look at people who work for the gangs that killed 200 million people last century, and continue to do most of the killing even today. Most people call them governments and they find nothing wrong with working for them. People have different worldviews, news at 11.
- 3JPLW 6y agoA thread from 2020 (v2 RC 1): https://news.ycombinator.com/item?id=22030928 https://news.ycombinator.com/item?id=22030928 2015: https://news.ycombinator.com/item?id=9315152 https://news.ycombinator.com/item?id=9315152 2014: https://news.ycombinator.com/item?id=7381877 https://news.ycombinator.com/item?id=7381877 2011: https://news.ycombinator.com/item?id=2784515 https://news.ycombinator.com/item?id=2784515
- dragontamer 6y agoCounterpoint: Parallel programming isn't very hard. Seriously: try fork() + wait(), or spawn in bash / cmd line. make -j offers parallelism (and Ninja is "make -j by default"). OpenMP's "#pragma omp parallel for" is extremely easy to use. Things get harder with thread libraries, but the "parallelism" part of pthreads is pretty easy. pthread_create(&thread_id, NULL, function, args);, at least if you're fine with the default settings. C++ Threads are even easier than pthreads. ----------- The hard part is: 1. Communication / Synchronization -- But really, condition variables are the most complicated thing most people need. fork() + wait() push the difficult communication part to the wait() or to pipes. But if you stick with wait() / waitpid() as your main communication mechanism, its pretty easy to think about. 2. High performance programming -- This is just hard, especially if you begin to reach for the highest performance "memory barrier / lock-free programming" styles. This is where "false sharing" comes in (your code correctly executes under false sharing, but slowly due to how the L1 cache handles multicore architectures). --------- However, if you stick with a simple fork-join communication model (ex: pthread_join, or wait()/waitpid() based synchronization)... its really not too hard at all. Just try to stay away from the high-performance techniques unless you really need them.
- volta83 6y ago- Knock Knock * Who's there? - GPUs. * Closes door. - Knock knock - Knock knock - Do you really want to ignore ~90% of the compute throughput and ~90% of the memory bandwidth of your machine? - Knock knock - ... ---- An AMD EPYC 7742 (Milan, Zen 3, announced last week, high-end server model) has ~100 GB/s DRAM bandwidth and 1-2 TFLOPS double precision. A single PCI-e A100 has ~2Tb/s DRAM bandwidth, and 10-20 TFLOPs double precision. You can pack 4x of these per CPU socket... so you end up at 8 TB/s and 40-80 TFLOPs... That makes the GPUs like 99% of both the compute and memory BW of a socket.
- dragontamer 6y agoWell, CPUs vs GPUs is pretty interesting actually. The #1 supercomputer right now is CPU-only, the A64Fx by Fujitsu. But you're right in that GPU-parallelism is the obvious parallel architecture available to modern programmers. For anyone spending less than $500,000 on a computer, GPUs are probably the most compute for the least amount of cost. --------- The issue is that there's two difference concepts here: 1. Parallelism -- When a problem you're working on has an undefined ordering to it explicitly. Protocols and communications for example have undefined orders. You could write a state-machine explicitly, but writing a thread is usually much simpler to understand. Ultimately: this is cofunctions, Python Yield / generators, async-code, etc. etc. Remember: Djikstra invented semaphores on a single-core processor. The concept of "single core parallelism" is hugely useful and can grossly simplify code if used correctly. 2. High performance -- This is a different concept entirely!! GPUs are within the realm of high-performance parallelism (and not just merely "conceptual parallelism" in #1). GPUs are hard because anyone who uses a GPU expects high performance. Be it the video gamer who wants raytracing on his video game, or the machine learning enthusiast training a billion neurons. Writing a simple "GPU Hello World" is easy as all heck actually. Its all the performance stuff that gets really hard.
- thorn 6y agoPlease, refrain from judging people here. I would rather prefer if we talked about this wonderful book that was released for free for our own enjoyment. I cannot even grasp the volume of work that went into this book.
- zozbot234 6y ago> I cannot even grasp the volume of work that went into this book. You actually can! The main format is PDF, but the author also has hard-copies for sale.
- deleted 6y ago[deleted]
- nickthemagicman 6y agoQuality pun.
- deleted 6y ago[deleted]
- lincpa 6y agoIt is very simple to use the parallel method of the manufacturing industry ---- Warehouse/Workshop Model. https://github.com/linpengcheng/PurefunctionPipelineDataflow#Warehouse-Workshop-Model https://github.com/linpengcheng/PurefunctionPipelineDataflow...
- Dowwie 6y agoMore free materials on concurrency, including a book near the top published in 2016: https://homes.cs.washington.edu/~djg/teachingMaterials/spac https://homes.cs.washington.edu/~djg/teachingMaterials/spac
- bullen 6y agoI think Joint Parallel is hard, as in when two threads on two different cores are trying to write+read to+from the same memory. My solutions are: 1) Use Java on the server; it's GC:ed VM has a complex memory model that can handle fast lock-free concurrency relatively well, even using OO though you'll hit cache-misses. 2) Use C on the client, primitive variable types are generally atomic so if you know what you are doing (I don't because I haven't gotten that far yet) you should be able to joint parallelize cores with char, int and float arrays as data, also avoiding cache-misses! There are many more arguments for this division, Java doesn't crash (usually) and C is truly portable (you cannot run Java on certain devices iOS, Switch, PS4).
- setpatchaddress 6y agoYou cannot generally trust that C primitive types are atomic.
- eschneider 6y agoCan't. You can't trust that unless the types are specifically declared atomic.
- CyberDildonics 6y agoThis seems more like someone taking wild guesses instead of information from experience and fundamentals. First, I don't think "Joint Parallel" is a common term and everything gets synced eventually at some level. Your solutions are languages, which are separate from understanding parallelism and concurrency. Using java doesn't solve anything directly. > it's GC:ed VM has a complex memory model that can handle fast lock-free concurrency relatively well, even using OO though you'll hit cache-misses. This is almost like talking about the sound of a color. Lock free programming is not a factor of a language. If there are good lock free data structures that work for your specific problems, then that can go a long way to solving synchronization. OO and cache misses have nothing to do with anything here, that comes down to how something is written, not the language. > Use C on the client, primitive variable types are generally atomic so if you know what you are doing This is not true. C has nothing to with this, it is a matter of getting access to atomics. Also primitive types are not "generally atomic", 32 bit stores can be written atomically, but that doesn't address ordering, compare and swap or atomic adds. > you should be able to joint parallelize cores with char, int and float arrays as data, also avoiding cache-misses! This seems like real information fed through google translate a few times. 'joint parallelize' doesn't mean anything. Avoiding cache misses is a matter of accessing memory sequentially. Arrays are orthogonal to atomics, each element needs to be dealt with individually.
- deleted 6y ago[deleted]
- belgesel 6y agoA question to parallel programming experts. Can we assume that a task that is well seperated to perform on single core is always better than multi-core? I meant using multiple instances working on single core vs. single instance with multi threads.
- aparsons 6y agoIf you flip some NUMA switches so tasks on the other core don’t add overhead, I’m inclined to say yes. But then modern schedulers are also so low-overhead that this case really only applies if you have a hard real-time requirement - in which case dedicated hardware is the norm
- jandrewrogers 6y agoData sharing between cores is the bane of parallel programming at every scale. Consequently, high-performance parallel systems are not multithreaded in any meaningful sense -- each thread operates almost exclusively on private local data. See also "thread-per-core" software architectures for maximizing absolute throughput, which has its origins in supercomputing. However, this creates a new problem for any workload that is not embarrassingly parallel. If all data is private to a single thread/core then the workload can become very unevenly distributed across cores depending on the data they own, destroying efficient parallelism in another way. Unlike multithreading, quick adaptive load shedding to smooth out these hotspots can scale surprisingly well across a very large number of cores/nodes with little overhead for many workloads. This is how many massively parallel codes are written today for workloads that are not embarrassingly parallel. Partitioning data across cores with out sharing is necessary to maximize throughput, and almost always better than multithreading, but insufficient. Fortunately, mitigating transient hotspots is a mostly solved problem.
- electricshampo1 6y agoFor this load shedding in a single node, are you imagining something more than work stealing style approaches? In a multi-node cooperative setting you need some way to transmit information that a given node is overloaded, some way to find nodes that have available capacity, and a low overhead way to shift the work over to them. If the work to be done depends solely on data that you have local to you, it seems silly to shift the data as well (depending on how big it is); this would only make sense when you have to combine data from a variety of nodes (which can be done on any other node, assuming that the current node is overloaded). Probably not worth doing anything about short lived hotspots (<1s). I wonder what kind of granularity you have used in your systems (probably different for within node vs across nodes).
- raspasov 6y agoAm I the only person that is surprised that this book does not mention immutability even once?
- jayd16 6y agoThey use the term read-only many times.
- deleted 6y ago[deleted]
- dr-detroit 6y agoPP isnt hard but kids these days think you just throw in a -flag or special keyword and thats what it does it makes it magically parallel.
- fearthetelomere 6y agoOne of the biggest issues I encounter with concurrency is whether the code I've written could be optimized to reduce cache misses. I feel like a lot of resources including this wonderful book gloss over reducing cache misses. Is it really that trivial? What are the tips and tricks involved?
- CyberDildonics 6y agoWhere is this a problem for you? The broad goal to minimize cache misses is the same as concurrency in general - minimize synchronization. This means understanding dependencies and doing work on each core in chunks that are as large as possible for the amount of latency you can tolerate between synchronizing.
- fearthetelomere 6y ago- "A high-performance implementation would of course use padding or special alignment directives to avoid false sharing." - "Cache alignment and padding often improves performance by reducing false sharing." - "Software can use the alignment directives available in many compilers to avoid false sharing, and adding such directives is a common step in tuning parallel software." There's just essentially two or three lines in the book that I could find that refer to optimizing cache use through padding/alignment, and that was after knowing what to search for. Although I wouldn't be surprised if I missed something. In my experience, the discourse stops at the surface level, which makes the topic appear like it's obvious or trivial. But there are many follow-up questions that naturally arise for me: - What are the trade-offs of cache-padding shared data? Why does it degrade performance for certain problems? - What is a good rule-of-thumb for when to prioritize cache padding/alignment over cache locality? - Are there other best practices like cache padding/alignment/locality that improve performance? - What is an alignment directive and how does one use it? I agree with what you've said, I was merely pointing out that I wish parallel programming resources delved more into this subject, as I feel that it's a practical and common issue. I'm sure it's not trivial and requires a fair bit of expertise, but that's why one would reach for a book like this after all.
- bob1029 6y agoThe hardest part of parallel programming is tricking yourself into believing you need to worry about it at all. 9/10 times, if you use the right tools & techniques, a single x86 core (assuming you have access to a 'real' one) can easily chew through millions of business transactions per second and easily satisfy all technical requirements for whatever business system. Latency is the ultimate devil. Getting your synchronous state to fit into L1/2 and processing transactions in (micro)batches of contiguous data is how you win the game. The closer your data is to your compute, the faster you can go. The biggest problem most developers are struggling with right now is that they got tricked into scattering their business systems across many computers and potentially across multiple physical regions. The latency in L1 is measured in nanoseconds. The latency between 2 datacenters is measured in milliseconds. This is SIX orders of magnitude difference. Even if you squander 3 of those to some bad programming practices, you are still 1000x better off on a single box using a single thread than if you scattered your infra to the seven winds. See: LMAX Disruptor for a case study in what the opposite end of the "infrastructure isnt my problem" spectrum looks like.
- VHRanger 6y agoA problem here is that people interpreted some common sense advice like SOLID into "break everything into classes pointing to each other". Which creates maximally cache hostile code very fast -- any operation is often several pointer dereferencing (eg. back-and-forth to RAM).
- flukus 6y agoEven in code like this just being aware of cache levels and there relative timings can sometimes produce some impressive speedups with simple things like sorting the initial data to make it cache friendly. A recent one I improved was along the lines of: for t in transactions t.client = clients[t.ClientId] t.product = products[t.ProductId] In this case sorting transactions by ClientId, ProductId was an orders of magnitude speedup because client and product were already in cache in L1/L2 cache most of the time. It could have been faster again with less SOLID but improvements like this are often easy and politically achievable because it keeps the sacrifice to the OO gods. Treating Big O as the be all of performance has quite a lot to answer for in real world code, often constants aren't so constant.
- Nasrudith 6y agoIt sounds like a dumb tautology but parallel programming is easy for parallel problems. It is harder to say maintain several interlinked grid cells in a simulation in parallel than it is to partion a non-interacting set to N cores to run in parallel and let the main thread know when everything is done.