4 ms·
I spent the first 10 years of my career thinking I wasn't smart enough to write multi-threaded C code. Since then I have been convinced that nobody is smart en
by aidenn0 4y ago
I spent the first 10 years of my career thinking I wasn't smart enough to write multi-threaded C code. Since then I have been convinced that nobody is smart enough to write multi-threaded C code, and multiprocessing should be used instead.
- jjgreen 4y agoTosh, a lot of use-cases need shared read-only access to a large structure, mark that const* and you can thread away without a mutex in sight, each thread has it own r/w struct to accumulate results.
- aidenn0 4y agoI agree, and with multiple processes that's just a single mmap() call away!
- CarVac 4y agoMultiprocessing as in openmp?
- yazaddaruvala 4y agoThe parent is using multi-processing to mean: “no shared memory concurrent processing” Which still isn’t quite good enough because files (especially mmaped files) can be abused as shared memory even between multiple processes. But the general point is good, “you’re better off with mutable xor shared memory while concurrently processing”.
- aidenn0 4y agoRight process aren't "shared nothing" but at least it makes you work a little bit before you can shoot yourself in the foot..
- aidenn0 4y agoSorry, the other reply explained what I meant. "Multiprocessing" has many meanings but specifically when contrasted with "multithreading" it usually implies the thread/process distinction in the Unix sense of the terms.
- deleted 4y ago[deleted]
- r-s 4y agoI came to a similar conclusion years ago. Yes, there are examples of applications that have done it, but they often have very elite engineers. Most teams are comprised of various levels of engineers and its extremely difficult to get right. I work more in web dev now and I suspect the majority of my team does not know the difference between a thread and a process. HN readers may scoff but these are productive engineers who have build some honestly pretty impressive apps and systems.
- cozzyd 4y agoAs always, it depends on what the interfaces between threads are. If they're just workers, it's easy to get right, or if you have producer-consumer queues. If everything can be shared then... yeah, good luck.
- aidenn0 4y agoThe problem with C (and many other languages too) is that with threads, the default is shared. A C-like language where all statically allocated values default to thread-local is already a big improvement. Now lets make it so file-descriptors can only be used by the thread that opened them and give each thread its own heap. Wow, this is starting to look a lot like a Unix process...
- spacechild1 4y agoI am not exceptionally smart and I am certainly able to write (and debug) multi-threaded C code, but it is painful :-) I try to use C++ instead whenever I can. BTW, "multiprocessing" is not a drop-in replacement for multithreading. In fact, there are many domains where "multiprocessing" just isn't possible or practical. So the real solution is to use another language.
- deleted 4y ago[deleted]
- pjmlp 4y agoMultiprocessing won't help you when accessing shared resources.
- gpderetta 4y agoRight. And most importantly you can build (even inadvertently) the equivalent of shared memory on top of multiprocessing and message passing so you always have to be careful.
- aidenn0 4y agoTrue, but in C, everything that isn't automatic storage duration defaults to being a shared resource with multithreading. I'm not saying that multiprocessing is enough to avoid all footguns, just that it eliminates a lot of them.
- Cloudef 4y agoThis, the moment you call into library or code you haven't written yourself, you can throw out all the guarantee your program will be thread safe, and I don't even want to go into details like program mixing forks and threads or unix signals... Multiprocessing has other benefits too, like making your code simpler, fault tolerant (you can restart failed processess, etc...) and you can make sure OS is handling all resource cleanup. I've seen many multithreaded monsters that would've been better as multi-process architecture instead, and maybe even more faster / safer too, this is usually how actor based languages like erlang is implemented, behind the scenes everything is actually a separate process. If you need to work with shared structure in threads, keep that part very small. Or consider if GPU or SIMD would be a better fit.
- pornel 4y agoThis isn't true in Rust. Thread safety checks are propagated all they way through libraries and 3rd party code. If it compiles, it's thread safe. If it isn't thread safe, Rust will show you exactly where. It is as amazing and magical as it sounds.
- Cloudef 4y agoTrue, but the above topic was not about rust. Though even in rust, if the code ever calls into C code the same pitfalls still apply. Also in rust because how everything has to take into account thread safety, programming simple or low-level stuff in it is frustrating experience. See old discussion here: https://news.ycombinator.com/item?id=33590864 https://news.ycombinator.com/item?id=33590864
- pornel 4y agoFor set-once global variables there's OnceCell. I understand it's super frustrating if you don't know it exists, and DIYing it from first principles requires knowledge of both Rust's low-level primitives as well as OS/multithreading techniques. OTOH it is an example how thread-safety extends to everything in Rust, and you can't be surprised by some function somewhere unsafely mutating global state.
- hedora 4y agoI’ve come out the other side of the tunnel with the conclusion that everyone should write lock free code, because it is too hard to reason about the performance of mutexs, let alone the sequential semantics of code that is too sloppy to be made lock free. Edit: Do you actually mean multiprocess programs? Those are the ones that communicate via lock free data structures stored in shared memory, with the invariant that processes can be kill -9’ed or segfault without leaving the shared memory in an inconsistent state. That’s like switching your compiler into nightmare mode. It’s technically possible, but first you need to master piles of obscure data structure research.