4 ms·

> Rewrite your code in rust, and get something better than the go race detector every time you compile. Congrats, rustc forced you to wrap all your types in Ar

by klabb3 1y ago

> Rewrite your code in rust, and get something better than the go race detector every time you compile. Congrats, rustc forced you to wrap all your types in Arc<Mutex<_>>, and you no longer have data races. As a gift, you will get logical race conditions instead, that are even more difficult to detect, while being equally difficult to reproduce reliably in unit tests and patch. Don’t get me wrong, Rust has done a ton for safety and pushed other languages to do better. I love probably 50% of Rust. But Rust doesn’t protect against logical races, lovelocks, deadlocks, and so on. To write concurrent programs that have the same standards of testable, composable, expressive etc as we are expecting with sequential programs is really really difficult. Either we need new languages, frameworks or (best case) design- and architectural patterns that are easy to apply. As far as I’m concerned large scale general purpose concurrent software development is an unsolved problem.

catigula 1y ago
If it's solved the solution has been discarded at some point by other developers for being too cumbersome, too much effort, and therefore in violation of some sacred principle of their job needing to be effortless.
ViewTrick1002 1y ago
A well formed Go program would have the same logical race conditions to manage as well. The Arc is only needed when you truly need to mutably share data. Rust like Go has the full suite of different channels and what other patterns to share data.
- jason_oster 1y ago
  Small correction: The Arc is for sharing across threads, the Mutex is for mutation. But you are generally correct that they can be used independently.
  - ViewTrick1002 1y ago
    Of course. But if you’re using a channel then it hides the inner constructs. Comparing writing a web service in Go and rust you would likely also utilize Tokio which has a wide variety of well designed sync primitives. https://docs.rs/tokio/latest/tokio/sync/index.html https://docs.rs/tokio/latest/tokio/sync/index.html
mr90210 1y ago
> Congrats, rustc forced you to wrap all your types in Arc<Mutex<_>> Also, don’t people know that a Mutex implies lower throughput depending on how long said Mutex is held? Lock-free data structures/algorithms are attempt to address the drawbacks of Mutexes. https://en.wikipedia.org/wiki/Lock_(computer_science)#Disadvantages https://en.wikipedia.org/wiki/Lock_(computer_science)#Disadv...
- speed_spread 1y ago
  The overhead of Mutex for uncontended cases is negligible. If Mutex acquisition starts to measurably limit your production performance, you have options but will probably need to reconsider the use of shared mutable anyway.
- valyala 1y ago
  Lock-free data structures and algorithms access shared memory via various atomic operations such as compare-and-swap and atomic arithmetic. The throughout of these operations do not scale with the number of CPU cores. Contrary, the throughput usually reduces with the growing number of CPU cores because they need more time for synchronizing local per-CPU caches with the main memory. So, lock-free data structures and algorithms do not scale on systems with big number of CPU cores. It is preferred to use "shared nothing" data structures and algorithms instead, where every CPU core processes its own portion of state, which isn't shared among other CPU cores. In this case the local state can be processed from local per-CPU caches at the speed which exceeds the main memory read/write bandwidth and has smaller access latency.
  - gpderetta 1y ago
    High write contention on a memory location do not scale with the number of cores (in fact it is bad even with two cores). This is independent of using a mutex, a lock free algorithm or message passing (because at the end of the day a queue is a memory location).
- johncolanduoni 1y ago
  Lock-free and even wait-free approaches are not a panacea. Memory contention is fundamentally expensive with today’s CPU architectures (they lock, even if you ostensibly don’t). High contention lock-free structures routinely perform worse than serialized locking.
CodeBrad 1y ago
I may be biased, as I definitely love more than 50% of Rust, but Go also does not protect against logical races, deadlocks, etc. I have heard positive things about the loom crate[1] for detecting races in general, but I have not used it much myself. But in general I agree, writing correct (and readable) concurrent and/or parallel programs is hard. No language has "solved" the problem completely. [1]: https://crates.io/crates/loom https://crates.io/crates/loom
TheDong 1y ago
As a sibling said, Go has all the same deadlocks, livelocks, etc you point out that rust doesn't cover, in addition to also having data-races that rust would prevent. But, also, Go has way worse semantics around various things, like mutexes, making it much more likely deadlocks happen. Like in go, you see all sorts of "mu.Lock(); f(); mu.Unlock()" type code, where if it's called inside an `http.Handler` and 'f' panics, the program's deadlocked forever. In go, panics are the expected way for an http middleware to abort the server ("panic(http.ErrAbortHandler)"). In rust, panics are expected to actually be fatal. Rust's mutexes also gate "ownership" of the inner object, which make a lot of trivial deadlocks compiler errors, while go makes it absolutely trivial to forget a "mu.Unlock" in a specific codepath and call 'Lock' twice in a case rust's ownership rules would have caught. In practice, for similarly sized codebases and similarly experienced engineers, I see only a tiny fraction of deadlocks in concurrent rust code when compared to concurrent go code, so like regardless that it's an "unsolved problem", it's clear that in reality, there's something that's at least sorta working.
- Xeoncross 1y ago
  > and 'f' panics, the program's deadlocked forever I don't see `mu.Lock(); f(); mu.Unlock()` anywhere really. `mu.Lock(); defer mu.Unlock(); f();` is how everyone does it to prevent that possibility.
  - ViewTrick1002 1y ago
    Until you have to call a slow function after the mutex access leading to the lock being held long enough to cause problems. Now you either refactor into multiple functions, while ensuring all copies of possibly shared data when passing function arguments are correctly guarded or ”manually” unlock when you don’t need the mutex access anymore.
    - jerf 1y ago
      OK, but you're not in "Go"-specific problems any more, that's just concurrency issues. There isn't any approach to concurrency that will rigorously prevent programmers from writing code that doesn't progress sufficiently, not even going to the extremes of Erlang or Haskell. Even when there are no locks qua locks to be seen in the system at all I've written code that starved the system for resources by doing things like trying to route too much stuff through one Erlang process.
empath75 1y ago
> Congrats, rustc forced you to wrap all your types in Arc<Mutex<_>>, and you no longer have data races. Or you can just avoid shared mutable state, or use channels, or many of the other patterns for avoiding data races in Rust. The fun thing is that you can be sure that no matter what you do, as long as it's not unsafe, it will not cause a data race.
pkolaczk 1y ago
I wrote plenty of concurrent Rust code and the number of times I had to use Arc<Mutex> is extremely low (maybe a few times per thousands lines). As for your statement that concurrency is generally hard - yes it is. But it is even harder with data races.

by klabb3 1y ago

catigula 1y ago

If it's solved the solution has been discarded at some point by other developers for being too cumbersome, too much effort, and therefore in violation of some sacred principle of their job needing to be effortless.

ViewTrick1002 1y ago

A well formed Go program would have the same logical race conditions to manage as well. The Arc is only needed when you truly need to mutably share data. Rust like Go has the full suite of different channels and what other patterns to share data.

jason_oster 1y ago
Small correction: The Arc is for sharing across threads, the Mutex is for mutation. But you are generally correct that they can be used independently.
- ViewTrick1002 1y ago
  Of course. But if you’re using a channel then it hides the inner constructs. Comparing writing a web service in Go and rust you would likely also utilize Tokio which has a wide variety of well designed sync primitives. https://docs.rs/tokio/latest/tokio/sync/index.html https://docs.rs/tokio/latest/tokio/sync/index.html

mr90210 1y ago

> Congrats, rustc forced you to wrap all your types in Arc<Mutex<_>> Also, don’t people know that a Mutex implies lower throughput depending on how long said Mutex is held? Lock-free data structures/algorithms are attempt to address the drawbacks of Mutexes. https://en.wikipedia.org/wiki/Lock_(computer_science)#Disadvantages https://en.wikipedia.org/wiki/Lock_(computer_science)#Disadv...

speed_spread 1y ago
The overhead of Mutex for uncontended cases is negligible. If Mutex acquisition starts to measurably limit your production performance, you have options but will probably need to reconsider the use of shared mutable anyway.
valyala 1y ago
Lock-free data structures and algorithms access shared memory via various atomic operations such as compare-and-swap and atomic arithmetic. The throughout of these operations do not scale with the number of CPU cores. Contrary, the throughput usually reduces with the growing number of CPU cores because they need more time for synchronizing local per-CPU caches with the main memory. So, lock-free data structures and algorithms do not scale on systems with big number of CPU cores. It is preferred to use "shared nothing" data structures and algorithms instead, where every CPU core processes its own portion of state, which isn't shared among other CPU cores. In this case the local state can be processed from local per-CPU caches at the speed which exceeds the main memory read/write bandwidth and has smaller access latency.
- gpderetta 1y ago
  High write contention on a memory location do not scale with the number of cores (in fact it is bad even with two cores). This is independent of using a mutex, a lock free algorithm or message passing (because at the end of the day a queue is a memory location).
johncolanduoni 1y ago
Lock-free and even wait-free approaches are not a panacea. Memory contention is fundamentally expensive with today’s CPU architectures (they lock, even if you ostensibly don’t). High contention lock-free structures routinely perform worse than serialized locking.

CodeBrad 1y ago

I may be biased, as I definitely love more than 50% of Rust, but Go also does not protect against logical races, deadlocks, etc. I have heard positive things about the loom crate[1] for detecting races in general, but I have not used it much myself. But in general I agree, writing correct (and readable) concurrent and/or parallel programs is hard. No language has "solved" the problem completely. [1]: https://crates.io/crates/loom https://crates.io/crates/loom

TheDong 1y ago

As a sibling said, Go has all the same deadlocks, livelocks, etc you point out that rust doesn't cover, in addition to also having data-races that rust would prevent. But, also, Go has way worse semantics around various things, like mutexes, making it much more likely deadlocks happen. Like in go, you see all sorts of "mu.Lock(); f(); mu.Unlock()" type code, where if it's called inside an `http.Handler` and 'f' panics, the program's deadlocked forever. In go, panics are the expected way for an http middleware to abort the server ("panic(http.ErrAbortHandler)"). In rust, panics are expected to actually be fatal. Rust's mutexes also gate "ownership" of the inner object, which make a lot of trivial deadlocks compiler errors, while go makes it absolutely trivial to forget a "mu.Unlock" in a specific codepath and call 'Lock' twice in a case rust's ownership rules would have caught. In practice, for similarly sized codebases and similarly experienced engineers, I see only a tiny fraction of deadlocks in concurrent rust code when compared to concurrent go code, so like regardless that it's an "unsolved problem", it's clear that in reality, there's something that's at least sorta working.

Xeoncross 1y ago
> and 'f' panics, the program's deadlocked forever I don't see `mu.Lock(); f(); mu.Unlock()` anywhere really. `mu.Lock(); defer mu.Unlock(); f();` is how everyone does it to prevent that possibility.
- ViewTrick1002 1y ago
  Until you have to call a slow function after the mutex access leading to the lock being held long enough to cause problems. Now you either refactor into multiple functions, while ensuring all copies of possibly shared data when passing function arguments are correctly guarded or ”manually” unlock when you don’t need the mutex access anymore.
  - jerf 1y ago
    OK, but you're not in "Go"-specific problems any more, that's just concurrency issues. There isn't any approach to concurrency that will rigorously prevent programmers from writing code that doesn't progress sufficiently, not even going to the extremes of Erlang or Haskell. Even when there are no locks qua locks to be seen in the system at all I've written code that starved the system for resources by doing things like trying to route too much stuff through one Erlang process.

empath75 1y ago

> Congrats, rustc forced you to wrap all your types in Arc<Mutex<_>>, and you no longer have data races. Or you can just avoid shared mutable state, or use channels, or many of the other patterns for avoiding data races in Rust. The fun thing is that you can be sure that no matter what you do, as long as it's not unsafe, it will not cause a data race.

pkolaczk 1y ago

I wrote plenty of concurrent Rust code and the number of times I had to use Arc<Mutex> is extremely low (maybe a few times per thousands lines). As for your statement that concurrency is generally hard - yes it is. But it is even harder with data races.