3 ms·
You can get into that type of lock congestion trouble in any language. It's an algorithm problem, not a language problem. I discovered last year that Wine has
by Animats 2y ago
You can get into that type of lock congestion trouble in any language. It's an algorithm problem, not a language problem.
I discovered last year that Wine has terrible internal lock problems inside its user-side storage allocator. That's in C. If you have enough threads calling "realloc", the allocator goes into futex congestion collapse and performance drops by two orders of magnitude. My graphics program went from 60 FPS to 0.5 FPS. They optimized too hard for the no-congestion case.
This is a Wine-only problem; Microsoft's own code doesn't have this problem.
I've had lock congestion problems in Rust. Sometimes you need a fair mutex, or something gets frozen out. Both fair and non-fair mutexes are available; see the "parking_lot" crate.
There's a place inside WGPU that has a lock congestion problem in one of three locks, and I'm going to have to add more profiling to someone else's code to find that. I can see the problem with Tracy, but need to add more profiling scopes to narrow it down.
But that is high-performance graphics stuff, where microseconds count. Sending spam (OK, bulk marketing emails) doesn't need to be that tightly coupled. Mailing list removal runs on a timescale of days, not milliseconds. What else in that space has to be tightly interlocked?
- forrestthewoods 2y ago> I can see the problem with Tracy, but need to add more profiling scopes to narrow it down. If you can run on Windows try Superluminal. https://superluminal.eu/ https://superluminal.eu/
- mgaunard 2y agoA good language just gives you the necessary tooling to do whatever you want, it doesn't magically fix problems. Only languages like C++ have a memory model that allows you to do lock-free programming for example (C and Rust copied the C++ model). Also, what kind of serious person allocates memory from the system allocator in a real-time loop? Your problems seem self-inflicted. Regardless there are many allocators that optimize for concurrent allocations: tcmalloc, jemalloc, mimalloc...
- rowanG077 2y agoConsidering the vast number of programs that wine works extremely well with I'm not so sure they spent too much optimizing the no-congestion case. You are just doing something extremely quirky in your program.
- Animats 2y agoI've looked at the code in a debugger. Wine has futexes three deep in "malloc". The innermost one is a pure spinlock. The problem with "realloc" is that, when it can't grow an array in place, it has to copy the contents. The Wine implementation does that with the main lock on allocation still held. So, if you have Rust code with a lot of multithreaded vector "push" operations, and more threads than CPUs, you get futex congestion. It's possible to write applications that don't hit this bug, but it's Wine-only, not Windows, so not worth it. What's "quirky" is trying to use all the CPUs with lower priority threads.