4 ms·

The mistake was using golang apparently. If you need high-performance and fine control of synchronization, just use a low-level systems programming language.

Animats 2y ago
You can get into that type of lock congestion trouble in any language. It's an algorithm problem, not a language problem. I discovered last year that Wine has terrible internal lock problems inside its user-side storage allocator. That's in C. If you have enough threads calling "realloc", the allocator goes into futex congestion collapse and performance drops by two orders of magnitude. My graphics program went from 60 FPS to 0.5 FPS. They optimized too hard for the no-congestion case. This is a Wine-only problem; Microsoft's own code doesn't have this problem. I've had lock congestion problems in Rust. Sometimes you need a fair mutex, or something gets frozen out. Both fair and non-fair mutexes are available; see the "parking_lot" crate. There's a place inside WGPU that has a lock congestion problem in one of three locks, and I'm going to have to add more profiling to someone else's code to find that. I can see the problem with Tracy, but need to add more profiling scopes to narrow it down. But that is high-performance graphics stuff, where microseconds count. Sending spam (OK, bulk marketing emails) doesn't need to be that tightly coupled. Mailing list removal runs on a timescale of days, not milliseconds. What else in that space has to be tightly interlocked?
- forrestthewoods 2y ago
  > I can see the problem with Tracy, but need to add more profiling scopes to narrow it down. If you can run on Windows try Superluminal. https://superluminal.eu/ https://superluminal.eu/
- mgaunard 2y ago
  A good language just gives you the necessary tooling to do whatever you want, it doesn't magically fix problems. Only languages like C++ have a memory model that allows you to do lock-free programming for example (C and Rust copied the C++ model). Also, what kind of serious person allocates memory from the system allocator in a real-time loop? Your problems seem self-inflicted. Regardless there are many allocators that optimize for concurrent allocations: tcmalloc, jemalloc, mimalloc...
  - samatman 2y ago
    Go has atomics.
    - mgaunard 2y ago
      Which don't have the same level of control as C++, it only provides a subset of sequentially consistent primitives, which are slow by design.
- rowanG077 2y ago
  Considering the vast number of programs that wine works extremely well with I'm not so sure they spent too much optimizing the no-congestion case. You are just doing something extremely quirky in your program.
  - Animats 2y ago
    I've looked at the code in a debugger. Wine has futexes three deep in "malloc". The innermost one is a pure spinlock. The problem with "realloc" is that, when it can't grow an array in place, it has to copy the contents. The Wine implementation does that with the main lock on allocation still held. So, if you have Rust code with a lot of multithreaded vector "push" operations, and more threads than CPUs, you get futex congestion. It's possible to write applications that don't hit this bug, but it's Wine-only, not Windows, so not worth it. What's "quirky" is trying to use all the CPUs with lower priority threads.
    - mgaunard 2y ago
      Why not send a patch?
      - Animats 2y ago
        It's hard to fix without a redesign of some crucial low-level code. The people who wrote it looked at the problem and decided it was too hard to fix.
faitswulff 2y ago
I suspect that when it was chosen at Mailgun, Golang was still being billed as a systems programming language.
- neonsunset 2y ago
  Exactly. Go makes you think you control the details except you don't. Hackernews makes you think you don't control the details in C# except you do. Yet another project that would have been able to solve its woes if it had picked a better option.
bee_rider 2y ago
Does the language make a huge difference here? In a distributed system a signal to be sent over the network travels at the same speed whether it was transmitted by a C or Python program, right?
jeffbee 2y ago
It would take you 1 minute to write a Go application that exploits per-CPU data structures and probably years to debug your attempt to do so in C.
- convolvatron 2y ago
  actually its really _hard_ in go to make cpu bound control flow, state, and allocation. do goroutines have any notion of locality? i've been looking and haven't been able to find anything
  - jeffbee 2y ago
    Go comes out of the box with sync.Pool
  - dilyevsky 2y ago
    You can pin the go routine to a current thread using[0] and then pin the thread to a core using[1] if that’s what’s you’re after [0] - https://pkg.go.dev/runtime#LockOSThread https://pkg.go.dev/runtime#LockOSThread [1] - https://pkg.go.dev/golang.org/x/sys/unix#SchedSetaffinity https://pkg.go.dev/golang.org/x/sys/unix#SchedSetaffinity
dilyevsky 2y ago
Nope. In system like this, if written correctly, main performance bottlenecks are network and disk latencies

Animats 2y ago

You can get into that type of lock congestion trouble in any language. It's an algorithm problem, not a language problem. I discovered last year that Wine has terrible internal lock problems inside its user-side storage allocator. That's in C. If you have enough threads calling "realloc", the allocator goes into futex congestion collapse and performance drops by two orders of magnitude. My graphics program went from 60 FPS to 0.5 FPS. They optimized too hard for the no-congestion case. This is a Wine-only problem; Microsoft's own code doesn't have this problem. I've had lock congestion problems in Rust. Sometimes you need a fair mutex, or something gets frozen out. Both fair and non-fair mutexes are available; see the "parking_lot" crate. There's a place inside WGPU that has a lock congestion problem in one of three locks, and I'm going to have to add more profiling to someone else's code to find that. I can see the problem with Tracy, but need to add more profiling scopes to narrow it down. But that is high-performance graphics stuff, where microseconds count. Sending spam (OK, bulk marketing emails) doesn't need to be that tightly coupled. Mailing list removal runs on a timescale of days, not milliseconds. What else in that space has to be tightly interlocked?

forrestthewoods 2y ago
> I can see the problem with Tracy, but need to add more profiling scopes to narrow it down. If you can run on Windows try Superluminal. https://superluminal.eu/ https://superluminal.eu/
mgaunard 2y ago
A good language just gives you the necessary tooling to do whatever you want, it doesn't magically fix problems. Only languages like C++ have a memory model that allows you to do lock-free programming for example (C and Rust copied the C++ model). Also, what kind of serious person allocates memory from the system allocator in a real-time loop? Your problems seem self-inflicted. Regardless there are many allocators that optimize for concurrent allocations: tcmalloc, jemalloc, mimalloc...
- samatman 2y ago
  Go has atomics.
  - mgaunard 2y ago
    Which don't have the same level of control as C++, it only provides a subset of sequentially consistent primitives, which are slow by design.
rowanG077 2y ago
Considering the vast number of programs that wine works extremely well with I'm not so sure they spent too much optimizing the no-congestion case. You are just doing something extremely quirky in your program.
- Animats 2y ago
  I've looked at the code in a debugger. Wine has futexes three deep in "malloc". The innermost one is a pure spinlock. The problem with "realloc" is that, when it can't grow an array in place, it has to copy the contents. The Wine implementation does that with the main lock on allocation still held. So, if you have Rust code with a lot of multithreaded vector "push" operations, and more threads than CPUs, you get futex congestion. It's possible to write applications that don't hit this bug, but it's Wine-only, not Windows, so not worth it. What's "quirky" is trying to use all the CPUs with lower priority threads.
  - mgaunard 2y ago
    Why not send a patch?
    - Animats 2y ago
      It's hard to fix without a redesign of some crucial low-level code. The people who wrote it looked at the problem and decided it was too hard to fix.

faitswulff 2y ago

I suspect that when it was chosen at Mailgun, Golang was still being billed as a systems programming language.

neonsunset 2y ago
Exactly. Go makes you think you control the details except you don't. Hackernews makes you think you don't control the details in C# except you do. Yet another project that would have been able to solve its woes if it had picked a better option.

bee_rider 2y ago

Does the language make a huge difference here? In a distributed system a signal to be sent over the network travels at the same speed whether it was transmitted by a C or Python program, right?

jeffbee 2y ago

It would take you 1 minute to write a Go application that exploits per-CPU data structures and probably years to debug your attempt to do so in C.

convolvatron 2y ago
actually its really _hard_ in go to make cpu bound control flow, state, and allocation. do goroutines have any notion of locality? i've been looking and haven't been able to find anything
- jeffbee 2y ago
  Go comes out of the box with sync.Pool
- dilyevsky 2y ago
  You can pin the go routine to a current thread using[0] and then pin the thread to a core using[1] if that’s what’s you’re after [0] - https://pkg.go.dev/runtime#LockOSThread https://pkg.go.dev/runtime#LockOSThread [1] - https://pkg.go.dev/golang.org/x/sys/unix#SchedSetaffinity https://pkg.go.dev/golang.org/x/sys/unix#SchedSetaffinity

dilyevsky 2y ago

Nope. In system like this, if written correctly, main performance bottlenecks are network and disk latencies