3 ms·

> Combine that with how cheap `std::mutex` is on most platforms and it's just really a last resort, at least on server-class gear. Mere 10-20M std::mutex opera

> Combine that with how cheap `std::mutex` is on most platforms and it's just really a last resort, at least on server-class gear. Mere 10-20M std::mutex operations per second can bring a server to a total halt. CPU interconnects and CPU internal core-to-core buses are quickly saturated by required synchronization traffic. So I wouldn't call it a particularly cheap operation. In fact, I can't think of many more expensive operations a CPU could perform. I guess just page walks and interrupts might qualify.

gpderetta 4y ago
Uncontended mutex operations are cheap and do not require core-core communication. Contended operations are expensive, whether they are mutex acquisitions, atomic RMW or plain writes.
- benreesman 4y ago
  Ah I missed your comment when writing my sibling. Your comment is much more succinct while making the point at least as well if not better.
benreesman 4y ago
The dominant term in the type of scenario you're describing is contention, not mutexes. I forget how big they're making Zen3 EPYCs right now, I think you can get like 64-128 physical cores. I can walk all over 128 different mutexes on each of those 128 cores for less than the cost of an L3 hit on that same EPYC box [1]. If I fuck up the affinity and they're all banging on one data structure? Yeah, it can jam the whole machine. This is why people who write good logging libraries are always playing games with buffering in thread-local storage and stuff. And sometimes you do need a bunch of producer threads dumping into a queue that simply can't be striped by anything (it's somewhat rare but it happens): in which case you want `folly::MPMCQueue` (or whatever Intel TBB calls their version of that) which Nathan Bronson was kind enough to get right so that other people don't have to end up with something both slow and wrong. A truly novel lock-free data structure that advances the SOTA in production software is a masters thesis, not a blog post. [1] This is a bit dated but it's good enough for eyeball work: https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832.

gpderetta 4y ago

Uncontended mutex operations are cheap and do not require core-core communication. Contended operations are expensive, whether they are mutex acquisitions, atomic RMW or plain writes.

benreesman 4y ago
Ah I missed your comment when writing my sibling. Your comment is much more succinct while making the point at least as well if not better.

benreesman 4y ago

The dominant term in the type of scenario you're describing is contention, not mutexes. I forget how big they're making Zen3 EPYCs right now, I think you can get like 64-128 physical cores. I can walk all over 128 different mutexes on each of those 128 cores for less than the cost of an L3 hit on that same EPYC box [1]. If I fuck up the affinity and they're all banging on one data structure? Yeah, it can jam the whole machine. This is why people who write good logging libraries are always playing games with buffering in thread-local storage and stuff. And sometimes you do need a bunch of producer threads dumping into a queue that simply can't be striped by anything (it's somewhat rare but it happens): in which case you want `folly::MPMCQueue` (or whatever Intel TBB calls their version of that) which Nathan Bronson was kind enough to get right so that other people don't have to end up with something both slow and wrong. A truly novel lock-free data structure that advances the SOTA in production software is a masters thesis, not a blog post. [1] This is a bit dated but it's good enough for eyeball work: https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832.