3 ms·

I’m curious how it was determined that the locking was causing problems? Was it just some intuition from cases with poor performance and staring at code, or som

I’m curious how it was determined that the locking was causing problems? Was it just some intuition from cases with poor performance and staring at code, or some perf-like tool (I don’t really know about the landscape of performance monitoring tools for MacOS) highlighting that a lot of time was spent in locks, or some known MacOS vs others difference? I think I also don’t understand the discussion about spinning in kernel space. Doesn’t that require context switches which the article points out have issues? I would have guessed the answer would be to spin a bit in userspace with architecture-specific instructions like pause and then syscall if the lock couldn’t be acquired. Maybe I just don’t really know how syscalls work on a Mac. I’m also weakly curious why jemalloc needs so much locking anyway? I haven’t thought about this at all so I’m surely missing something but my guess would be allocation from per-thread pools with occasional synchronisation, and that this synchronisation would be too infrequent to have these issues.

saagarjha 4y ago
From the blog post it seems like the team was using OSSpinLock, which was deprecated (for good reason) and then when trying to use the replacement (os_unfair_lock) degraded performance on some tests. I don’t have details on how they determined what the issues were but considering if the results were across a single change and one had access to the source it doesn’t seem all that difficult to see why there are performance differences. The various profiling tools on macOS (Instruments, mostly) would help confirm this. Spinning in kernel space does require a transition into the kernel. But, critically, it does not require rescheduling another thread and waiting around, which is even more expensive. The kernel has the context to know what is running and what isn’t, as well as some information on who owns the lock, so it can do a pretty good job here. Spinning in userspace is bad because the thread holding the lock may be parked by the kernel, which means that you’re just spinning for no reason. If you don’t include any special instructions you burn power, and even if you do you don’t benefit from the knowledge the kernel has of whether you should just be scheduled off while waiting anyways.
enragedcacti 4y ago
> I’m also weakly curious why jemalloc needs so much locking anyway? I'd love to hear from someone with more firsthand knowledge, but from a quick read of the implementation details for jemalloc, it seems there is a balance that has to be struck between lock contention and memory fragmentation. Too few arenas and lock contention slows you down, too many arenas (or enabling thread-specific caches) and the cache misses from fragmentation will slow you down. the blog post also mentioned that Firefox's jemalloc is highly customized, maybe firefox's specific profile gets value out of a more lock-heavy strategy. https://jemalloc.net/jemalloc.3.html https://jemalloc.net/jemalloc.3.html (see "Implementation Notes")
soheil 4y ago
By probably monitoring CPU cache hit rates would be one way.
- dan-robertson 4y ago
  Is there even a way to access cache hit rates on MacOS?
  - soheil 4y ago
    You can probably infer them by measuring computation time.
    - dan-robertson 4y ago
      I guess you mean wrapping very small sections of the program with timing code and making sure the cpu executes the timing instructions in the right order. Maybe that works but it sounds like choosing between poor accuracy and massive overhead. On Linux on x86 you can use perf to get access to model-specific registers on the cpu for a large number of performance counters (from cache hit counts to branch prediction misses to all kinds of crazy things). I’m not aware of an equivalent to perf on macOS and I’m not sure the kernel provides appropriate apis. Maybe instruments is it but I’m not sure that it can give access to those counters (they surely exist on the M1 cpus) or that it can be run in a non-interactive way.

saagarjha 4y ago

From the blog post it seems like the team was using OSSpinLock, which was deprecated (for good reason) and then when trying to use the replacement (os_unfair_lock) degraded performance on some tests. I don’t have details on how they determined what the issues were but considering if the results were across a single change and one had access to the source it doesn’t seem all that difficult to see why there are performance differences. The various profiling tools on macOS (Instruments, mostly) would help confirm this. Spinning in kernel space does require a transition into the kernel. But, critically, it does not require rescheduling another thread and waiting around, which is even more expensive. The kernel has the context to know what is running and what isn’t, as well as some information on who owns the lock, so it can do a pretty good job here. Spinning in userspace is bad because the thread holding the lock may be parked by the kernel, which means that you’re just spinning for no reason. If you don’t include any special instructions you burn power, and even if you do you don’t benefit from the knowledge the kernel has of whether you should just be scheduled off while waiting anyways.

enragedcacti 4y ago

> I’m also weakly curious why jemalloc needs so much locking anyway? I'd love to hear from someone with more firsthand knowledge, but from a quick read of the implementation details for jemalloc, it seems there is a balance that has to be struck between lock contention and memory fragmentation. Too few arenas and lock contention slows you down, too many arenas (or enabling thread-specific caches) and the cache misses from fragmentation will slow you down. the blog post also mentioned that Firefox's jemalloc is highly customized, maybe firefox's specific profile gets value out of a more lock-heavy strategy. https://jemalloc.net/jemalloc.3.html https://jemalloc.net/jemalloc.3.html (see "Implementation Notes")

soheil 4y ago

By probably monitoring CPU cache hit rates would be one way.

dan-robertson 4y ago
Is there even a way to access cache hit rates on MacOS?
- soheil 4y ago
  You can probably infer them by measuring computation time.
  - dan-robertson 4y ago
    I guess you mean wrapping very small sections of the program with timing code and making sure the cpu executes the timing instructions in the right order. Maybe that works but it sounds like choosing between poor accuracy and massive overhead. On Linux on x86 you can use perf to get access to model-specific registers on the cpu for a large number of performance counters (from cache hit counts to branch prediction misses to all kinds of crazy things). I’m not aware of an equivalent to perf on macOS and I’m not sure the kernel provides appropriate apis. Maybe instruments is it but I’m not sure that it can give access to those counters (they surely exist on the M1 cpus) or that it can be run in a non-interactive way.