3 ms·

I'd love to learn more about Windows pthreads poor implementation of spinlocks mutexes. Do you have any cool links?

• dragontamer 8y ago
  Citation: myself. I'm not an expert but I think I know enough to see a bad implementation when I see one. As such, please take this analysis with a grain of salt. Second: I'm not entirely sure if this applies to "modern" pthreads-win32, but the Blender project uses a relatively old build of pthreads-win32. Third: my analysis is derived from the Blender diff here: https://dev-files.blender.org/file/data/rjmzozouj44bs6qicd3y/PHID-FILE-cre57pid5mvoizjirptg/file https://dev-files.blender.org/file/data/rjmzozouj44bs6qicd3y... I'm assuming that the Blender devs are smarter than me and have already tested this. :-) ------------ So going through the Blender source code we come across pthread_spin_lock as part of the diff. Considering the comment thread (as well as the diff) as adding Win32 #ifdefs and such, we can assume this was the cause of the performance issue. --------- I assume... okay, so my logic isn't 100% tight, sorry :-(... that the implementation uses this pthreads-win32: http://sourceware.org/pthreads-win32/ http://sourceware.org/pthreads-win32/ Which brings us to here: ftp://sourceware.org/pub/pthreads-win32/sources/pthreads-w32-2-9-1-release/pthread_spin_lock.c --------- Lets break it down why this is a bad implementation: 1. No "pause" instruction, causing latency issues when leaving the spinlock. See https://msdn.microsoft.com/en-us/library/windows/desktop/ms687419(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/ms6... or https://software.intel.com/en-us/node/524249 https://software.intel.com/en-us/node/524249 for more information on why "pause" should be used in every spinlock. Strangely enough, the "pause" instruction is found in Linux implementations of pthreads (so the author for pthreads-w32 just... didn't look at the Linux implementation or something?) 2. PTW32_INTERLOCKED_COMPARE_EXCHANGE_LONG seems inefficient to me. I'd imagine that the usage should be using a swap instead (aka: https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/content/wdm/nf-wdm-interlockedexchange https://docs.microsoft.com/en-us/windows-hardware/drivers/dd... ). Although I don't have any microbenchmark that "proves" this, I'd imagine that a swap is more efficient than a compare-and-swap. 3. A potential "slowpath" which includes a mutex-lock. Note that Windows Mutexes are incredibly heavy: including security features and even a "filesystem like" naming scheme. (See Windows Object Handles if you don't believe me). Windows Mutexes are very fully featured: recursive, multiple levels of security, and more. This is a common problem in Linux system-coders -> Windows system coders. The Windows "Critical Section" is a more appropriate replacement for Linux pthread_mutexes. I'd say Windows Mutexes might be more similar to a Linux filesystem-level semaphore. Linux semaphores provide those security features integrated with the filesystem. The to Linux mutexes and spinlocks is something called a "Critical Section" in Windows land. https://docs.microsoft.com/en-us/windows/desktop/Sync/critical-section-objects https://docs.microsoft.com/en-us/windows/desktop/Sync/critic... Critical sections are Windows's lightweight yielding / task sleeping mechanism. --------- Finally, Windows provides a spinlock: https://docs.microsoft.com/en-us/windows/desktop/api/synchapi/nf-synchapi-initializecriticalsectionandspincount https://docs.microsoft.com/en-us/windows/desktop/api/synchap... Which is probably is what pthreads-w32 should really be using. I mean, Linux devs basically just don't know what Windows offers, and the pthread-win32 library is a poor fit at the moment. ----------------- Sorry if you were expecting more complete analysis and testing. Lol. Its mostly me looking at the thing and making a guess and reverse-engineering the patch-notes from the Blender discussion.
  • robocat 8y ago
    Tangentially relevant: Microsoft .NET was 50% slower on Skylake (some loads) due to changes by Intel to the pause instruction interfering with poor spinlock implementation by Microsoft: https://news.ycombinator.com/item?id=17336853 https://news.ycombinator.com/item?id=17336853
  • caf 8y ago
    These days it would likely be best to implement pthread-win32 mutexes using WaitOnAddress.
    • deleted 8y ago
      [deleted]
    • dragontamer 8y ago
      Huh, I didn't know about this new primitive. Thanks! It seems to have been introduced into Windows 8 and later, as a competitor against Linux's futex. I did a brief search online, and Raymond Chen had this to say: https://blogs.msdn.microsoft.com/oldnewthing/20160825-00/?p=94165 https://blogs.msdn.microsoft.com/oldnewthing/20160825-00/?p=... > [snip] in real life, you should just use Enter­Critical­Section because it has stuff like spin counts and lock convoy resistance. Is there anything that pthread_mutex_lock does that EnterCriticalSection doesn't do? It seems like a good "translation" to me. I'd only use the WaitOnAddress thing if pthread_mutex_lock had some features that a raw EnterCriticalSection wouldn't do.
      • caf 8y ago
        You need to be able to link your pthread condition variable implementation to your pthread mutex implementation such that a pthread_cond_wait() does an atomic wait-on-condition-variable-and-unlock-mutex, noting that another thread can signal the condition variable without holding any mutex. In addition, pthread mutexes can be statically initialised which requires some kind of initialise-on-first-use for those if you implement them using Critical Section objects. I believe WaitOnAddress(), rather than being a "competitor" for futex was actually added (or at least, the underlying infrastructure was) in order to be able to implement futex() in the WSL.