10 ms·

Is there a known recent Linux locking bug that affects the OCaml runtime?

108 points by lelf 6y ago

jeremyjh 6y ago
Its an interesting read but the synopsis is a glibc bug[1] in 2.27 that causes pthread_cond_signal to sometimes fail to wake waiting threads. This manifests as various hangs/locks that may seem to happen randomly in different application runtimes, including reports of users of .NET Core, Python, OCaml. https://sourceware.org/bugzilla/show_bug.cgi?id=25847 https://sourceware.org/bugzilla/show_bug.cgi?id=25847
- mlvljr 6y ago
  There is also https://sourceware.org/bugzilla/show_bug.cgi?id=12683 https://sourceware.org/bugzilla/show_bug.cgi?id=12683 , for those curious about how glibc holds up in terms of quality.
remexre 6y ago
I'm kinda surprised none of the model-checking people have tried verifying glibc's NPTL implementation; I guess it might be hard to formalize futex's behavior?
- javert 6y ago
  This sounds to me like a good PhD project and/or series of research papers. Disclaimer: I'm not in the model-checking community. If it's hard to formalize futex's behavior, all the better for academics, though it's unfortunate for the rest of us.
- mbac32768 6y ago
  This bug is really, really subtle. The repro that was posted that just tries to stress this bug can take hours to deadlock on some machines. Confirming that you've fixed it properly is a bit harrowing. Even as an OCaml snob I haven't a clue how you would verify the behavior of this C library formally.
anonymousiam 6y ago
Looks like this fix takes care of the problem. Now the challenge is to get it rolled out everywhere: https://sourceware.org/bugzilla/attachment.cgi?id=12484 https://sourceware.org/bugzilla/attachment.cgi?id=12484
- Dylan16807 6y ago
  The comment there makes it sound like it might be giving up on the stealing optimization entirely? If that's the case then this might not be the fix to roll out everywhere.
  - wbl 6y ago
    Do you want to wake up at 3am to determine that yes, the broken pthreads implementation really is broken?
    - Dylan16807 6y ago
      Do you think a desire to put more than five minutes into designing the patch means I don't want it fixed, or don't want it fixed quickly?
  - ComputerGuru 6y ago
    No, this is only in the undo path when stealing already failed. But waking all waiters is kind of crazy. Microsoft’s equivalent API (WFMO) is free of spurious wakes, I can’t imagine how much harder that makes everything
hacker_newz 6y ago
I'm going to go out on a limb and guess mbacarella works at Jane Street?
- whimsicalism 6y ago
  Of course https://blog.janestreet.com/author/mbacarella/ https://blog.janestreet.com/author/mbacarella/
  - mbac32768 6y ago
    Used to, I left in 2015. Still using OCaml!
pengaru 6y ago
If pthread_cond_signal() is broken pthreads is useless for all but the most trivial threaded programs.
- geocar 6y ago
  Nail on head. Pthreads is an extremely difficult API to use, owing first to a poor choice of primitives, the consequences of which are that none of the implementations are particularly good and their interactions with the rest of the machine/model are ill-defined. For example: You know how hard it is to close() in a multithreaded program? [1] And that’s just one system call— one almost everyone thinks they understand. What hope is there for all the other pthreads-code? For cond/etc: One trick I use is handle passing via an epoll[2]. This works much more reliably than the pthreads API, and whilst it’s slower in a microbenchmark that spins the pthread_cond/mutex/etc apis, real programs don’t do that (or don’t need to), and so it is usually a clear net-win. [1]: http://geocar.sdf1.org/close.html http://geocar.sdf1.org/close.html [2]: http://geocar.sdf1.org/fast-servers.html http://geocar.sdf1.org/fast-servers.html
  - rwmj 6y ago
    I believe the latest POSIX spec has fixed the close issue. Long story is https://www.austingroupbugs.net/view.php?id=529 https://www.austingroupbugs.net/view.php?id=529
exabrial 6y ago
> One thread, however, is blocked in libc read on stdin. When I hit Enter, the read finishes and the whole program unhangs. Seems like a problem easily solved with a Cron job! /me exits building
egberts1 6y ago
Still experiencing KDE Plasma lockups in Debian Bulleye (11). It’s time for me to hook up a dumb terminal to its serial port and leave it there.

108 points by lelf 6y ago

Its an interesting read but the synopsis is a glibc bug[1] in 2.27 that causes pthread_cond_signal to sometimes fail to wake waiting threads. This manifests as various hangs/locks that may seem to happen randomly in different application runtimes, including reports of users of .NET Core, Python, OCaml. https://sourceware.org/bugzilla/show_bug.cgi?id=25847 https://sourceware.org/bugzilla/show_bug.cgi?id=25847

mlvljr 6y ago
There is also https://sourceware.org/bugzilla/show_bug.cgi?id=12683 https://sourceware.org/bugzilla/show_bug.cgi?id=12683 , for those curious about how glibc holds up in terms of quality.

remexre 6y ago

I'm kinda surprised none of the model-checking people have tried verifying glibc's NPTL implementation; I guess it might be hard to formalize futex's behavior?

javert 6y ago
This sounds to me like a good PhD project and/or series of research papers. Disclaimer: I'm not in the model-checking community. If it's hard to formalize futex's behavior, all the better for academics, though it's unfortunate for the rest of us.
mbac32768 6y ago
This bug is really, really subtle. The repro that was posted that just tries to stress this bug can take hours to deadlock on some machines. Confirming that you've fixed it properly is a bit harrowing. Even as an OCaml snob I haven't a clue how you would verify the behavior of this C library formally.

anonymousiam 6y ago

Looks like this fix takes care of the problem. Now the challenge is to get it rolled out everywhere: https://sourceware.org/bugzilla/attachment.cgi?id=12484 https://sourceware.org/bugzilla/attachment.cgi?id=12484

Dylan16807 6y ago
The comment there makes it sound like it might be giving up on the stealing optimization entirely? If that's the case then this might not be the fix to roll out everywhere.
- wbl 6y ago
  Do you want to wake up at 3am to determine that yes, the broken pthreads implementation really is broken?
  - Dylan16807 6y ago
    Do you think a desire to put more than five minutes into designing the patch means I don't want it fixed, or don't want it fixed quickly?
- ComputerGuru 6y ago
  No, this is only in the undo path when stealing already failed. But waking all waiters is kind of crazy. Microsoft’s equivalent API (WFMO) is free of spurious wakes, I can’t imagine how much harder that makes everything

hacker_newz 6y ago

I'm going to go out on a limb and guess mbacarella works at Jane Street?

whimsicalism 6y ago
Of course https://blog.janestreet.com/author/mbacarella/ https://blog.janestreet.com/author/mbacarella/
- mbac32768 6y ago
  Used to, I left in 2015. Still using OCaml!

pengaru 6y ago

If pthread_cond_signal() is broken pthreads is useless for all but the most trivial threaded programs.

geocar 6y ago
Nail on head. Pthreads is an extremely difficult API to use, owing first to a poor choice of primitives, the consequences of which are that none of the implementations are particularly good and their interactions with the rest of the machine/model are ill-defined. For example: You know how hard it is to close() in a multithreaded program? [1] And that’s just one system call— one almost everyone thinks they understand. What hope is there for all the other pthreads-code? For cond/etc: One trick I use is handle passing via an epoll[2]. This works much more reliably than the pthreads API, and whilst it’s slower in a microbenchmark that spins the pthread_cond/mutex/etc apis, real programs don’t do that (or don’t need to), and so it is usually a clear net-win. [1]: http://geocar.sdf1.org/close.html http://geocar.sdf1.org/close.html [2]: http://geocar.sdf1.org/fast-servers.html http://geocar.sdf1.org/fast-servers.html
- rwmj 6y ago
  I believe the latest POSIX spec has fixed the close issue. Long story is https://www.austingroupbugs.net/view.php?id=529 https://www.austingroupbugs.net/view.php?id=529

exabrial 6y ago

> One thread, however, is blocked in libc read on stdin. When I hit Enter, the read finishes and the whole program unhangs. Seems like a problem easily solved with a Cron job! /me exits building

egberts1 6y ago

Still experiencing KDE Plasma lockups in Debian Bulleye (11). It’s time for me to hook up a dumb terminal to its serial port and leave it there.