13 ms·

“POSIX advisory locks are broken by design”

150 points by okket 8y ago

tonysdg 8y ago
I'm pretty sure this is a well-known problem in general on Linux -- file locking is almost never a good solution. It sucks, since there are definitely times where it would be useful. In any event, this is probably a low priority for glibc/kernel devs -- the fact that syncing doesn't actually behave as expected is much more important (https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/).
- mjw1007 8y ago
  They added a solution for this three or four years ago, by adding "Open File Description Locks" which don't have the misbehaviour that that SQLite comment is complaining about. See https://www.gnu.org/software/libc/manual/html_mono/libc.html#Open-File-Description-Locks https://www.gnu.org/software/libc/manual/html_mono/libc.html...
  - jwilk 8y ago
    Page with only the relevant chapter (rather than 5MB monstrosity): https://www.gnu.org/software/libc/manual/html_node/Open-File-Description-Locks.html https://www.gnu.org/software/libc/manual/html_node/Open-File...
  - wahern 8y ago
    You can also use the BSD extension, flock(2), which is supported everywhere except commercial SysV clones like Solaris and HP/UX. flock has the semantics you would expect[1] and isn't Linux-only. This LWN article has a great explanation of all the behaviors: https://lwn.net/Articles/586904/ https://lwn.net/Articles/586904/ [1] Effectively identical to the Linux extension/POSIX proposal except that (1) they don't support ranges, (2) don't contend with POSIX locks (they do on real BSDs but not Linux), and (3) BSD flock won't work across NFS.
    - amerianwookie 8y ago
      I was under the impression that since[1] 2.6.12, when using BSD flock(), NFS will use POSIX locks that emulate BSD locking behavior and work over NFS. So long as you use consistent NFS implementations, shouldn't this work? [1] http://nfs.sourceforge.net/#faq_d10 http://nfs.sourceforge.net/#faq_d10
      - wahern 8y ago
        I stand corrected.
  - SQLite 8y ago
    Yes. And there is code in the SQLite source tree (currently on a branch) that supports OFD locks. The problem we have is that SQLite is so widely deployed and is in so many systems and on so many platforms, that we have to continue to support Posix Advisory Locks (PAL) for the foreseeable future, for compatibility. It will be great if someday we can remove the PAL code. But for now, it has to stay in the tree.
    - amluto 8y ago
      Neat! Is there any reason it’s in a branch? OFD locks fully interoperate with regular POSIX locks, so there shouldn’t be any compatibility issue, except that SQLite would need to fall back on older kernels.
      - Annatar 8y ago
        Since OFD locks are a GNUism, putting that code into the mainline would immediately break portability. I quite like my SQLite databases on illumos and ZFS, so if it couldn’t work on illumos because of a Linux-specific feature, that’d be a serious problem.
        bonzini 8y ago
        That's what #ifdef is for. SQLite has a number of those, given it runs even on platforms which are not POSIX at all!
        Annatar 8y ago
        I know what they are for and don’t need instruction on that, thank you very much. I also know that #ifdefs are a bad practice with many pitfalls so their use should be kept to a minimum. There is a treatise on the subject written in the early ‘90’s that I now see is very much relevant, judging by your comment. I wish I could quote it but since I’m using a smartphone to type this it’s too much hassle to find it. Regardless, from a software architecture’s point of view, introducing a GNUism or a Linuxism where it isn’t strictly necessary is an obviously bad idea. How’d you fare with that GCC build on the mailing list from couple of years back, by the way?
  - bch 8y ago
    GNU != POSIX. Glibc work won’t fix (e.g.) the BSDs.
    - JdeBP 8y ago
      The BSDs already have flock(), as mentioned by M. Ahern on this very page at https://news.ycombinator.com/item?id=17605317 https://news.ycombinator.com/item?id=17605317 .
- kazinator 8y ago
  Actually, it wasn't a problem on Linux under the initial pthreads implementation. The API worked as you would expect for threads; the advisory locks were owned by individual threads. Eventually (probably after the change to NPTL and improved kernel support for threads) it was fixed to conform to the POSIX design whereby the locks are owned by the process.
  - mmt 8y ago
    > the advisory locks were owned by individual threads. Eventually (probably after the change to NPTL and improved kernel support for threads) Isn't that because, in the time prior to that "improved kernel support", threads were really processes, and it was once they became full-fledge (i.e. fully lightweight) threads that they lost the ability to own the advisory locks? Or am I misremembering?
    - JdeBP 8y ago
      It was one of the known problems. * http://jdebp.info./FGA/linux-thread-problems.html http://jdebp.info./FGA/linux-thread-problems.html
- blablabla123 8y ago
  File locking seems good enough to me. I wasn't consiciously aware of this short-coming but when you think about it, it makes total sense: the syscalls for file locks (at least on Linux and macOS) are related to the file descriptors. So when you close the file, the lock is gone. Why would you in any case hold 2 file descriptors open of the same file in the same process, especially when you use locking? For a "multi-process application" (this sounds so antiquated) probably one thread should hold the fd and exclusively write on it.
mmt 8y ago
My (admittedly quick) reading of this is that the issues stem, primarily, if not exclusively, from using multiple file descriptors for the same file, necessitated by using/supporting threading for concurrency. If that's the case, then perhaps it's not so much that PAL was broken by design but that the original design merely didn't anticipate threading. I'm not sure how the advisory locking scheme compares in terms of timeline with threading support, but Unix file locking in general easily predates even SMP.
- Hello71 8y ago
  I don't think that's correct. According to Wikipedia, SMP dates to approximately the early 1960s, whereas according to the FreeBSD man pages, flock and fcntl were both introduced in 4.2BSD, released in 1983, two decades later. It's at most fair to say that Unix file locking came around the same time as Unix SMP.
  - deleted 8y ago
    [deleted]
  - throwaway76543 8y ago
    "Unix SMP" is not synonymous with a system threading interface -- processes give parallelism as well and predate the introduction of threading interfaces. The pthread interface came about in the 1990s. The unix interface is broken with respect to threads in many places besides this. Signals. Forking. Problems abound because the concept was bolted on decades later.
    - mmt 8y ago
      > processes give parallelism as well and predate the introduction of threading interfaces Right, but the reverse isn't true, which is what I was getting at. That is, availability of hardware parallelism was a prerequisite for anyone wanting threading in the first place.
      - MaxBarraclough 8y ago
emmelaich 8y ago
If you want locks from the same user you'd use other mechanisms that file locks. File locks are for processes that don't have a lighter weight way to cooperate. (e.g. semaphores.) Also, there's a reason that databases have processes dedicated to writing (e.g. Postgres writer process, Oracle dbwr) Fun fact - IBM's MVS didn't have file locks.
- pvg 8y ago
  there's a reason that databases have processes dedicated to writing That's going to be a little tricky in an embedded db.
  - emmelaich 8y ago
    Sure, but you could still make sure writes are done in a mutex?
    - int_19h 8y ago
      Don't forget that it still needs to synchronize across process boundaries, too - two processes, both running SQLite, and both accessing the same database file, is a supported scenario.
CaliforniaKarl 8y ago
An unrelated comment: Looking through the rest of the file, I love how many comments there are in general! I am able to get a sense of what each function does, and what the individual code blocks within the function are doing. This is something that I do not see as often as I want, and I'm really glad to see it here!
Annatar 8y ago
When he gets a file descriptor, he could also get an inode. Since on hard links inode will be the same, the prior lock state, if any, can be obtained and action taken accordingly.
- knorker 8y ago
  You didn't read the whole thing, did you?
tinus_hn 8y ago
As far as I know these alternative thread libraries are ancient history. I can understand you want to defend against sabotage like people using hard links to open a database under two names but is that really a dealbreaker?

150 points by okket 8y ago

I'm pretty sure this is a well-known problem in general on Linux -- file locking is almost never a good solution. It sucks, since there are definitely times where it would be useful. In any event, this is probably a low priority for glibc/kernel devs -- the fact that syncing doesn't actually behave as expected is much more important (https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/).

mjw1007 8y ago
They added a solution for this three or four years ago, by adding "Open File Description Locks" which don't have the misbehaviour that that SQLite comment is complaining about. See https://www.gnu.org/software/libc/manual/html_mono/libc.html#Open-File-Description-Locks https://www.gnu.org/software/libc/manual/html_mono/libc.html...
- jwilk 8y ago
  Page with only the relevant chapter (rather than 5MB monstrosity): https://www.gnu.org/software/libc/manual/html_node/Open-File-Description-Locks.html https://www.gnu.org/software/libc/manual/html_node/Open-File...
- wahern 8y ago
  You can also use the BSD extension, flock(2), which is supported everywhere except commercial SysV clones like Solaris and HP/UX. flock has the semantics you would expect[1] and isn't Linux-only. This LWN article has a great explanation of all the behaviors: https://lwn.net/Articles/586904/ https://lwn.net/Articles/586904/ [1] Effectively identical to the Linux extension/POSIX proposal except that (1) they don't support ranges, (2) don't contend with POSIX locks (they do on real BSDs but not Linux), and (3) BSD flock won't work across NFS.
  - amerianwookie 8y ago
    I was under the impression that since[1] 2.6.12, when using BSD flock(), NFS will use POSIX locks that emulate BSD locking behavior and work over NFS. So long as you use consistent NFS implementations, shouldn't this work? [1] http://nfs.sourceforge.net/#faq_d10 http://nfs.sourceforge.net/#faq_d10
    - wahern 8y ago
      I stand corrected.
- SQLite 8y ago
  Yes. And there is code in the SQLite source tree (currently on a branch) that supports OFD locks. The problem we have is that SQLite is so widely deployed and is in so many systems and on so many platforms, that we have to continue to support Posix Advisory Locks (PAL) for the foreseeable future, for compatibility. It will be great if someday we can remove the PAL code. But for now, it has to stay in the tree.
  - amluto 8y ago
    Neat! Is there any reason it’s in a branch? OFD locks fully interoperate with regular POSIX locks, so there shouldn’t be any compatibility issue, except that SQLite would need to fall back on older kernels.
    - Annatar 8y ago
      Since OFD locks are a GNUism, putting that code into the mainline would immediately break portability. I quite like my SQLite databases on illumos and ZFS, so if it couldn’t work on illumos because of a Linux-specific feature, that’d be a serious problem.
      - bonzini 8y ago
        That's what #ifdef is for. SQLite has a number of those, given it runs even on platforms which are not POSIX at all!
        Annatar 8y ago
        I know what they are for and don’t need instruction on that, thank you very much. I also know that #ifdefs are a bad practice with many pitfalls so their use should be kept to a minimum. There is a treatise on the subject written in the early ‘90’s that I now see is very much relevant, judging by your comment. I wish I could quote it but since I’m using a smartphone to type this it’s too much hassle to find it. Regardless, from a software architecture’s point of view, introducing a GNUism or a Linuxism where it isn’t strictly necessary is an obviously bad idea. How’d you fare with that GCC build on the mailing list from couple of years back, by the way?
- bch 8y ago
  GNU != POSIX. Glibc work won’t fix (e.g.) the BSDs.
  - JdeBP 8y ago
    The BSDs already have flock(), as mentioned by M. Ahern on this very page at https://news.ycombinator.com/item?id=17605317 https://news.ycombinator.com/item?id=17605317 .
kazinator 8y ago
Actually, it wasn't a problem on Linux under the initial pthreads implementation. The API worked as you would expect for threads; the advisory locks were owned by individual threads. Eventually (probably after the change to NPTL and improved kernel support for threads) it was fixed to conform to the POSIX design whereby the locks are owned by the process.
- mmt 8y ago
  > the advisory locks were owned by individual threads. Eventually (probably after the change to NPTL and improved kernel support for threads) Isn't that because, in the time prior to that "improved kernel support", threads were really processes, and it was once they became full-fledge (i.e. fully lightweight) threads that they lost the ability to own the advisory locks? Or am I misremembering?
  - JdeBP 8y ago
    It was one of the known problems. * http://jdebp.info./FGA/linux-thread-problems.html http://jdebp.info./FGA/linux-thread-problems.html
blablabla123 8y ago
File locking seems good enough to me. I wasn't consiciously aware of this short-coming but when you think about it, it makes total sense: the syscalls for file locks (at least on Linux and macOS) are related to the file descriptors. So when you close the file, the lock is gone. Why would you in any case hold 2 file descriptors open of the same file in the same process, especially when you use locking? For a "multi-process application" (this sounds so antiquated) probably one thread should hold the fd and exclusively write on it.

mmt 8y ago

My (admittedly quick) reading of this is that the issues stem, primarily, if not exclusively, from using multiple file descriptors for the same file, necessitated by using/supporting threading for concurrency. If that's the case, then perhaps it's not so much that PAL was broken by design but that the original design merely didn't anticipate threading. I'm not sure how the advisory locking scheme compares in terms of timeline with threading support, but Unix file locking in general easily predates even SMP.

Hello71 8y ago
I don't think that's correct. According to Wikipedia, SMP dates to approximately the early 1960s, whereas according to the FreeBSD man pages, flock and fcntl were both introduced in 4.2BSD, released in 1983, two decades later. It's at most fair to say that Unix file locking came around the same time as Unix SMP.
- deleted 8y ago
  [deleted]
- throwaway76543 8y ago
  "Unix SMP" is not synonymous with a system threading interface -- processes give parallelism as well and predate the introduction of threading interfaces. The pthread interface came about in the 1990s. The unix interface is broken with respect to threads in many places besides this. Signals. Forking. Problems abound because the concept was bolted on decades later.
  - mmt 8y ago
    > processes give parallelism as well and predate the introduction of threading interfaces Right, but the reverse isn't true, which is what I was getting at. That is, availability of hardware parallelism was a prerequisite for anyone wanting threading in the first place.
    - MaxBarraclough 8y ago

emmelaich 8y ago

If you want locks from the same user you'd use other mechanisms that file locks. File locks are for processes that don't have a lighter weight way to cooperate. (e.g. semaphores.) Also, there's a reason that databases have processes dedicated to writing (e.g. Postgres writer process, Oracle dbwr) Fun fact - IBM's MVS didn't have file locks.

pvg 8y ago
there's a reason that databases have processes dedicated to writing That's going to be a little tricky in an embedded db.
- emmelaich 8y ago
  Sure, but you could still make sure writes are done in a mutex?
  - int_19h 8y ago
    Don't forget that it still needs to synchronize across process boundaries, too - two processes, both running SQLite, and both accessing the same database file, is a supported scenario.

CaliforniaKarl 8y ago

An unrelated comment: Looking through the rest of the file, I love how many comments there are in general! I am able to get a sense of what each function does, and what the individual code blocks within the function are doing. This is something that I do not see as often as I want, and I'm really glad to see it here!

Annatar 8y ago

When he gets a file descriptor, he could also get an inode. Since on hard links inode will be the same, the prior lock state, if any, can be obtained and action taken accordingly.

knorker 8y ago
You didn't read the whole thing, did you?

tinus_hn 8y ago

As far as I know these alternative thread libraries are ancient history. I can understand you want to defend against sabotage like people using hard links to open a database under two names but is that really a dealbreaker?