6 ms·

I'm pretty sure this is a well-known problem in general on Linux -- file locking is almost never a good solution. It sucks, since there are definitely times whe

I'm pretty sure this is a well-known problem in general on Linux -- file locking is almost never a good solution. It sucks, since there are definitely times where it would be useful. In any event, this is probably a low priority for glibc/kernel devs -- the fact that syncing doesn't actually behave as expected is much more important (https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/).

mjw1007 8y ago
They added a solution for this three or four years ago, by adding "Open File Description Locks" which don't have the misbehaviour that that SQLite comment is complaining about. See https://www.gnu.org/software/libc/manual/html_mono/libc.html#Open-File-Description-Locks https://www.gnu.org/software/libc/manual/html_mono/libc.html...
- jwilk 8y ago
  Page with only the relevant chapter (rather than 5MB monstrosity): https://www.gnu.org/software/libc/manual/html_node/Open-File-Description-Locks.html https://www.gnu.org/software/libc/manual/html_node/Open-File...
- wahern 8y ago
  You can also use the BSD extension, flock(2), which is supported everywhere except commercial SysV clones like Solaris and HP/UX. flock has the semantics you would expect[1] and isn't Linux-only. This LWN article has a great explanation of all the behaviors: https://lwn.net/Articles/586904/ https://lwn.net/Articles/586904/ [1] Effectively identical to the Linux extension/POSIX proposal except that (1) they don't support ranges, (2) don't contend with POSIX locks (they do on real BSDs but not Linux), and (3) BSD flock won't work across NFS.
  - amerianwookie 8y ago
    I was under the impression that since[1] 2.6.12, when using BSD flock(), NFS will use POSIX locks that emulate BSD locking behavior and work over NFS. So long as you use consistent NFS implementations, shouldn't this work? [1] http://nfs.sourceforge.net/#faq_d10 http://nfs.sourceforge.net/#faq_d10
    - wahern 8y ago
      I stand corrected.
- SQLite 8y ago
  Yes. And there is code in the SQLite source tree (currently on a branch) that supports OFD locks. The problem we have is that SQLite is so widely deployed and is in so many systems and on so many platforms, that we have to continue to support Posix Advisory Locks (PAL) for the foreseeable future, for compatibility. It will be great if someday we can remove the PAL code. But for now, it has to stay in the tree.
  - amluto 8y ago
    Neat! Is there any reason it’s in a branch? OFD locks fully interoperate with regular POSIX locks, so there shouldn’t be any compatibility issue, except that SQLite would need to fall back on older kernels.
    - Annatar 8y ago
      Since OFD locks are a GNUism, putting that code into the mainline would immediately break portability. I quite like my SQLite databases on illumos and ZFS, so if it couldn’t work on illumos because of a Linux-specific feature, that’d be a serious problem.
      - bonzini 8y ago
        That's what #ifdef is for. SQLite has a number of those, given it runs even on platforms which are not POSIX at all!
        Annatar 8y ago
        I know what they are for and don’t need instruction on that, thank you very much. I also know that #ifdefs are a bad practice with many pitfalls so their use should be kept to a minimum. There is a treatise on the subject written in the early ‘90’s that I now see is very much relevant, judging by your comment. I wish I could quote it but since I’m using a smartphone to type this it’s too much hassle to find it. Regardless, from a software architecture’s point of view, introducing a GNUism or a Linuxism where it isn’t strictly necessary is an obviously bad idea. How’d you fare with that GCC build on the mailing list from couple of years back, by the way?
- bch 8y ago
  GNU != POSIX. Glibc work won’t fix (e.g.) the BSDs.
  - JdeBP 8y ago
    The BSDs already have flock(), as mentioned by M. Ahern on this very page at https://news.ycombinator.com/item?id=17605317 https://news.ycombinator.com/item?id=17605317 .
kazinator 8y ago
Actually, it wasn't a problem on Linux under the initial pthreads implementation. The API worked as you would expect for threads; the advisory locks were owned by individual threads. Eventually (probably after the change to NPTL and improved kernel support for threads) it was fixed to conform to the POSIX design whereby the locks are owned by the process.
- mmt 8y ago
  > the advisory locks were owned by individual threads. Eventually (probably after the change to NPTL and improved kernel support for threads) Isn't that because, in the time prior to that "improved kernel support", threads were really processes, and it was once they became full-fledge (i.e. fully lightweight) threads that they lost the ability to own the advisory locks? Or am I misremembering?
  - JdeBP 8y ago
    It was one of the known problems. * http://jdebp.info./FGA/linux-thread-problems.html http://jdebp.info./FGA/linux-thread-problems.html
blablabla123 8y ago
File locking seems good enough to me. I wasn't consiciously aware of this short-coming but when you think about it, it makes total sense: the syscalls for file locks (at least on Linux and macOS) are related to the file descriptors. So when you close the file, the lock is gone. Why would you in any case hold 2 file descriptors open of the same file in the same process, especially when you use locking? For a "multi-process application" (this sounds so antiquated) probably one thread should hold the fd and exclusively write on it.

mjw1007 8y ago

They added a solution for this three or four years ago, by adding "Open File Description Locks" which don't have the misbehaviour that that SQLite comment is complaining about. See https://www.gnu.org/software/libc/manual/html_mono/libc.html#Open-File-Description-Locks https://www.gnu.org/software/libc/manual/html_mono/libc.html...

jwilk 8y ago
Page with only the relevant chapter (rather than 5MB monstrosity): https://www.gnu.org/software/libc/manual/html_node/Open-File-Description-Locks.html https://www.gnu.org/software/libc/manual/html_node/Open-File...
wahern 8y ago
You can also use the BSD extension, flock(2), which is supported everywhere except commercial SysV clones like Solaris and HP/UX. flock has the semantics you would expect[1] and isn't Linux-only. This LWN article has a great explanation of all the behaviors: https://lwn.net/Articles/586904/ https://lwn.net/Articles/586904/ [1] Effectively identical to the Linux extension/POSIX proposal except that (1) they don't support ranges, (2) don't contend with POSIX locks (they do on real BSDs but not Linux), and (3) BSD flock won't work across NFS.
- amerianwookie 8y ago
  I was under the impression that since[1] 2.6.12, when using BSD flock(), NFS will use POSIX locks that emulate BSD locking behavior and work over NFS. So long as you use consistent NFS implementations, shouldn't this work? [1] http://nfs.sourceforge.net/#faq_d10 http://nfs.sourceforge.net/#faq_d10
  - wahern 8y ago
    I stand corrected.
SQLite 8y ago
Yes. And there is code in the SQLite source tree (currently on a branch) that supports OFD locks. The problem we have is that SQLite is so widely deployed and is in so many systems and on so many platforms, that we have to continue to support Posix Advisory Locks (PAL) for the foreseeable future, for compatibility. It will be great if someday we can remove the PAL code. But for now, it has to stay in the tree.
- amluto 8y ago
  Neat! Is there any reason it’s in a branch? OFD locks fully interoperate with regular POSIX locks, so there shouldn’t be any compatibility issue, except that SQLite would need to fall back on older kernels.
  - Annatar 8y ago
    Since OFD locks are a GNUism, putting that code into the mainline would immediately break portability. I quite like my SQLite databases on illumos and ZFS, so if it couldn’t work on illumos because of a Linux-specific feature, that’d be a serious problem.
    - bonzini 8y ago
      That's what #ifdef is for. SQLite has a number of those, given it runs even on platforms which are not POSIX at all!
      - Annatar 8y ago
        I know what they are for and don’t need instruction on that, thank you very much. I also know that #ifdefs are a bad practice with many pitfalls so their use should be kept to a minimum. There is a treatise on the subject written in the early ‘90’s that I now see is very much relevant, judging by your comment. I wish I could quote it but since I’m using a smartphone to type this it’s too much hassle to find it. Regardless, from a software architecture’s point of view, introducing a GNUism or a Linuxism where it isn’t strictly necessary is an obviously bad idea. How’d you fare with that GCC build on the mailing list from couple of years back, by the way?
bch 8y ago
GNU != POSIX. Glibc work won’t fix (e.g.) the BSDs.
- JdeBP 8y ago
  The BSDs already have flock(), as mentioned by M. Ahern on this very page at https://news.ycombinator.com/item?id=17605317 https://news.ycombinator.com/item?id=17605317 .

kazinator 8y ago

Actually, it wasn't a problem on Linux under the initial pthreads implementation. The API worked as you would expect for threads; the advisory locks were owned by individual threads. Eventually (probably after the change to NPTL and improved kernel support for threads) it was fixed to conform to the POSIX design whereby the locks are owned by the process.

mmt 8y ago
> the advisory locks were owned by individual threads. Eventually (probably after the change to NPTL and improved kernel support for threads) Isn't that because, in the time prior to that "improved kernel support", threads were really processes, and it was once they became full-fledge (i.e. fully lightweight) threads that they lost the ability to own the advisory locks? Or am I misremembering?
- JdeBP 8y ago
  It was one of the known problems. * http://jdebp.info./FGA/linux-thread-problems.html http://jdebp.info./FGA/linux-thread-problems.html

blablabla123 8y ago

File locking seems good enough to me. I wasn't consiciously aware of this short-coming but when you think about it, it makes total sense: the syscalls for file locks (at least on Linux and macOS) are related to the file descriptors. So when you close the file, the lock is gone. Why would you in any case hold 2 file descriptors open of the same file in the same process, especially when you use locking? For a "multi-process application" (this sounds so antiquated) probably one thread should hold the fd and exclusively write on it.