4 ms·

Yes. And there is code in the SQLite source tree (currently on a branch) that supports OFD locks. The problem we have is that SQLite is so widely deployed and

by SQLite 8y ago

Yes. And there is code in the SQLite source tree (currently on a branch) that supports OFD locks. The problem we have is that SQLite is so widely deployed and is in so many systems and on so many platforms, that we have to continue to support Posix Advisory Locks (PAL) for the foreseeable future, for compatibility. It will be great if someday we can remove the PAL code. But for now, it has to stay in the tree.

amluto 8y ago
Neat! Is there any reason it’s in a branch? OFD locks fully interoperate with regular POSIX locks, so there shouldn’t be any compatibility issue, except that SQLite would need to fall back on older kernels.
- Annatar 8y ago
  Since OFD locks are a GNUism, putting that code into the mainline would immediately break portability. I quite like my SQLite databases on illumos and ZFS, so if it couldn’t work on illumos because of a Linux-specific feature, that’d be a serious problem.
  - bonzini 8y ago
    That's what #ifdef is for. SQLite has a number of those, given it runs even on platforms which are not POSIX at all!
    - Annatar 8y ago
      I know what they are for and don’t need instruction on that, thank you very much. I also know that #ifdefs are a bad practice with many pitfalls so their use should be kept to a minimum. There is a treatise on the subject written in the early ‘90’s that I now see is very much relevant, judging by your comment. I wish I could quote it but since I’m using a smartphone to type this it’s too much hassle to find it. Regardless, from a software architecture’s point of view, introducing a GNUism or a Linuxism where it isn’t strictly necessary is an obviously bad idea. How’d you fare with that GCC build on the mailing list from couple of years back, by the way?
      - zaarn 8y ago
        Using #ifdefs to get more out of certain architectures or working around quirks of a platform is definitely not something I'd consider bad practice. As long as it has a default way to fall back to that works everywhere, there is no harm in platform specific code guarded by #ifdefs or other but similar mechanisms provided by the language you use (golang has platform and OS detection!)
        Annatar 8y ago
        They are not a bad practice if their use is kept to a minimum, and if the #ifdef in question is picked out thoughtfully. Dump the built-in preprocessor macros between two different compilers on the same OS and diff the output, it should become immediately clear what I mean. Now go to a different OS and do the same thing and you’ll see that the number of possible combinations of defines to pick explodes, and depending on the situation, the common subset might not even be useful for portability, as in not enough overlap.
        zaarn 8y ago
        Then what is the alternative where I don't loose good integration into OS capabilities while also being portable and not bloating the code?
        Annatar 8y ago
        Where POSIX or XPG6 are insufficient, use core functions from libc, having verified that they exist on all OS targets and that they behave the same, and build the required functionality on top of those.
        zaarn 8y ago
        But then you would not be able to, for example, take advantage of FreeBSD's pledge when available or SELinux to protect your application and the user. Libc doesn't even remotely offer what some libraries want and it leaves out everything that OS can offer on top of POSIX and libc. And what if I want to support Windows, Linux and BSD? Windows doesn't really have a libc and is definitely not even remotely compatible with POSIX.
101km 8y ago
Ah, darn, I'm a few hours late so you will probably miss it but I wanted to ask for a long time: Sounds like a lot of stuff like this has accumulated. Your dedication to backwards compatibility (and testing) is always very impressive, but don't you ever get the urge to do a parallel "SQLite Next" effort as it were? Kind of like python had the 2/3 years.
- timClicks 8y ago
  There was an attempt to re-architect sqlite. Try searching for sqlite4.
  - okket 8y ago
    Here is a discussion about how the sqlite4 experiment ended, from 8 months ago https://news.ycombinator.com/item?id=15648280 https://news.ycombinator.com/item?id=15648280 (157 comments) And the relevant commit (timestamp 2017-10-27): https://sqlite.org/src4/artifact/56683d66cbd41c2e https://sqlite.org/src4/artifact/56683d66cbd41c2e > All development work on SQLite4 has ended. The experiment has concluded. > Lessons learned from SQLite4 have been folded into SQLite3 which continues to be actively maintained and developed. This repository exists as an historical record. There are no plans at this time to resume development of SQLite4.
- tqkxzugoaupvwqr 8y ago
  I read somewhere that the developers of SQLite have a multi-decade support agreement with some big companies. If true, they will probably keep all the accumulated stuff. Any changes would only be additions and never break backwards compatibility.

by SQLite 8y ago

amluto 8y ago

Neat! Is there any reason it’s in a branch? OFD locks fully interoperate with regular POSIX locks, so there shouldn’t be any compatibility issue, except that SQLite would need to fall back on older kernels.

Annatar 8y ago
Since OFD locks are a GNUism, putting that code into the mainline would immediately break portability. I quite like my SQLite databases on illumos and ZFS, so if it couldn’t work on illumos because of a Linux-specific feature, that’d be a serious problem.
- bonzini 8y ago
  That's what #ifdef is for. SQLite has a number of those, given it runs even on platforms which are not POSIX at all!
  - Annatar 8y ago
    I know what they are for and don’t need instruction on that, thank you very much. I also know that #ifdefs are a bad practice with many pitfalls so their use should be kept to a minimum. There is a treatise on the subject written in the early ‘90’s that I now see is very much relevant, judging by your comment. I wish I could quote it but since I’m using a smartphone to type this it’s too much hassle to find it. Regardless, from a software architecture’s point of view, introducing a GNUism or a Linuxism where it isn’t strictly necessary is an obviously bad idea. How’d you fare with that GCC build on the mailing list from couple of years back, by the way?
    - zaarn 8y ago
      Using #ifdefs to get more out of certain architectures or working around quirks of a platform is definitely not something I'd consider bad practice. As long as it has a default way to fall back to that works everywhere, there is no harm in platform specific code guarded by #ifdefs or other but similar mechanisms provided by the language you use (golang has platform and OS detection!)
      - Annatar 8y ago
        They are not a bad practice if their use is kept to a minimum, and if the #ifdef in question is picked out thoughtfully. Dump the built-in preprocessor macros between two different compilers on the same OS and diff the output, it should become immediately clear what I mean. Now go to a different OS and do the same thing and you’ll see that the number of possible combinations of defines to pick explodes, and depending on the situation, the common subset might not even be useful for portability, as in not enough overlap.
        zaarn 8y ago
        Then what is the alternative where I don't loose good integration into OS capabilities while also being portable and not bloating the code?
        Annatar 8y ago
        Where POSIX or XPG6 are insufficient, use core functions from libc, having verified that they exist on all OS targets and that they behave the same, and build the required functionality on top of those.
        zaarn 8y ago
        But then you would not be able to, for example, take advantage of FreeBSD's pledge when available or SELinux to protect your application and the user. Libc doesn't even remotely offer what some libraries want and it leaves out everything that OS can offer on top of POSIX and libc. And what if I want to support Windows, Linux and BSD? Windows doesn't really have a libc and is definitely not even remotely compatible with POSIX.

101km 8y ago

Ah, darn, I'm a few hours late so you will probably miss it but I wanted to ask for a long time: Sounds like a lot of stuff like this has accumulated. Your dedication to backwards compatibility (and testing) is always very impressive, but don't you ever get the urge to do a parallel "SQLite Next" effort as it were? Kind of like python had the 2/3 years.

timClicks 8y ago
There was an attempt to re-architect sqlite. Try searching for sqlite4.
- okket 8y ago
  Here is a discussion about how the sqlite4 experiment ended, from 8 months ago https://news.ycombinator.com/item?id=15648280 https://news.ycombinator.com/item?id=15648280 (157 comments) And the relevant commit (timestamp 2017-10-27): https://sqlite.org/src4/artifact/56683d66cbd41c2e https://sqlite.org/src4/artifact/56683d66cbd41c2e > All development work on SQLite4 has ended. The experiment has concluded. > Lessons learned from SQLite4 have been folded into SQLite3 which continues to be actively maintained and developed. This repository exists as an historical record. There are no plans at this time to resume development of SQLite4.
tqkxzugoaupvwqr 8y ago
I read somewhere that the developers of SQLite have a multi-decade support agreement with some big companies. If true, they will probably keep all the accumulated stuff. Any changes would only be additions and never break backwards compatibility.