4 ms·

The problem is that the POSIX style API is too low level to do easily do anything interesting and correct. Enter rocksdb, sqlite, etc.

coldtea 8y ago
Is that a problem though? It also means that POSIX doesn't constrict you -- it gives you the tools to build rocksdb, sqlite, etc and have them be performant.
- heinrichhartman 8y ago
  I hear there is quite some complexity in sqlite to work around quirks of the file system API, in order to avoid data loss, yet there are still ways this can happen: https://sqlite.org/howtocorrupt.html https://sqlite.org/howtocorrupt.html It's a bad sign if you have to repair the shortcomings of a lower layer in your application.
  - mmt 8y ago
    How many of those characteristics identified on that page are really shortcomings rather than, as the parent comment says, freedom-enhancing (if not performance-enhancing) features? The workarounds seem to have more to do with SQLite's use as an embedded library than with filesystem behavior. All of section 1 is about rogue overwriting. Absent a particularly fine-grained filesystem permission model, or enforced locking. The former might only have "freedom" implications without any performance penalty, but perhaps not the latter. Section 2 is about locking, so see above. Even if it's advisory, that just moves any slowdown from the kernel to the application. Of course, broken locking is clearly a shortcoming, and this is one of the quirkiest (maybe newest?) aspects of the filesystem. If anything qualifies under the "bad sign" characterization, this does. However, it's also not exactly fundamental to the filesystem, as even section 2.3 describes what SQLite will do if POSIX adivsory locking is not supported over NFS (i.e. it is possible for locking to be unavailable). The remaining sections all appear to be about bugs, either hardware, thread library, elsewhere in the kernel, SQLite itself, or in configuration/usage (disabling corruption protections).
- SamReidHughes 8y ago
  POSIX does constrict you. That is, it doesn't let you get the full performance you could get by talking directly to the hardware. It doesn't offer abstract features that would be useful to people making rocksdb, sqlite, etc. In fact, it does a rather brutal job of constricting you. Of course, disk drive protocols also constrict you, relative to having direct control of firmware and visibility over raw flash devices, drive head position, etc. For example, it would be neat if you could send block A and block B to the disk at the same time, only to have the disk compute the checksum of block A and put it at offset k of block B before writing block B.
  - coldtea 8y ago
    >POSIX does constrict you. That is, it doesn't let you get the full performance you could get by talking directly to the hardware. If it gave you that, it wouldn't be a standard anymore, just a bunch of instructions to talk to hardware. So, of course it constricts you in some degree, but way lesser than the higher order abstractions the parent talk about.
    - SamReidHughes 8y ago
      Well it's more than just not being a bunch of instructions. It doesn't give you a ton of not hardware specific stuff that would be useful.
- fnord123 8y ago
  The difficulty in getting things correct is a constriction for users of the API. And the overly strong API guarantees is a constriction on implementer. POSIX I/O is like C. It's super powerful. It's everywhere. It's not going away. But it has sharp edges. And we can sacrifice a little efficiency on the small scale to grab huge gains in other areas like productivity and reliability. Also relevant is Glenn Lockwood's excellent article on What's bad about POSIX I/O: https://www.nextplatform.com/2017/09/11/whats-bad-posix-io/ https://www.nextplatform.com/2017/09/11/whats-bad-posix-io/
amyjess 8y ago
I agree. In fact, I find myself looking at Classic Mac OS's resource fork and feeling profoundly sad that it feel by the wayside as *nix took over the world. The resource fork was a beautiful idea; structured data built into the filesystem itself, and I wonder what the computing world would have looked like if it was used by the dominant operating system of the Internet instead of a sideshow.
- fjsolwmv 8y ago
  Why is the resource fork better than having say a foo.rsrc file alongside foo? Or operating on directories instead of files, like MacOSX bundles (like .app and .rtfd)?

coldtea 8y ago

Is that a problem though? It also means that POSIX doesn't constrict you -- it gives you the tools to build rocksdb, sqlite, etc and have them be performant.

heinrichhartman 8y ago
I hear there is quite some complexity in sqlite to work around quirks of the file system API, in order to avoid data loss, yet there are still ways this can happen: https://sqlite.org/howtocorrupt.html https://sqlite.org/howtocorrupt.html It's a bad sign if you have to repair the shortcomings of a lower layer in your application.
- mmt 8y ago
  How many of those characteristics identified on that page are really shortcomings rather than, as the parent comment says, freedom-enhancing (if not performance-enhancing) features? The workarounds seem to have more to do with SQLite's use as an embedded library than with filesystem behavior. All of section 1 is about rogue overwriting. Absent a particularly fine-grained filesystem permission model, or enforced locking. The former might only have "freedom" implications without any performance penalty, but perhaps not the latter. Section 2 is about locking, so see above. Even if it's advisory, that just moves any slowdown from the kernel to the application. Of course, broken locking is clearly a shortcoming, and this is one of the quirkiest (maybe newest?) aspects of the filesystem. If anything qualifies under the "bad sign" characterization, this does. However, it's also not exactly fundamental to the filesystem, as even section 2.3 describes what SQLite will do if POSIX adivsory locking is not supported over NFS (i.e. it is possible for locking to be unavailable). The remaining sections all appear to be about bugs, either hardware, thread library, elsewhere in the kernel, SQLite itself, or in configuration/usage (disabling corruption protections).
SamReidHughes 8y ago
POSIX does constrict you. That is, it doesn't let you get the full performance you could get by talking directly to the hardware. It doesn't offer abstract features that would be useful to people making rocksdb, sqlite, etc. In fact, it does a rather brutal job of constricting you. Of course, disk drive protocols also constrict you, relative to having direct control of firmware and visibility over raw flash devices, drive head position, etc. For example, it would be neat if you could send block A and block B to the disk at the same time, only to have the disk compute the checksum of block A and put it at offset k of block B before writing block B.
- coldtea 8y ago
  >POSIX does constrict you. That is, it doesn't let you get the full performance you could get by talking directly to the hardware. If it gave you that, it wouldn't be a standard anymore, just a bunch of instructions to talk to hardware. So, of course it constricts you in some degree, but way lesser than the higher order abstractions the parent talk about.
  - SamReidHughes 8y ago
    Well it's more than just not being a bunch of instructions. It doesn't give you a ton of not hardware specific stuff that would be useful.
fnord123 8y ago
The difficulty in getting things correct is a constriction for users of the API. And the overly strong API guarantees is a constriction on implementer. POSIX I/O is like C. It's super powerful. It's everywhere. It's not going away. But it has sharp edges. And we can sacrifice a little efficiency on the small scale to grab huge gains in other areas like productivity and reliability. Also relevant is Glenn Lockwood's excellent article on What's bad about POSIX I/O: https://www.nextplatform.com/2017/09/11/whats-bad-posix-io/ https://www.nextplatform.com/2017/09/11/whats-bad-posix-io/

amyjess 8y ago

I agree. In fact, I find myself looking at Classic Mac OS's resource fork and feeling profoundly sad that it feel by the wayside as *nix took over the world. The resource fork was a beautiful idea; structured data built into the filesystem itself, and I wonder what the computing world would have looked like if it was used by the dominant operating system of the Internet instead of a sideshow.

fjsolwmv 8y ago
Why is the resource fork better than having say a foo.rsrc file alongside foo? Or operating on directories instead of files, like MacOSX bundles (like .app and .rtfd)?