5 ms·

> The whole point with "everything is a file" is not that you have some random filename (indeed, sockets and pipes show that "file" and "filename" have nothing

> The whole point with "everything is a file" is not that you have some random filename (indeed, sockets and pipes show that "file" and "filename" have nothing to do with each other), but the fact that you can use common tools to operate on different things.

fnord123 8y ago
The problem is that the POSIX style API is too low level to do easily do anything interesting and correct. Enter rocksdb, sqlite, etc.
- coldtea 8y ago
  Is that a problem though? It also means that POSIX doesn't constrict you -- it gives you the tools to build rocksdb, sqlite, etc and have them be performant.
  - heinrichhartman 8y ago
    I hear there is quite some complexity in sqlite to work around quirks of the file system API, in order to avoid data loss, yet there are still ways this can happen: https://sqlite.org/howtocorrupt.html https://sqlite.org/howtocorrupt.html It's a bad sign if you have to repair the shortcomings of a lower layer in your application.
    - mmt 8y ago
      How many of those characteristics identified on that page are really shortcomings rather than, as the parent comment says, freedom-enhancing (if not performance-enhancing) features? The workarounds seem to have more to do with SQLite's use as an embedded library than with filesystem behavior. All of section 1 is about rogue overwriting. Absent a particularly fine-grained filesystem permission model, or enforced locking. The former might only have "freedom" implications without any performance penalty, but perhaps not the latter. Section 2 is about locking, so see above. Even if it's advisory, that just moves any slowdown from the kernel to the application. Of course, broken locking is clearly a shortcoming, and this is one of the quirkiest (maybe newest?) aspects of the filesystem. If anything qualifies under the "bad sign" characterization, this does. However, it's also not exactly fundamental to the filesystem, as even section 2.3 describes what SQLite will do if POSIX adivsory locking is not supported over NFS (i.e. it is possible for locking to be unavailable). The remaining sections all appear to be about bugs, either hardware, thread library, elsewhere in the kernel, SQLite itself, or in configuration/usage (disabling corruption protections).
  - SamReidHughes 8y ago
    POSIX does constrict you. That is, it doesn't let you get the full performance you could get by talking directly to the hardware. It doesn't offer abstract features that would be useful to people making rocksdb, sqlite, etc. In fact, it does a rather brutal job of constricting you. Of course, disk drive protocols also constrict you, relative to having direct control of firmware and visibility over raw flash devices, drive head position, etc. For example, it would be neat if you could send block A and block B to the disk at the same time, only to have the disk compute the checksum of block A and put it at offset k of block B before writing block B.
    - coldtea 8y ago
      >POSIX does constrict you. That is, it doesn't let you get the full performance you could get by talking directly to the hardware. If it gave you that, it wouldn't be a standard anymore, just a bunch of instructions to talk to hardware. So, of course it constricts you in some degree, but way lesser than the higher order abstractions the parent talk about.
      - SamReidHughes 8y ago
        Well it's more than just not being a bunch of instructions. It doesn't give you a ton of not hardware specific stuff that would be useful.
  - fnord123 8y ago
    The difficulty in getting things correct is a constriction for users of the API. And the overly strong API guarantees is a constriction on implementer. POSIX I/O is like C. It's super powerful. It's everywhere. It's not going away. But it has sharp edges. And we can sacrifice a little efficiency on the small scale to grab huge gains in other areas like productivity and reliability. Also relevant is Glenn Lockwood's excellent article on What's bad about POSIX I/O: https://www.nextplatform.com/2017/09/11/whats-bad-posix-io/ https://www.nextplatform.com/2017/09/11/whats-bad-posix-io/
- amyjess 8y ago
  I agree. In fact, I find myself looking at Classic Mac OS's resource fork and feeling profoundly sad that it feel by the wayside as *nix took over the world. The resource fork was a beautiful idea; structured data built into the filesystem itself, and I wonder what the computing world would have looked like if it was used by the dominant operating system of the Internet instead of a sideshow.
  - fjsolwmv 8y ago
    Why is the resource fork better than having say a foo.rsrc file alongside foo? Or operating on directories instead of files, like MacOSX bundles (like .app and .rtfd)?
perl4ever 8y ago
Being able to operate on everything with the same tools is not a goal that everyone desires all the time. For instance, look at all the things with torx screws instead of phillips heads. Or, type systems in computer languages. Or, support for non-variadic functions. Or, file systems with forks.
- simongr3dal 8y ago
  This might just be because I'm european, but phillips head screws really are the worst. They are so easy to strip because it is so easy to use an over- or undersized screwdriver. Torx or Hex screws are good because you can't really use the wrong sized screwdriver and the screws doesn't get stripped as easily.
  - zxcmx 8y ago
    “you must have the correct tool” vs “if you have to, you can open this with a butter knife from the kitchen drawer”. They’ve both got their place IMHO and vive-la-difference :)
  - rascul 8y ago
    I'm an American (of the U.S. variety) and I've been using and hating phillips head screws all my life. I agree with everything you said.
  - fjsolwmv 8y ago
    Phillips was specifically designed so that cheap screws (or drivers) would take damage instead of the expensive thing being screwed into. http://www.acontinuouslean.com/2013/04/05/a-better-way-to-screw/ http://www.acontinuouslean.com/2013/04/05/a-better-way-to-sc...
    - mmt 8y ago
      My read on it is that it was to avoid anything taking damage, especially the screw (and screw head, other than, perhaps, the slots themselves, insofar as they could be used for future driving purposes).
pjmlp 8y ago
Try to apply the file concept to graphics programming. You won't get Crysis running out of it.
- ori_b 8y ago
  Linux already uses file operations like mmap with a few extra ioctls to implement graphics.
- coldtea 8y ago
  If I am not mistaken, this is not unlike how all game graphics were done in the 80s... directly writing to some frame buffer...
  - theoh 8y ago
    The point is that writing to a file is basically passing a message to the kernel. Writing to a framebuffer is writing straight into memory.
    - nly 8y ago
      So mmap() the file?
      - ninkendo 8y ago
        In other words, don't treat it as a file any more?
        JdeBP 8y ago
        Only if one also opines that memory mapping is a thing that one cannot generally do to files. (-:
        zbentley 8y ago
        If we posit that memory mapping is a thing that one can generally do to files, then a ton of other abstractions break: how do you memory map a socket? Not a fixed piece of a socket's buffer, but communication on the socket itself?

fnord123 8y ago

The problem is that the POSIX style API is too low level to do easily do anything interesting and correct. Enter rocksdb, sqlite, etc.

coldtea 8y ago
Is that a problem though? It also means that POSIX doesn't constrict you -- it gives you the tools to build rocksdb, sqlite, etc and have them be performant.
- heinrichhartman 8y ago
  I hear there is quite some complexity in sqlite to work around quirks of the file system API, in order to avoid data loss, yet there are still ways this can happen: https://sqlite.org/howtocorrupt.html https://sqlite.org/howtocorrupt.html It's a bad sign if you have to repair the shortcomings of a lower layer in your application.
  - mmt 8y ago
    How many of those characteristics identified on that page are really shortcomings rather than, as the parent comment says, freedom-enhancing (if not performance-enhancing) features? The workarounds seem to have more to do with SQLite's use as an embedded library than with filesystem behavior. All of section 1 is about rogue overwriting. Absent a particularly fine-grained filesystem permission model, or enforced locking. The former might only have "freedom" implications without any performance penalty, but perhaps not the latter. Section 2 is about locking, so see above. Even if it's advisory, that just moves any slowdown from the kernel to the application. Of course, broken locking is clearly a shortcoming, and this is one of the quirkiest (maybe newest?) aspects of the filesystem. If anything qualifies under the "bad sign" characterization, this does. However, it's also not exactly fundamental to the filesystem, as even section 2.3 describes what SQLite will do if POSIX adivsory locking is not supported over NFS (i.e. it is possible for locking to be unavailable). The remaining sections all appear to be about bugs, either hardware, thread library, elsewhere in the kernel, SQLite itself, or in configuration/usage (disabling corruption protections).
- SamReidHughes 8y ago
  POSIX does constrict you. That is, it doesn't let you get the full performance you could get by talking directly to the hardware. It doesn't offer abstract features that would be useful to people making rocksdb, sqlite, etc. In fact, it does a rather brutal job of constricting you. Of course, disk drive protocols also constrict you, relative to having direct control of firmware and visibility over raw flash devices, drive head position, etc. For example, it would be neat if you could send block A and block B to the disk at the same time, only to have the disk compute the checksum of block A and put it at offset k of block B before writing block B.
  - coldtea 8y ago
    >POSIX does constrict you. That is, it doesn't let you get the full performance you could get by talking directly to the hardware. If it gave you that, it wouldn't be a standard anymore, just a bunch of instructions to talk to hardware. So, of course it constricts you in some degree, but way lesser than the higher order abstractions the parent talk about.
    - SamReidHughes 8y ago
      Well it's more than just not being a bunch of instructions. It doesn't give you a ton of not hardware specific stuff that would be useful.
- fnord123 8y ago
  The difficulty in getting things correct is a constriction for users of the API. And the overly strong API guarantees is a constriction on implementer. POSIX I/O is like C. It's super powerful. It's everywhere. It's not going away. But it has sharp edges. And we can sacrifice a little efficiency on the small scale to grab huge gains in other areas like productivity and reliability. Also relevant is Glenn Lockwood's excellent article on What's bad about POSIX I/O: https://www.nextplatform.com/2017/09/11/whats-bad-posix-io/ https://www.nextplatform.com/2017/09/11/whats-bad-posix-io/
amyjess 8y ago
I agree. In fact, I find myself looking at Classic Mac OS's resource fork and feeling profoundly sad that it feel by the wayside as *nix took over the world. The resource fork was a beautiful idea; structured data built into the filesystem itself, and I wonder what the computing world would have looked like if it was used by the dominant operating system of the Internet instead of a sideshow.
- fjsolwmv 8y ago
  Why is the resource fork better than having say a foo.rsrc file alongside foo? Or operating on directories instead of files, like MacOSX bundles (like .app and .rtfd)?

perl4ever 8y ago

Being able to operate on everything with the same tools is not a goal that everyone desires all the time. For instance, look at all the things with torx screws instead of phillips heads. Or, type systems in computer languages. Or, support for non-variadic functions. Or, file systems with forks.

simongr3dal 8y ago
This might just be because I'm european, but phillips head screws really are the worst. They are so easy to strip because it is so easy to use an over- or undersized screwdriver. Torx or Hex screws are good because you can't really use the wrong sized screwdriver and the screws doesn't get stripped as easily.
- zxcmx 8y ago
  “you must have the correct tool” vs “if you have to, you can open this with a butter knife from the kitchen drawer”. They’ve both got their place IMHO and vive-la-difference :)
- rascul 8y ago
  I'm an American (of the U.S. variety) and I've been using and hating phillips head screws all my life. I agree with everything you said.
- fjsolwmv 8y ago
  Phillips was specifically designed so that cheap screws (or drivers) would take damage instead of the expensive thing being screwed into. http://www.acontinuouslean.com/2013/04/05/a-better-way-to-screw/ http://www.acontinuouslean.com/2013/04/05/a-better-way-to-sc...
  - mmt 8y ago
    My read on it is that it was to avoid anything taking damage, especially the screw (and screw head, other than, perhaps, the slots themselves, insofar as they could be used for future driving purposes).

pjmlp 8y ago

Try to apply the file concept to graphics programming. You won't get Crysis running out of it.

ori_b 8y ago
Linux already uses file operations like mmap with a few extra ioctls to implement graphics.
coldtea 8y ago
If I am not mistaken, this is not unlike how all game graphics were done in the 80s... directly writing to some frame buffer...
- theoh 8y ago
  The point is that writing to a file is basically passing a message to the kernel. Writing to a framebuffer is writing straight into memory.
  - nly 8y ago
    So mmap() the file?
    - ninkendo 8y ago
      In other words, don't treat it as a file any more?
      - JdeBP 8y ago
        Only if one also opines that memory mapping is a thing that one cannot generally do to files. (-:
        zbentley 8y ago
        If we posit that memory mapping is a thing that one can generally do to files, then a ton of other abstractions break: how do you memory map a socket? Not a fixed piece of a socket's buffer, but communication on the socket itself?