10 ms·

Rethinking the filesystem as global mutable state, the root of all evil

103 points by stargrave 7y ago

naasking 7y ago
This was first explained by the capability security community. Plan 9's private nameapaces is an approach to capability secure file systems, with the default being the empty namespace. I'm surprised the article didn't mention Plan 9 actually since it discusses capabilities.
- EdwardCoffin 7y ago
  Plan 9 is mentioned in the article that this one is a follow-on to.
  - ryanjshaw 7y ago
    Plan 9 was mentioned but dismissed as "bizarre and unwieldy" (FWIW, I have no opinion on this). Based on what I read, the author is probably not aware of modern capability-based security research and would benefit from exploring the space further as he appears to be re-discovering some of the concepts but is missing a clear understanding of the broader problems.
- ryanjshaw 7y ago
  I just realised it's 2019 and capabilities are still misunderstood, and the ACL-capability-equivalency myth continues to result in poor solutions to security problems. For anybody who is curious, the general problem here is described in two great papers as "the confused deputy" [1] and "designation without authority" [2]. Roughly put, systems built with ACLs as the primitive mechanism for authorization can never produce practically secure systems. [1] http://zoo.cs.yale.edu/classes/cs422/2010/bib/hardy88confused.pdf http://zoo.cs.yale.edu/classes/cs422/2010/bib/hardy88confuse... [2] http://srl.cs.jhu.edu/pubs/SRL2003-02.pdf http://srl.cs.jhu.edu/pubs/SRL2003-02.pdf
  - boapnuaput 7y ago
    I fear that it's because we don't know how to make globes. [0] [0] https://corbinsimpson.com/words/globe.html https://corbinsimpson.com/words/globe.html
    - ryanjshaw 7y ago
      Reading that felt surprisingly familiar! I pottered about for 2 years trying to build a GUI for a Globe-based world and gave up.
gugagore 7y ago
One thing missing from this rethinking, as far as I can tell, is names. You can't move pointers between different processes in general, because each process has its own "memory address namespace". But an absolute path generated by one program does make sense to another program.
- hlandau 7y ago
  That's true. I mainly cover this point - filesystems as a means of connecting different applications together - in another article: https://www.devever.net/~hl/nexuses https://www.devever.net/~hl/nexuses
mnw21cam 7y ago
As anyone who has put together a chroot knows, it isn't as simple as just preventing a process from accessing the filesystem. Most programs need to access a filesystem, just so they can load libc and the various other libraries it might need.
- AstralStorm 7y ago
  Do they now? Usually, the linker does all the loading, also known in Linux as ld.so. (even when you use dlopen) Lazy loading can thunk through kernel so that ld.so service loads things into the process - and preferably strict loading should be used.
  - mnw21cam 7y ago
    Indeed. But ld.so runs in the context of the process that is starting up, so if it doesn't have access to anything, it can't.
    - AstralStorm 7y ago
      It doesn't actually have to, but you'd need a fun bit of stub code in place of unloaded function pointers etc.
  - pjc50 7y ago
    Not a useful distinction, since it needs to be visible inside the chroot either way. Not only that but libc via getpwnam() and friends needs to access PAM and thence all the libpam modules.
    - mnw21cam 7y ago
      That also depends. I have a chroot set up here to run sftp-server for incoming ssh connections, and it has 55 files in it, 40 of which appear to be libraries - none of them having anything to do with PAM.
- nemaar 7y ago
  I think the idea that the program can load files as code is insane in itself. There shouldn't be such a mechanism in an ideal world. The operating system would decide what you can actually load and load it for you.
pjc50 7y ago
Interesting, and should definitely be read with the companion https://www.devever.net/~hl/objectworld https://www.devever.net/~hl/objectworld Given the same title, I would write a rather different article. So many of our traditional filesystem problems center around concurrency: what kind of read-after-write or durability semantics are guaranteed. A lot of effort goes into this, which is necessary for databases to work on top of a general-purpose filesystem on top of a disintermediated storage system that may be a disk or SSD with a variety of block sizes. But for most operations it's unnecessary overhead. Many games ship with a "packfile", which is a pseudo-filesystem that appears as a giant blob to the OS. Usually it's faster to seek individual small items out of the blob than if they were separate files. Further to that is the security problem; we've moved from "apps are all trusted, but you need to watch the files closely between multiple users" to "there's only one user, but you can't trust the apps". Note how the cloud has avoided both of these by moving storage to three different areas with different semantics. Apps can speak to "blob storage" (S3) which has a transactionality/security granularity of one blob. Or they can speak to a database (which has intelligence of its own), or separate raw block storage if neither of those suits. What if we moved from "everything is a file" to "everything is a URL"? Possibly adding a system-default packfile mechanism. So an app would be allowed access to everything that it ships with, but nothing outside except what it could request as URLs with various sorts of security mechanism.
- pixl97 7y ago
  >What if we moved from "everything is a file" to "everything is a URL Not sure what you're accomplishing exactly. Something would have to serve said url and you are moving your security problems into it. Mostly you can accomplish many of the same things with file security, with some difficulty.
- AnimalMuppet 7y ago
  > What if we moved from "everything is a file" to "everything is a URL"? Do you mean "everything not on my machine is a URL"? We're getting somewhat close to that already. Or do you mean "everything on my machine is a URL"? Then I don't see how it's different from "everything is a file with a full path", except that someone else is now responsible for the security mechanism. "Everything is a URL" would give you a unified view of on-machine and off-machine resources...
ncmncm 7y ago
This is a very important idea, but it will take a long time to become practical for typical system design environments. Best get started.
x3ro 7y ago
This reminded me of this [1] talk from 32c3, which talks about the same issues and presents and overview of some approaches ppl have come up with. Somewhat BSD centric, but I found it interesting back then. [1]: https://media.ccc.de/v/32c3-7231-cloudabi https://media.ccc.de/v/32c3-7231-cloudabi
mrfredward 7y ago
Every piece of hardware on a computer is a piece of global mutable state. There are many good reasons to hide that fact behind an abstraction, but I can't help but think hiding global mutability is better handled at the application level than the OS level, because there are too many cases where the abstraction becomes extremely limiting. As an example, most people would want to be able to import an image into a word processor regardless of where that image is located (local drive, network drive, floppy disk, etc.). To support that, most end user programs would want to be offered access to the entire filesystem. The moment two applications do this at once, you have all the shared mutable state problems we do now.
- satyenr 7y ago
  It doesn't have to be that way. Consider the following scenario: 1. Every file is owned by application. 2. A file-store application is the custodian of files meant to be used by multiple applications. 3. An application that needs to edit a file must take ownership of the file for the duration of the editing. 4. An application that needs to read a file must borrow the files. 5. A file can be borrowed by multiple applications, but owned by only one. 6. Applications can provide the ability for shared ownership, but those would by specialized applications capable of handling and merging simultaneous changes. This is a straw-man solution, and I am sure multiple problems will have to be solved before such a system can become a reality. > As an example, most people would want to be able to import an image into a word processor regardless of where that image is located (local drive, network drive, floppy disk, etc.). To support that, most end user programs would want to be offered access to the entire filesystem. The moment two applications do this at once, you have all the shared mutable state problems we do now. I don't see why the custodian of photos can't be a photo management application. In fact, a filesystem is the lowest common denominator. It is possible to build higher level abstractions like PhotoStore, MusicStore, MovieStore, CodeStore etc. which accommodate and make use of the properties of individual data types to offer an enhanced experience.
  - Retra 7y ago
    I have photos on my computer, even though I don't use a photo management application. I have compressed files. I have files taken from other machines. I remove applications. I create my own file formats. Applications themselves are files. Who owns all the unassigned stuff? The file system. Give it a different name and you change nothing.
perlgeek 7y ago
This is a fascinating idea, and totally obvious in retrospect :-) I've noticed features that work towards cutting off access from global file system. Containers are mentioned in the article, but there are other things like systemd giving easy access to per-service private temp files, blocking access to /home, whitelisting paths that are allowed at all etc. The problem really is that this global mutable state has existed for 50+ years, so lots and lots of things now rely on it: dynamic loading of libraries (could be separate objects from files), configuration (doesn't need to be stored in files), UNIX sockets that are accessed by file name etc. And not only have many programs started to rely on files, but many have relied on them being mutable. Webservers can be told to re-read their config from a running process, things like that. So, is there any way forward for UNIX-based systems that don't break the world? Edit: another interesting observation is that iOS has already done away with the global file system, at least from the perspective of the apps.
specialp 7y ago
Web assembly is a good step in the elimination of the globally shared FS. Dynamically linked libraries are explicitly provided with an index only that links to the library location. https://webassembly.org/docs/dynamic-linking/ https://webassembly.org/docs/dynamic-linking/ WASI https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/ https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webas... will provide access to files given to the function by the runtime, not vice versa as in operating systems where a syscall is sent to ask permission for a file.
JoeAltmaier 7y ago
I've thought for years that 'file systems' are an abomination. We use a cobbled-together schema of parent-directory, filename, some random date/time stamps and maybe a three-letter extension. Why? Because we inheirited that from some DOS days. Why not a collection of immutable UUID-labelled resources with an arbitrary schema of attributes? Like a relational database or some such. Overkill? Doesn't every major app have to do this itself, in some container (document/mail folder/image definition/contact list and on and on)? Designers of operating systems have made no effort to capture this kind of service and provide it as a fundamental OS feature. And OSs are exactly where this belongs - where you can carefully get it right and everybody writing apps can depend on it.
- AnIdiotOnTheNet 7y ago
  > Designers of operating systems have made no effort to capture this kind of service and provide it as a fundamental OS feature. Actually they've made several. See BeOS and WinFS, for starters, though the idea is even older than that.
- liara_k 7y ago
  I've had very similar ideas, and think this is really critical missing feature that's holding back a lot of progress. IMHO, what's needed is something like a document database (in that it allows arbitrary schema) but even more importantly, it needs to have some system-level indexing. Right now, applications that deal with large amounts of small pieces of data each roll their own, e.g., embedded database, because (1) most filesystems aren't capable of adequate performance with very large numbers of small objects and (2) you can't maintain an app-independent index anyway, so... These applications then can't interact with each other without some app-specific API, which is often costly to develop against (at least, compared with a common filesystem API) and with only the capabilities that the app developer sees fit to give you. Which they often have very little incentive to do--their real incentives are usually to keep things proprietary and customers captive. The absence of these two capabilities makes "the unix way" where "everything is a file" an impossible data model for these types of applications. IMHO, the right data model is probably some combination of user (UUIDs or strings doesn't matter) and content-hash indexing with versioning and conflict resolution similar to Git (and CouchDB).
asimpletune 7y ago
Isn’t this sort of how sandboxing iOS apps already feels like?
pfdietz 7y ago
> root of all evil I see what you did there.
skdotdan 7y ago
One thing that has always astounded me is the fact that the following script can be executed without any special permission (apart from execution, read and write): rm -rf ~ A key to a more modern file system would be a much more granular permission systems apart from "rwx" for a certain group of users.
- nybble41 7y ago
  What permissions do you think that command should require? You could add a separate "delete" capability, but the fact that you have permission to write to the file already means that you can delete or overwrite its contents, which is effectively the same from the user's point of view. (Granted, the fact that you can delete a read-only file if you have write permission on the directory is a bit odd, but that situation doesn't come up often.) I don't think more granular permission can solve this problem. What does solve it, however, is read-only copy-on-write snapshots. No matter how files are renamed, deleted, or modified, you can always recover the original version from the snapshot.

103 points by stargrave 7y ago

This was first explained by the capability security community. Plan 9's private nameapaces is an approach to capability secure file systems, with the default being the empty namespace. I'm surprised the article didn't mention Plan 9 actually since it discusses capabilities.

EdwardCoffin 7y ago
Plan 9 is mentioned in the article that this one is a follow-on to.
- ryanjshaw 7y ago
  Plan 9 was mentioned but dismissed as "bizarre and unwieldy" (FWIW, I have no opinion on this). Based on what I read, the author is probably not aware of modern capability-based security research and would benefit from exploring the space further as he appears to be re-discovering some of the concepts but is missing a clear understanding of the broader problems.
ryanjshaw 7y ago
I just realised it's 2019 and capabilities are still misunderstood, and the ACL-capability-equivalency myth continues to result in poor solutions to security problems. For anybody who is curious, the general problem here is described in two great papers as "the confused deputy" [1] and "designation without authority" [2]. Roughly put, systems built with ACLs as the primitive mechanism for authorization can never produce practically secure systems. [1] http://zoo.cs.yale.edu/classes/cs422/2010/bib/hardy88confused.pdf http://zoo.cs.yale.edu/classes/cs422/2010/bib/hardy88confuse... [2] http://srl.cs.jhu.edu/pubs/SRL2003-02.pdf http://srl.cs.jhu.edu/pubs/SRL2003-02.pdf
- boapnuaput 7y ago
  I fear that it's because we don't know how to make globes. [0] [0] https://corbinsimpson.com/words/globe.html https://corbinsimpson.com/words/globe.html
  - ryanjshaw 7y ago
    Reading that felt surprisingly familiar! I pottered about for 2 years trying to build a GUI for a Globe-based world and gave up.

gugagore 7y ago

One thing missing from this rethinking, as far as I can tell, is names. You can't move pointers between different processes in general, because each process has its own "memory address namespace". But an absolute path generated by one program does make sense to another program.

hlandau 7y ago
That's true. I mainly cover this point - filesystems as a means of connecting different applications together - in another article: https://www.devever.net/~hl/nexuses https://www.devever.net/~hl/nexuses

mnw21cam 7y ago

As anyone who has put together a chroot knows, it isn't as simple as just preventing a process from accessing the filesystem. Most programs need to access a filesystem, just so they can load libc and the various other libraries it might need.

AstralStorm 7y ago
Do they now? Usually, the linker does all the loading, also known in Linux as ld.so. (even when you use dlopen) Lazy loading can thunk through kernel so that ld.so service loads things into the process - and preferably strict loading should be used.
- mnw21cam 7y ago
  Indeed. But ld.so runs in the context of the process that is starting up, so if it doesn't have access to anything, it can't.
  - AstralStorm 7y ago
    It doesn't actually have to, but you'd need a fun bit of stub code in place of unloaded function pointers etc.
- pjc50 7y ago
  Not a useful distinction, since it needs to be visible inside the chroot either way. Not only that but libc via getpwnam() and friends needs to access PAM and thence all the libpam modules.
  - mnw21cam 7y ago
    That also depends. I have a chroot set up here to run sftp-server for incoming ssh connections, and it has 55 files in it, 40 of which appear to be libraries - none of them having anything to do with PAM.
nemaar 7y ago
I think the idea that the program can load files as code is insane in itself. There shouldn't be such a mechanism in an ideal world. The operating system would decide what you can actually load and load it for you.

pjc50 7y ago

Interesting, and should definitely be read with the companion https://www.devever.net/~hl/objectworld https://www.devever.net/~hl/objectworld Given the same title, I would write a rather different article. So many of our traditional filesystem problems center around concurrency: what kind of read-after-write or durability semantics are guaranteed. A lot of effort goes into this, which is necessary for databases to work on top of a general-purpose filesystem on top of a disintermediated storage system that may be a disk or SSD with a variety of block sizes. But for most operations it's unnecessary overhead. Many games ship with a "packfile", which is a pseudo-filesystem that appears as a giant blob to the OS. Usually it's faster to seek individual small items out of the blob than if they were separate files. Further to that is the security problem; we've moved from "apps are all trusted, but you need to watch the files closely between multiple users" to "there's only one user, but you can't trust the apps". Note how the cloud has avoided both of these by moving storage to three different areas with different semantics. Apps can speak to "blob storage" (S3) which has a transactionality/security granularity of one blob. Or they can speak to a database (which has intelligence of its own), or separate raw block storage if neither of those suits. What if we moved from "everything is a file" to "everything is a URL"? Possibly adding a system-default packfile mechanism. So an app would be allowed access to everything that it ships with, but nothing outside except what it could request as URLs with various sorts of security mechanism.

pixl97 7y ago
>What if we moved from "everything is a file" to "everything is a URL Not sure what you're accomplishing exactly. Something would have to serve said url and you are moving your security problems into it. Mostly you can accomplish many of the same things with file security, with some difficulty.
AnimalMuppet 7y ago
> What if we moved from "everything is a file" to "everything is a URL"? Do you mean "everything not on my machine is a URL"? We're getting somewhat close to that already. Or do you mean "everything on my machine is a URL"? Then I don't see how it's different from "everything is a file with a full path", except that someone else is now responsible for the security mechanism. "Everything is a URL" would give you a unified view of on-machine and off-machine resources...

ncmncm 7y ago

This is a very important idea, but it will take a long time to become practical for typical system design environments. Best get started.

x3ro 7y ago

This reminded me of this [1] talk from 32c3, which talks about the same issues and presents and overview of some approaches ppl have come up with. Somewhat BSD centric, but I found it interesting back then. [1]: https://media.ccc.de/v/32c3-7231-cloudabi https://media.ccc.de/v/32c3-7231-cloudabi

mrfredward 7y ago

Every piece of hardware on a computer is a piece of global mutable state. There are many good reasons to hide that fact behind an abstraction, but I can't help but think hiding global mutability is better handled at the application level than the OS level, because there are too many cases where the abstraction becomes extremely limiting. As an example, most people would want to be able to import an image into a word processor regardless of where that image is located (local drive, network drive, floppy disk, etc.). To support that, most end user programs would want to be offered access to the entire filesystem. The moment two applications do this at once, you have all the shared mutable state problems we do now.

satyenr 7y ago
It doesn't have to be that way. Consider the following scenario: 1. Every file is owned by application. 2. A file-store application is the custodian of files meant to be used by multiple applications. 3. An application that needs to edit a file must take ownership of the file for the duration of the editing. 4. An application that needs to read a file must borrow the files. 5. A file can be borrowed by multiple applications, but owned by only one. 6. Applications can provide the ability for shared ownership, but those would by specialized applications capable of handling and merging simultaneous changes. This is a straw-man solution, and I am sure multiple problems will have to be solved before such a system can become a reality. > As an example, most people would want to be able to import an image into a word processor regardless of where that image is located (local drive, network drive, floppy disk, etc.). To support that, most end user programs would want to be offered access to the entire filesystem. The moment two applications do this at once, you have all the shared mutable state problems we do now. I don't see why the custodian of photos can't be a photo management application. In fact, a filesystem is the lowest common denominator. It is possible to build higher level abstractions like PhotoStore, MusicStore, MovieStore, CodeStore etc. which accommodate and make use of the properties of individual data types to offer an enhanced experience.
- Retra 7y ago
  I have photos on my computer, even though I don't use a photo management application. I have compressed files. I have files taken from other machines. I remove applications. I create my own file formats. Applications themselves are files. Who owns all the unassigned stuff? The file system. Give it a different name and you change nothing.

perlgeek 7y ago

This is a fascinating idea, and totally obvious in retrospect :-) I've noticed features that work towards cutting off access from global file system. Containers are mentioned in the article, but there are other things like systemd giving easy access to per-service private temp files, blocking access to /home, whitelisting paths that are allowed at all etc. The problem really is that this global mutable state has existed for 50+ years, so lots and lots of things now rely on it: dynamic loading of libraries (could be separate objects from files), configuration (doesn't need to be stored in files), UNIX sockets that are accessed by file name etc. And not only have many programs started to rely on files, but many have relied on them being mutable. Webservers can be told to re-read their config from a running process, things like that. So, is there any way forward for UNIX-based systems that don't break the world? Edit: another interesting observation is that iOS has already done away with the global file system, at least from the perspective of the apps.

specialp 7y ago

Web assembly is a good step in the elimination of the globally shared FS. Dynamically linked libraries are explicitly provided with an index only that links to the library location. https://webassembly.org/docs/dynamic-linking/ https://webassembly.org/docs/dynamic-linking/ WASI https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/ https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webas... will provide access to files given to the function by the runtime, not vice versa as in operating systems where a syscall is sent to ask permission for a file.

JoeAltmaier 7y ago

I've thought for years that 'file systems' are an abomination. We use a cobbled-together schema of parent-directory, filename, some random date/time stamps and maybe a three-letter extension. Why? Because we inheirited that from some DOS days. Why not a collection of immutable UUID-labelled resources with an arbitrary schema of attributes? Like a relational database or some such. Overkill? Doesn't every major app have to do this itself, in some container (document/mail folder/image definition/contact list and on and on)? Designers of operating systems have made no effort to capture this kind of service and provide it as a fundamental OS feature. And OSs are exactly where this belongs - where you can carefully get it right and everybody writing apps can depend on it.

AnIdiotOnTheNet 7y ago
> Designers of operating systems have made no effort to capture this kind of service and provide it as a fundamental OS feature. Actually they've made several. See BeOS and WinFS, for starters, though the idea is even older than that.
liara_k 7y ago
I've had very similar ideas, and think this is really critical missing feature that's holding back a lot of progress. IMHO, what's needed is something like a document database (in that it allows arbitrary schema) but even more importantly, it needs to have some system-level indexing. Right now, applications that deal with large amounts of small pieces of data each roll their own, e.g., embedded database, because (1) most filesystems aren't capable of adequate performance with very large numbers of small objects and (2) you can't maintain an app-independent index anyway, so... These applications then can't interact with each other without some app-specific API, which is often costly to develop against (at least, compared with a common filesystem API) and with only the capabilities that the app developer sees fit to give you. Which they often have very little incentive to do--their real incentives are usually to keep things proprietary and customers captive. The absence of these two capabilities makes "the unix way" where "everything is a file" an impossible data model for these types of applications. IMHO, the right data model is probably some combination of user (UUIDs or strings doesn't matter) and content-hash indexing with versioning and conflict resolution similar to Git (and CouchDB).

asimpletune 7y ago

Isn’t this sort of how sandboxing iOS apps already feels like?

pfdietz 7y ago

> root of all evil I see what you did there.

skdotdan 7y ago

One thing that has always astounded me is the fact that the following script can be executed without any special permission (apart from execution, read and write): rm -rf ~ A key to a more modern file system would be a much more granular permission systems apart from "rwx" for a certain group of users.

nybble41 7y ago
What permissions do you think that command should require? You could add a separate "delete" capability, but the fact that you have permission to write to the file already means that you can delete or overwrite its contents, which is effectively the same from the user's point of view. (Granted, the fact that you can delete a read-only file if you have write permission on the directory is a bit odd, but that situation doesn't come up often.) I don't think more granular permission can solve this problem. What does solve it, however, is read-only copy-on-write snapshots. No matter how files are renamed, deleted, or modified, you can always recover the original version from the snapshot.