7 ms·

I would like to see an updated file system architecture that's closer to a database, with tagging and all that. And I could see that extending to processes too.

I would like to see an updated file system architecture that's closer to a database, with tagging and all that. And I could see that extending to processes too. But what are you imagining as alternatives for i/o and terminals?

bluetomcat 7y ago
What if we surpass file systems completely? Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts, with a clean OS interface to share that state to other processes? Terminals and process hierarchy are a complete no-go in a fresh design. Every entity in the system can be identified through unique IDs which can be handed down to other processes based on different OS policies.
- thaumaturgy 7y ago
  > Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts... I'll treat this seriously: as software development practices are just now beginning to mature, with more emphasis on test coverage being considered best practice, sure ... maybe. But there's still a lot of software out there for which "restart the application" (or even, "restart the stinking OS") is the only practical solution when it's behaving badly. > Every entity in the system can be identified through unique IDs which can be handed down to other processes based on different OS policies. Okay, but how do you write a process which is capable of communicating with any other process?
  - bluetomcat 7y ago
    > But there's still a lot of software out there for which "restart the application" (or even, "restart the stinking OS") is the only practical solution when it's behaving badly. Which, in principle, doesn't prevent a program from crashing or misbehaving when it encounters a corrupted file left over from its last run. In the imagined system, the application developer would be fully aware whether he is putting a data structure in the "volatile" or in the "non-volatile" memory areas, with some safety guarantees from the OS. Restarting would zero only the volatile area, enabling a clean starting state. > Okay, but how do you write a process which is capable of communicating with any other process? Why would you need to communicate with any process? Maybe I need to communicate with the currently running instance of "HTTP Server App" or "Database Engine App", not with an unrelated process my program knows nothing about.
    - thaumaturgy 7y ago
      > Which, in principle, doesn't prevent a program from crashing or misbehaving when it encounters a corrupted file left over from its last run. Of course. But saving state to nvram also has this problem, plus the additional problem of saving broken program state. Like I say, I'm not totally pessimistic on this anymore: there are some promising trends that make me think we might get there in the not distant future. But we're not there yet. Maybe supporting some way for a user to manually reset broken program state would be a good enough compromise -- as long as application developers didn't do something dumb, like store their license information in their program state. Autodesk immediately comes to mind there. > Why would you need to communicate with any process? Because some other process wants to communicate with you. Composability and common interfaces and loose coupling is a huge advantage in Unix-like operating systems or any other software architecture that embraces those principles. They mean that you can write software with a much longer lifespan, and usually for less effort. In your example, someone may come along wanting to write "Firewall App" long after you've abandoned your software. If your software is extensible and supports standardized IPC, "Firewall App" is possible. If it doesn't, then it gets thrown out and replaced with something else.
    - Khoth 7y ago
      > Why would you need to communicate with any process? Maybe I need to communicate with the currently running instance of "HTTP Server App" or "Database Engine App", not with an unrelated process my program knows nothing about. Today, I piped the output of some process I was running into grep. Neither of these programs knew about the existence of the other. Later, I used an image editor to create an image, then used a web browser to upload it to a website. The web browser and image editor were unaware of each other's existence.
      - bluetomcat 7y ago
        > Today, I piped the output of some process I was running into grep. Neither of these programs knew about the existence of the other. Only because "an unstructured stream of bytes" is the way for Unix processes to communicate. It allows for composability, but a very fragile and unsafe one, requiring programs to spit out and read free text, with all the crazy filtering and guesswork needed.
  - AnIdiotOnTheNet 7y ago
    Plan 9 did a pretty good job.
- stcredzero 7y ago
  Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts, with a clean OS interface to share that state to other processes? People have been thinking about this for decades. (Probably because it's a compelling idea.) Google "orthogonal persistence."
- wongarsu 7y ago
  Currently my mother has a Pictures folder which contains her pictures, where she can open them with the default photo viewer, with an editing program, and where she can trivially add files or make backups with the Windows Explorer. Any alternative interface I can imagine that offers the same flexibilty of using the same picture in multiple programs just ends up mimicking files. There's room to replace traditional folder structures with paths or anything else, and most traditional file system implementations have really slow search. But I don't see a future in replacing the notion of a file, it maps too well to the intentions of the user.
  - laughinghan 7y ago
    Well, in a traditional file system, the basic unit of data is a specific notion of a file which is an untyped blob of octets with limited metadata like size, filename, some timestamps, and some very coarse access controls. You could quibble about whether this counts as "replacing the notion of a file", but I could certainly imagine it might be useful to have a system that talks directly to disk whose basic unit of data has much more useful metadata, such as a canonicalized MIME type, much more granular access controls, much more granular access and edit history, etc. Has your mother ever downloaded a file with the wrong extension and been unable to open it? I know I have. Similarly, the abstracted "everything is a file" notion of a file without random access, which includes sockets and named pipes and stuff, is an untyped, unsized stream of octets. Message-oriented protocols like WebSockets and HTTP can and in fact are built on top of that, of course, but it could have been the reverse: instead of a stream of octets, TCP could have been a stream of arbitrarily-large but finitely sized messages. There almost certainly would have been advantages to such an approach, and applications that didn't want the message framing and just wanted a stream of octets could have easily ignored it.
- the-rc 7y ago
  That's the design of the System/38, AS/400, iSeries and System i. The unique IDs are 128bit pointers or capabilities.
- warrenm 7y ago
  >Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts, with a clean OS interface to share that state to other processes? So ... something Docker-esque?
the_af 7y ago
Wasn't this kind of filesystem an abandoned plan for one of the (now old) versions of Windows? Like Windows 2000 or XP? I don't remember why they ditched this plan. Maybe it wasn't such a good idea after all, who knows.
- thaumaturgy 7y ago
  There have been a few toy filesystems which have explored this. BeOS back in the late 90s came the closest to broad adoption, even though it didn't fully embrace the paradigm, just aspects of it. I think there are a few reasons it hasn't happened: 1. Filesystems are hard. Every new filesystem architecture ends up requiring a large pool of talented developers. 2. Nobody wants to break backwards compatibility. Current filesystem design is so integral to all kinds of software that you just can't expect all software in the world to be updated just to work with a new filesystem paradigm. 3. Databases are also hard, so a database-like filesystem is doubly so. 4. Current filesystem architecture is good enough for most stuff. The pain of continuing to use it isn't as great as the pain of changing it. None of these reasons make a database-like filesystem inherently bad. It's just not practical right now. I think we're moving in that direction though, with things like object storage.
  - the_af 7y ago
    I agree and didn't mean to imply database filesystems are necessarily bad. Just wondering. I forgot about BeOS! Do you remember which version of Windows was going to do it, or at least had preliminary plans for it? I distinctly remember reading about it. Was it an early version of Windows XP or what? I can't remember...
    - thaumaturgy 7y ago
      Are you thinking of WinFS? I think that was sometime between XP and Vista.
      - the_af 7y ago
        Yes! WinFS. Thank you!

bluetomcat 7y ago

What if we surpass file systems completely? Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts, with a clean OS interface to share that state to other processes? Terminals and process hierarchy are a complete no-go in a fresh design. Every entity in the system can be identified through unique IDs which can be handed down to other processes based on different OS policies.

thaumaturgy 7y ago
> Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts... I'll treat this seriously: as software development practices are just now beginning to mature, with more emphasis on test coverage being considered best practice, sure ... maybe. But there's still a lot of software out there for which "restart the application" (or even, "restart the stinking OS") is the only practical solution when it's behaving badly. > Every entity in the system can be identified through unique IDs which can be handed down to other processes based on different OS policies. Okay, but how do you write a process which is capable of communicating with any other process?
- bluetomcat 7y ago
  > But there's still a lot of software out there for which "restart the application" (or even, "restart the stinking OS") is the only practical solution when it's behaving badly. Which, in principle, doesn't prevent a program from crashing or misbehaving when it encounters a corrupted file left over from its last run. In the imagined system, the application developer would be fully aware whether he is putting a data structure in the "volatile" or in the "non-volatile" memory areas, with some safety guarantees from the OS. Restarting would zero only the volatile area, enabling a clean starting state. > Okay, but how do you write a process which is capable of communicating with any other process? Why would you need to communicate with any process? Maybe I need to communicate with the currently running instance of "HTTP Server App" or "Database Engine App", not with an unrelated process my program knows nothing about.
  - thaumaturgy 7y ago
    > Which, in principle, doesn't prevent a program from crashing or misbehaving when it encounters a corrupted file left over from its last run. Of course. But saving state to nvram also has this problem, plus the additional problem of saving broken program state. Like I say, I'm not totally pessimistic on this anymore: there are some promising trends that make me think we might get there in the not distant future. But we're not there yet. Maybe supporting some way for a user to manually reset broken program state would be a good enough compromise -- as long as application developers didn't do something dumb, like store their license information in their program state. Autodesk immediately comes to mind there. > Why would you need to communicate with any process? Because some other process wants to communicate with you. Composability and common interfaces and loose coupling is a huge advantage in Unix-like operating systems or any other software architecture that embraces those principles. They mean that you can write software with a much longer lifespan, and usually for less effort. In your example, someone may come along wanting to write "Firewall App" long after you've abandoned your software. If your software is extensible and supports standardized IPC, "Firewall App" is possible. If it doesn't, then it gets thrown out and replaced with something else.
  - Khoth 7y ago
    > Why would you need to communicate with any process? Maybe I need to communicate with the currently running instance of "HTTP Server App" or "Database Engine App", not with an unrelated process my program knows nothing about. Today, I piped the output of some process I was running into grep. Neither of these programs knew about the existence of the other. Later, I used an image editor to create an image, then used a web browser to upload it to a website. The web browser and image editor were unaware of each other's existence.
    - bluetomcat 7y ago
      > Today, I piped the output of some process I was running into grep. Neither of these programs knew about the existence of the other. Only because "an unstructured stream of bytes" is the way for Unix processes to communicate. It allows for composability, but a very fragile and unsafe one, requiring programs to spit out and read free text, with all the crazy filtering and guesswork needed.
- AnIdiotOnTheNet 7y ago
  Plan 9 did a pretty good job.
stcredzero 7y ago
Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts, with a clean OS interface to share that state to other processes? People have been thinking about this for decades. (Probably because it's a compelling idea.) Google "orthogonal persistence."
wongarsu 7y ago
Currently my mother has a Pictures folder which contains her pictures, where she can open them with the default photo viewer, with an editing program, and where she can trivially add files or make backups with the Windows Explorer. Any alternative interface I can imagine that offers the same flexibilty of using the same picture in multiple programs just ends up mimicking files. There's room to replace traditional folder structures with paths or anything else, and most traditional file system implementations have really slow search. But I don't see a future in replacing the notion of a file, it maps too well to the intentions of the user.
- laughinghan 7y ago
  Well, in a traditional file system, the basic unit of data is a specific notion of a file which is an untyped blob of octets with limited metadata like size, filename, some timestamps, and some very coarse access controls. You could quibble about whether this counts as "replacing the notion of a file", but I could certainly imagine it might be useful to have a system that talks directly to disk whose basic unit of data has much more useful metadata, such as a canonicalized MIME type, much more granular access controls, much more granular access and edit history, etc. Has your mother ever downloaded a file with the wrong extension and been unable to open it? I know I have. Similarly, the abstracted "everything is a file" notion of a file without random access, which includes sockets and named pipes and stuff, is an untyped, unsized stream of octets. Message-oriented protocols like WebSockets and HTTP can and in fact are built on top of that, of course, but it could have been the reverse: instead of a stream of octets, TCP could have been a stream of arbitrarily-large but finitely sized messages. There almost certainly would have been advantages to such an approach, and applications that didn't want the message framing and just wanted a stream of octets could have easily ignored it.
the-rc 7y ago
That's the design of the System/38, AS/400, iSeries and System i. The unique IDs are 128bit pointers or capabilities.
warrenm 7y ago
>Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts, with a clean OS interface to share that state to other processes? So ... something Docker-esque?

the_af 7y ago

Wasn't this kind of filesystem an abandoned plan for one of the (now old) versions of Windows? Like Windows 2000 or XP? I don't remember why they ditched this plan. Maybe it wasn't such a good idea after all, who knows.

thaumaturgy 7y ago
There have been a few toy filesystems which have explored this. BeOS back in the late 90s came the closest to broad adoption, even though it didn't fully embrace the paradigm, just aspects of it. I think there are a few reasons it hasn't happened: 1. Filesystems are hard. Every new filesystem architecture ends up requiring a large pool of talented developers. 2. Nobody wants to break backwards compatibility. Current filesystem design is so integral to all kinds of software that you just can't expect all software in the world to be updated just to work with a new filesystem paradigm. 3. Databases are also hard, so a database-like filesystem is doubly so. 4. Current filesystem architecture is good enough for most stuff. The pain of continuing to use it isn't as great as the pain of changing it. None of these reasons make a database-like filesystem inherently bad. It's just not practical right now. I think we're moving in that direction though, with things like object storage.
- the_af 7y ago
  I agree and didn't mean to imply database filesystems are necessarily bad. Just wondering. I forgot about BeOS! Do you remember which version of Windows was going to do it, or at least had preliminary plans for it? I distinctly remember reading about it. Was it an early version of Windows XP or what? I can't remember...
  - thaumaturgy 7y ago
    Are you thinking of WinFS? I think that was sometime between XP and Vista.
    - the_af 7y ago
      Yes! WinFS. Thank you!