8 ms·

We need entirely new OS abstractions to replace the dated notions of hierarchical file systems built around the metaphor of file cabinets, I/O as streams of byt

We need entirely new OS abstractions to replace the dated notions of hierarchical file systems built around the metaphor of file cabinets, I/O as streams of bytes, terminals, process hierarchy. Essentially, say goodbye to the Unix model after 50 years. It would open up an entirely new world of software experimentation and craftsmanship.

Ericson2314 7y ago
Amen to that. Unix was never a good design, and now is severely out of date. We can no longer afford to hack around it.
- jaas 7y ago
  "Unix was never a good design" Nothing's perfect. Unix was a great design that served its purpose well for a long time, and evolved a bit along the way. Saying it was "never good" is trivializing and, in my opinion, arrogant.
  - dllthomas 7y ago
    There were always objections. "Unix went from being the worst operating system available, to being the best operating system available, without getting appreciably better." https://news.ycombinator.com/item?id=19416485 https://news.ycombinator.com/item?id=19416485 Which isn't to say that those POV were necessarily correct, just that it isn't all hindsight.
    - the_af 7y ago
      To be fair, there are always objections to any working piece of technology. It's not helpful to claim "it was never good design". It was a working design that drove technology to the point it is now, and in that sense it was hugely successful. What kind of perfect and pure tech do some people want, anyway? Pick anything, whatever they like -- say, Plan 9 or OS/2 or whatever -- and I can bet you in a parallel universe where that tech won, someone on para-HN will claim that it sucked and it was never a good design and if only Unix had won.
      - deleted 7y ago
        [deleted]
      - dllthomas 7y ago
        I certainly don't disagree that it was successful, and (coming into technology when I did) I've always personally liked it. I just found the writings around what came before interesting. > I can bet you in a parallel universe where that tech won, someone on para-HN will claim that it sucked and it was never a good design and if only Unix had won. I'd take the same side of that bet as you :-P
- thaumaturgy 7y ago
  Yeah, Unix was a terrible design. Composable architecture combined with standardized i/o and IPC models. I can't imagine anything worse, especially compared to today's proliferation of data formats and inscrutable APIs. It's a good thing nobody uses it anymore.
  - taborj 7y ago
    My Online Sarcasm Detector(tm) is beeping wildly
  - pjmlp 7y ago
    Had Bell Labs been allowed to sell it in first place and that would have been exactly its outcome.
    - jstimpfle 7y ago
      missing the sarcasm?
      - Ericson2314 7y ago
        I think you are.
- pzs 7y ago
  Would you care to elaborate why Unix, in your opinion, was never a good design?
  - pjmlp 7y ago
    It was re-written on an unsafe language that completely disregarded what has happened outside Bell Labs since 1961, at Burroughs, IBM, MIT, Xerox and other research cathedrals. It only took off thanks to Bell Labs being forbidden to sell it, so it got offered for a symbolic price, alongside source code to major universities, which then decided to build on top, instead of paying OS street prices. Had UNIX been sold in the same vein as other mainframe OSes and no one would be talking about whatever quality it might have had.
    - systemBuilder 7y ago
      I worked on the xerox product OS (Pilot) written by my manager, and was on the team with the designers of Multics and Swift, MIT's OSes. At MIT we all ran 4.15bsd and then 4.2bsd and UNIX was so far ahead we focused on improving it, mostly. Pilot, Multics, CTSS, were nowhere near as powerful or developer friendly as UNIX. UNIX killed off OS research for a long time!
      - pjmlp 7y ago
        Given what I know from XDE and Cedar, I fail to see how a text based CLI, without the graphical debuggers, REPL and GUI workflows was more developer friendly.
    - Ericson2314 7y ago
      Thank you for being a breadth of wisdom in the UNIX-worshiping wilderness. You can always get mindshare being first massively underpriced thing to market.
      - jstimpfle 7y ago
        Only that this doesn't even touch on design, or what parts could have been done better.
- FussyZeus 7y ago
  What a ridiculous statement. Yeah if it was possible to just jump to the best solution each time then we'd benefit massively, but that's not how any science works. The new awesome thing is always built on the back of what was done before. Electric cars have been around since the inception of the internal combustion engine but we didn't have electrical supplies with the energy density of gasoline to make electric cars possible until decades later, and modern life would've been (and remains, unfortunately) impossible without the energy density of fossil fuels. Hopefully someday we'll get past that, but when that happens, that doesn't then mean that fossil fuels were just a bad idea, they were the only stepping stone to what replaced them.
  - Ericson2314 7y ago
    The other OSes from Unix's era were already as good or better (or at very least there was a diversity of ideas). Then the regulation that prevent AT&T from being sold like the others causes Unix to win prematurely. Maybe that's how Ford beat the earlier luxury cars in raw profit, but that's not how the ICE beat electric cars.
- dekhn 7y ago
  UNIX was a great design for a wide range of people. FOr example, multiple domains of science were transformed because UNIX made it easy to implement microcomputers for data collection on realtime data and share the resulting software among a pool of other researchers.
- wongarsu 7y ago
  Which is why we have Windows with the NT family of kernels. It's a reimagining that contains great ideas like a flexible file system stack that allows you to dynamically add drivers that add encryption, compression, antivirus checks etc, and a few bad ideas like a flexible file system stack that's orders of magnitude slower when working with small files than unix implementations. Windows is a mainstream OS that is very different from Unix in many regards (even if it still has files). And it really shows that the grass isn't greener on the other side: some parts are great, some parts are terrible, but there aren't a lot of things that offer a universally better tradeoff.
  - Ericson2314 7y ago
    The VMS people Microsoft poached and then straightjacketed with legacy concerns did have good ideas, yes.
- ncmncm 7y ago
  There are only two kinds of OS: the ones people complain about, and the ones no one uses.
thaumaturgy 7y ago
I would like to see an updated file system architecture that's closer to a database, with tagging and all that. And I could see that extending to processes too. But what are you imagining as alternatives for i/o and terminals?
- bluetomcat 7y ago
  What if we surpass file systems completely? Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts, with a clean OS interface to share that state to other processes? Terminals and process hierarchy are a complete no-go in a fresh design. Every entity in the system can be identified through unique IDs which can be handed down to other processes based on different OS policies.
  - thaumaturgy 7y ago
    > Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts... I'll treat this seriously: as software development practices are just now beginning to mature, with more emphasis on test coverage being considered best practice, sure ... maybe. But there's still a lot of software out there for which "restart the application" (or even, "restart the stinking OS") is the only practical solution when it's behaving badly. > Every entity in the system can be identified through unique IDs which can be handed down to other processes based on different OS policies. Okay, but how do you write a process which is capable of communicating with any other process?
    - bluetomcat 7y ago
      > But there's still a lot of software out there for which "restart the application" (or even, "restart the stinking OS") is the only practical solution when it's behaving badly. Which, in principle, doesn't prevent a program from crashing or misbehaving when it encounters a corrupted file left over from its last run. In the imagined system, the application developer would be fully aware whether he is putting a data structure in the "volatile" or in the "non-volatile" memory areas, with some safety guarantees from the OS. Restarting would zero only the volatile area, enabling a clean starting state. > Okay, but how do you write a process which is capable of communicating with any other process? Why would you need to communicate with any process? Maybe I need to communicate with the currently running instance of "HTTP Server App" or "Database Engine App", not with an unrelated process my program knows nothing about.
sytelus 7y ago
I am doubtful if hierarchical file systems have anything to do with slow disks. It a design pattern that you encounter all over in real life for human mind to manage large quantity of information. Using tags is another pattern but with its own pro and cons. Same goes for I/O steams, process hierarchy etc. There may be better designs patterns out there but I don’t see why these existing design patterns would become obsolete even if disks become as fast as RAM.
taborj 7y ago
Every system is driven by user adoption. At this point, it will be nearly impossible to dethrone the current methodologies. Not saying it shouldn't be done, just that it might fail.
- zwkrt 7y ago
  The classic way that this has always happened in the past is by solving a specific problem in a hacky-but-cheap way as compared to existing tech. The issue I see here is I can already run Unix on a smartwatch or a car radio, so it seems difficult to understand where new computing paradigms will come from. Google's Fuchsia is a contender, but I wonder how realistic it is that it would ever become 'the standard'. Maybe the future is a world of truly heterogenous operating system design, but more likely we will employ a one-size-fits all approach just like every other industry. All car engines are about the same, all shoes are constructed with similar parts, all stoves are basically interchaneable, it is unrealistic to expect that software will be any different.
- Razengan 7y ago
  > Every system is driven by user adoption. At this point, it will be nearly impossible to dethrone the current methodologies. iOS, when the iPhone first came out, turned many of the entrenched perceptions about computing devices around on their head, and people embraced it. Even without the celebrity power of someone like Apple, an experimental system could still thrive today in the shadows with a small cult of followers nurturing and developing it, until it breaks out.
strictfp 7y ago
One idea I've been throwing around is to replace files with HTTP resources; everything is a resource. Effectively Plan9s idea, but the time might be right.
- BGyss 7y ago
  Believe it or not there was a system at Sony Pictures Imageworks that worked like this - no idea what it was doing under the hood though. Worked great in practice though - every possible production resource was a URL that started with spref:// iirc.
- worble 7y ago
  I don't know much about the OS and designs, but is that similar to how Redox is approaching it with "everything is a url"[0]? [0]https://doc.redox-os.org/book/design/urls_schemes_resources.html https://doc.redox-os.org/book/design/urls_schemes_resources....
  - PeterCorless 7y ago
    If you go that route, bind unique objects to Uniform Resource Names (URN), and then have a mapping to instantiations (URLs), so that the same resource can be found in its multiple locations. Would help immeasurably for a distribution model.
Spivak 7y ago
What would you replace it with? At the very core you need two things: the notion of a unit of data and some metadata to address it. To be as flexible as possible that unit of data would probably modeled as a sequence of individually addressable bytes but it could be more structured. Such a thing is dangerous because if your structure isn't sufficiently expressive 20 years down the road people will end up imposing their own structure on top of obj.binary and there goes all the work. Once you have data and handles to access it you might start wanting some convenience features like access control, locking, namespacing, constraints, relationships. I don't disagree that we can drop many of the current filesystem semantics but fundamentally all that really means is changing is the query language to access objects and manipulate their metadata. I also don't disagree about process hierarchy. Being able to express relationships between processes beyond parent-child natively without farming out to an external scheduler would be awesome.
- scroot 7y ago
  Object systems
  - silversconfused 7y ago
    Ok, but what does the level below that look like?
    - scroot 7y ago
      I'm not sure there has to be a lower level at all, aside from asm and machine language
      - Spivak 7y ago
        Okay so you have a device driver that exposes your underlying hard disk as a single contiguous segment of blocks of a given size. You have objects which are blobs of heterogeneously sized arbitrary data with an addressing scheme. The software that marries these two things is basically a filesystem driver (in that you can implement filesystem semantics on top of it -- hell Ceph does it right now). Nothing about a modern Linux/BSD system really stands in the way of doing this.
jnurmine 7y ago
I think I don't understand. Is there something preventing people from experimenting and craftsmanship at this very moment?
AnimalMuppet 7y ago
Then... what? Do you have a positive to recommend, or just "not what we've been doing"? (It's not necessarily bad if you don't - when you're doing the wrong thing, the first step to improvement is to stop doing it.) But it seems to me that much of this can be done within the existing structure. You don't want I/O as streams of bytes? Great. Whatever new thing you think it should be, you can build that on streams of bytes. Knock yourself out. (It may not be as fast as it would be if it were directly supported by the OS, but you can prove the value of the concept by building on top of the OS.) Same thing with the metaphor of file cabinets (I presume you mean the hierarchical file system.) Well, does your OS let you read and write raw disk sectors? No? Fine. Create one giant file that takes up the whole disk, and manage it yourself. Try out whatever different way of managing that space that floats your boat. Again, it will be slower, but again, you can experiment and prove out your concepts right now. You don't need to wait.
- garmaine 7y ago
  E.g. use the relational model for data storage instead of files (SQLite is a purposeful step in this direction). Use a single system image with no distinction between network, processor, or core communication unless needed.
olliej 7y ago
...except humans instinctively organize things by hierarchies (even something as counting is done hierarchically). Just saying “we need to change this” without saying what the short comings you want to address isn’t a super useful statement. Multiple platforms have attempted to have the user interface to their data be tag based, but that simply doesn’t scale for the amount of data people can manage with hierarchies. Finally what the heck are you talking about in that last sentence: how does changing representation result in a new age of experimentation and (???) craftsmanship? Why is craftsmanship gated on the user level abstraction to bytes? Experimentation already happens today, what does this change?
robbyt 7y ago
Correct me if I'm wrong, but isn't this what Plan 9 does?
- AsyncAwait 7y ago
  I think Plan9 still has the notion of files, folders and such?
dorlaor 7y ago
At Scylla we initially defined a new filesystem which adheres shard-per-core and thus a single physical hyperthread has the sole access to the data and thus there is no contention. While it will be better than current XFS, we've made aio improvement to the later over the years and today it's good enough for ScyllaDB. Practically, even though Scylla has its tcp/ip stack in userspace on top of DPDK, we learned over the years that it's ok to use the less efficient kernel tcp stack. Most of the overhead and the optimizations can still happen within the DB itself as long as it controls the memory, the cache and manages the networking queues
notduncansmith 7y ago
Why are these insufficient? What abstractions should we building on?
fragmede 7y ago
PalmOS had this! Applications connected to databases and read and wrote records. Sqlite acts similar for the more modern era, with a traditional hierarchical filesystem underneath the sqlite DB. Looking at modern app-based "file" access using Google docs and its ilk are that reimagining. The UI is a list of recent files, a small number of features, and then a search box. There's not File -> Save, nor am I forced to pick using a folder metaphor, where I want to put it. That there's (likely) an underlying hierarchical filesystem somewhere below, in the stack seems like an implementation detail. As a programmer there's a library/middleware to be used to access resources, but once inside, object based access already exists. Looking at video game save files, that's been the case for a while, with the state of objects (in fact, visible objects that the user interacts with) being saved and restored from disk. I agree it's not as satisfying as a total paradigm shift in computing on every single level, but the notion that file system, byte stream access is a holdover from a previous era ignores practical, user facing progress we've made since.
MathematicalArt 7y ago
See: “Designing better file organization around tags, not hierarchies”[1] The author has done a thorough preliminary exploration on this matter. [1] https://www.nayuki.io/page/designing-better-file-organization-around-tags-not-hierarchies https://www.nayuki.io/page/designing-better-file-organizatio...

Ericson2314 7y ago

Amen to that. Unix was never a good design, and now is severely out of date. We can no longer afford to hack around it.

jaas 7y ago
"Unix was never a good design" Nothing's perfect. Unix was a great design that served its purpose well for a long time, and evolved a bit along the way. Saying it was "never good" is trivializing and, in my opinion, arrogant.
- dllthomas 7y ago
  There were always objections. "Unix went from being the worst operating system available, to being the best operating system available, without getting appreciably better." https://news.ycombinator.com/item?id=19416485 https://news.ycombinator.com/item?id=19416485 Which isn't to say that those POV were necessarily correct, just that it isn't all hindsight.
  - the_af 7y ago
    To be fair, there are always objections to any working piece of technology. It's not helpful to claim "it was never good design". It was a working design that drove technology to the point it is now, and in that sense it was hugely successful. What kind of perfect and pure tech do some people want, anyway? Pick anything, whatever they like -- say, Plan 9 or OS/2 or whatever -- and I can bet you in a parallel universe where that tech won, someone on para-HN will claim that it sucked and it was never a good design and if only Unix had won.
    - deleted 7y ago
      [deleted]
    - dllthomas 7y ago
      I certainly don't disagree that it was successful, and (coming into technology when I did) I've always personally liked it. I just found the writings around what came before interesting. > I can bet you in a parallel universe where that tech won, someone on para-HN will claim that it sucked and it was never a good design and if only Unix had won. I'd take the same side of that bet as you :-P
thaumaturgy 7y ago
Yeah, Unix was a terrible design. Composable architecture combined with standardized i/o and IPC models. I can't imagine anything worse, especially compared to today's proliferation of data formats and inscrutable APIs. It's a good thing nobody uses it anymore.
- taborj 7y ago
  My Online Sarcasm Detector(tm) is beeping wildly
- pjmlp 7y ago
  Had Bell Labs been allowed to sell it in first place and that would have been exactly its outcome.
  - jstimpfle 7y ago
    missing the sarcasm?
    - Ericson2314 7y ago
      I think you are.
pzs 7y ago
Would you care to elaborate why Unix, in your opinion, was never a good design?
- pjmlp 7y ago
  It was re-written on an unsafe language that completely disregarded what has happened outside Bell Labs since 1961, at Burroughs, IBM, MIT, Xerox and other research cathedrals. It only took off thanks to Bell Labs being forbidden to sell it, so it got offered for a symbolic price, alongside source code to major universities, which then decided to build on top, instead of paying OS street prices. Had UNIX been sold in the same vein as other mainframe OSes and no one would be talking about whatever quality it might have had.
  - systemBuilder 7y ago
    I worked on the xerox product OS (Pilot) written by my manager, and was on the team with the designers of Multics and Swift, MIT's OSes. At MIT we all ran 4.15bsd and then 4.2bsd and UNIX was so far ahead we focused on improving it, mostly. Pilot, Multics, CTSS, were nowhere near as powerful or developer friendly as UNIX. UNIX killed off OS research for a long time!
    - pjmlp 7y ago
      Given what I know from XDE and Cedar, I fail to see how a text based CLI, without the graphical debuggers, REPL and GUI workflows was more developer friendly.
  - Ericson2314 7y ago
    Thank you for being a breadth of wisdom in the UNIX-worshiping wilderness. You can always get mindshare being first massively underpriced thing to market.
    - jstimpfle 7y ago
      Only that this doesn't even touch on design, or what parts could have been done better.
FussyZeus 7y ago
What a ridiculous statement. Yeah if it was possible to just jump to the best solution each time then we'd benefit massively, but that's not how any science works. The new awesome thing is always built on the back of what was done before. Electric cars have been around since the inception of the internal combustion engine but we didn't have electrical supplies with the energy density of gasoline to make electric cars possible until decades later, and modern life would've been (and remains, unfortunately) impossible without the energy density of fossil fuels. Hopefully someday we'll get past that, but when that happens, that doesn't then mean that fossil fuels were just a bad idea, they were the only stepping stone to what replaced them.
- Ericson2314 7y ago
  The other OSes from Unix's era were already as good or better (or at very least there was a diversity of ideas). Then the regulation that prevent AT&T from being sold like the others causes Unix to win prematurely. Maybe that's how Ford beat the earlier luxury cars in raw profit, but that's not how the ICE beat electric cars.
dekhn 7y ago
UNIX was a great design for a wide range of people. FOr example, multiple domains of science were transformed because UNIX made it easy to implement microcomputers for data collection on realtime data and share the resulting software among a pool of other researchers.
wongarsu 7y ago
Which is why we have Windows with the NT family of kernels. It's a reimagining that contains great ideas like a flexible file system stack that allows you to dynamically add drivers that add encryption, compression, antivirus checks etc, and a few bad ideas like a flexible file system stack that's orders of magnitude slower when working with small files than unix implementations. Windows is a mainstream OS that is very different from Unix in many regards (even if it still has files). And it really shows that the grass isn't greener on the other side: some parts are great, some parts are terrible, but there aren't a lot of things that offer a universally better tradeoff.
- Ericson2314 7y ago
  The VMS people Microsoft poached and then straightjacketed with legacy concerns did have good ideas, yes.
ncmncm 7y ago
There are only two kinds of OS: the ones people complain about, and the ones no one uses.

thaumaturgy 7y ago

I would like to see an updated file system architecture that's closer to a database, with tagging and all that. And I could see that extending to processes too. But what are you imagining as alternatives for i/o and terminals?

bluetomcat 7y ago
What if we surpass file systems completely? Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts, with a clean OS interface to share that state to other processes? Terminals and process hierarchy are a complete no-go in a fresh design. Every entity in the system can be identified through unique IDs which can be handed down to other processes based on different OS policies.
- thaumaturgy 7y ago
  > Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts... I'll treat this seriously: as software development practices are just now beginning to mature, with more emphasis on test coverage being considered best practice, sure ... maybe. But there's still a lot of software out there for which "restart the application" (or even, "restart the stinking OS") is the only practical solution when it's behaving badly. > Every entity in the system can be identified through unique IDs which can be handed down to other processes based on different OS policies. Okay, but how do you write a process which is capable of communicating with any other process?
  - bluetomcat 7y ago
    > But there's still a lot of software out there for which "restart the application" (or even, "restart the stinking OS") is the only practical solution when it's behaving badly. Which, in principle, doesn't prevent a program from crashing or misbehaving when it encounters a corrupted file left over from its last run. In the imagined system, the application developer would be fully aware whether he is putting a data structure in the "volatile" or in the "non-volatile" memory areas, with some safety guarantees from the OS. Restarting would zero only the volatile area, enabling a clean starting state. > Okay, but how do you write a process which is capable of communicating with any other process? Why would you need to communicate with any process? Maybe I need to communicate with the currently running instance of "HTTP Server App" or "Database Engine App", not with an unrelated process my program knows nothing about.

sytelus 7y ago

I am doubtful if hierarchical file systems have anything to do with slow disks. It a design pattern that you encounter all over in real life for human mind to manage large quantity of information. Using tags is another pattern but with its own pro and cons. Same goes for I/O steams, process hierarchy etc. There may be better designs patterns out there but I don’t see why these existing design patterns would become obsolete even if disks become as fast as RAM.

taborj 7y ago

Every system is driven by user adoption. At this point, it will be nearly impossible to dethrone the current methodologies. Not saying it shouldn't be done, just that it might fail.

zwkrt 7y ago
The classic way that this has always happened in the past is by solving a specific problem in a hacky-but-cheap way as compared to existing tech. The issue I see here is I can already run Unix on a smartwatch or a car radio, so it seems difficult to understand where new computing paradigms will come from. Google's Fuchsia is a contender, but I wonder how realistic it is that it would ever become 'the standard'. Maybe the future is a world of truly heterogenous operating system design, but more likely we will employ a one-size-fits all approach just like every other industry. All car engines are about the same, all shoes are constructed with similar parts, all stoves are basically interchaneable, it is unrealistic to expect that software will be any different.
Razengan 7y ago
> Every system is driven by user adoption. At this point, it will be nearly impossible to dethrone the current methodologies. iOS, when the iPhone first came out, turned many of the entrenched perceptions about computing devices around on their head, and people embraced it. Even without the celebrity power of someone like Apple, an experimental system could still thrive today in the shadows with a small cult of followers nurturing and developing it, until it breaks out.

strictfp 7y ago

One idea I've been throwing around is to replace files with HTTP resources; everything is a resource. Effectively Plan9s idea, but the time might be right.

BGyss 7y ago
Believe it or not there was a system at Sony Pictures Imageworks that worked like this - no idea what it was doing under the hood though. Worked great in practice though - every possible production resource was a URL that started with spref:// iirc.
worble 7y ago
I don't know much about the OS and designs, but is that similar to how Redox is approaching it with "everything is a url"[0]? [0]https://doc.redox-os.org/book/design/urls_schemes_resources.html https://doc.redox-os.org/book/design/urls_schemes_resources....
- PeterCorless 7y ago
  If you go that route, bind unique objects to Uniform Resource Names (URN), and then have a mapping to instantiations (URLs), so that the same resource can be found in its multiple locations. Would help immeasurably for a distribution model.

Spivak 7y ago

What would you replace it with? At the very core you need two things: the notion of a unit of data and some metadata to address it. To be as flexible as possible that unit of data would probably modeled as a sequence of individually addressable bytes but it could be more structured. Such a thing is dangerous because if your structure isn't sufficiently expressive 20 years down the road people will end up imposing their own structure on top of obj.binary and there goes all the work. Once you have data and handles to access it you might start wanting some convenience features like access control, locking, namespacing, constraints, relationships. I don't disagree that we can drop many of the current filesystem semantics but fundamentally all that really means is changing is the query language to access objects and manipulate their metadata. I also don't disagree about process hierarchy. Being able to express relationships between processes beyond parent-child natively without farming out to an external scheduler would be awesome.

scroot 7y ago
Object systems
- silversconfused 7y ago
  Ok, but what does the level below that look like?
  - scroot 7y ago
    I'm not sure there has to be a lower level at all, aside from asm and machine language
    - Spivak 7y ago
      Okay so you have a device driver that exposes your underlying hard disk as a single contiguous segment of blocks of a given size. You have objects which are blobs of heterogeneously sized arbitrary data with an addressing scheme. The software that marries these two things is basically a filesystem driver (in that you can implement filesystem semantics on top of it -- hell Ceph does it right now). Nothing about a modern Linux/BSD system really stands in the way of doing this.

jnurmine 7y ago

I think I don't understand. Is there something preventing people from experimenting and craftsmanship at this very moment?

AnimalMuppet 7y ago

Then... what? Do you have a positive to recommend, or just "not what we've been doing"? (It's not necessarily bad if you don't - when you're doing the wrong thing, the first step to improvement is to stop doing it.) But it seems to me that much of this can be done within the existing structure. You don't want I/O as streams of bytes? Great. Whatever new thing you think it should be, you can build that on streams of bytes. Knock yourself out. (It may not be as fast as it would be if it were directly supported by the OS, but you can prove the value of the concept by building on top of the OS.) Same thing with the metaphor of file cabinets (I presume you mean the hierarchical file system.) Well, does your OS let you read and write raw disk sectors? No? Fine. Create one giant file that takes up the whole disk, and manage it yourself. Try out whatever different way of managing that space that floats your boat. Again, it will be slower, but again, you can experiment and prove out your concepts right now. You don't need to wait.

garmaine 7y ago
E.g. use the relational model for data storage instead of files (SQLite is a purposeful step in this direction). Use a single system image with no distinction between network, processor, or core communication unless needed.

olliej 7y ago

...except humans instinctively organize things by hierarchies (even something as counting is done hierarchically). Just saying “we need to change this” without saying what the short comings you want to address isn’t a super useful statement. Multiple platforms have attempted to have the user interface to their data be tag based, but that simply doesn’t scale for the amount of data people can manage with hierarchies. Finally what the heck are you talking about in that last sentence: how does changing representation result in a new age of experimentation and (???) craftsmanship? Why is craftsmanship gated on the user level abstraction to bytes? Experimentation already happens today, what does this change?

robbyt 7y ago

Correct me if I'm wrong, but isn't this what Plan 9 does?

AsyncAwait 7y ago
I think Plan9 still has the notion of files, folders and such?

dorlaor 7y ago

At Scylla we initially defined a new filesystem which adheres shard-per-core and thus a single physical hyperthread has the sole access to the data and thus there is no contention. While it will be better than current XFS, we've made aio improvement to the later over the years and today it's good enough for ScyllaDB. Practically, even though Scylla has its tcp/ip stack in userspace on top of DPDK, we learned over the years that it's ok to use the less efficient kernel tcp stack. Most of the overhead and the optimizations can still happen within the DB itself as long as it controls the memory, the cache and manages the networking queues

notduncansmith 7y ago

Why are these insufficient? What abstractions should we building on?

fragmede 7y ago

PalmOS had this! Applications connected to databases and read and wrote records. Sqlite acts similar for the more modern era, with a traditional hierarchical filesystem underneath the sqlite DB. Looking at modern app-based "file" access using Google docs and its ilk are that reimagining. The UI is a list of recent files, a small number of features, and then a search box. There's not File -> Save, nor am I forced to pick using a folder metaphor, where I want to put it. That there's (likely) an underlying hierarchical filesystem somewhere below, in the stack seems like an implementation detail. As a programmer there's a library/middleware to be used to access resources, but once inside, object based access already exists. Looking at video game save files, that's been the case for a while, with the state of objects (in fact, visible objects that the user interacts with) being saved and restored from disk. I agree it's not as satisfying as a total paradigm shift in computing on every single level, but the notion that file system, byte stream access is a holdover from a previous era ignores practical, user facing progress we've made since.

MathematicalArt 7y ago

See: “Designing better file organization around tags, not hierarchies”[1] The author has done a thorough preliminary exploration on this matter. [1] https://www.nayuki.io/page/designing-better-file-organization-around-tags-not-hierarchies https://www.nayuki.io/page/designing-better-file-organizatio...