14 ms·
Non-Posix File Systems
- phendrenad2 6y agoInteresting POSIX history. It used to come down to Windows/Mac and POSIX, but for the past 20 or so years we've been seeing new filesystem styles emerge, which blend cloud and on-device storage. Navigating the iOS file save menu is a good example. 3rd party apps like Google Drive and DropBox hook into the OS and show up in the list. I think this is probably the way things are headed, and POSIX and the "Windows/DOS/Classic Mac" style will both fade away.
- deleted 6y ago[deleted]
- boogies 6y agoMenus ≠ filesystems. POSIX will last forever because it’s simple, standard, solid, portable and fairly flexible (see FUSE). DOS will last forever because inertia is the only reason Windows still exists, and MS knows this and prioritizes backwards compatibility, eg. it’s still basically impossible to name a file CON on Windows 10 because of backwards compatibility.
- kccqzy 6y agoThat's a shortsighted and very programmer-centric view of what a filesystem is. Increasingly we are seeing a bifurcation between the kind of filesystems developers use for programming (POSIX style) and the kind of "filesystems" being exposed to end users. I put the word in quotes because they are not thought of as filesystems by its traditional definition. But it's a reality that many users now think of these siloed-by-app systems as the real filesystem. The iOS menu mentioned by GP is a great example because that's what's user facing, not the developer-facing one like /private/var/mobile/Containers/blah-blah-blah. Today even many developers won't have to deal with POSIX-style filesystems that often, since they generally write code to store structured data in a database (sqlite on mobile, more sophisticated ones on the server side) while large blobs are stored in services like S3, making the idea of a POSIX filesystem quaint. My own prediction is that POSIX filesystems in another decade or so will be like assembly language today: it's still there, still being taught and learned, but users and a majority of developers won't need to know about it.
- boogies 6y agoI think it’s very typical of very casual and exploitable users to have product names deeply connected with a certain activity — eg. using Google or Zoom as verbs — and “cloud” storage companies would like them to think of their brand name as the main storage area on their devices (and certainly not as just rented space on normal drives in a big room full of someone else’s computers), and I think they are and will be successful, and these users won’t think of them with generic terms like “filesystem”.
- _jal 6y agoCasual users' confusion about the presentation of technical details is a different conversation. This one is about actual filesystems. It may well be that end users are abstracted away from their filesystems, but you seem to assume both that 'The Cloud' is the natural end-state of systems evolution and that everyone comes along for the ride. I don't think either of those are true.
- anthk 6y agoPalm tried what you said. Now Palm is dead.
- jborean93 6y agoecho "cough" > \\?\C:\temp\COM edit: meant CON not COM
- ardy42 6y ago>> eg. it’s still basically impossible to name a file CON on Windows 10 because of backwards compatibility. > echo "cough" > \\?\C:\temp\COM I think he's referring to CON not COM: https://en.wikipedia.org/wiki/DOS#Reserved_device_names https://en.wikipedia.org/wiki/DOS#Reserved_device_names > There are reserved device names in DOS that cannot be used as filenames regardless of extension as they are occupied by built-in character devices.... > The reserved names are: > CON, for console
- boogies 6y agoYes, I mistakenly had COM in the original comment, before editing it a while ago but maybe before jborean93’s last refresh before commenting.
- jborean93 6y agoThat's my bad (I thought it looked weird when I typed it). Still the same syntax works with CON and any other reserved DOS devices echo "hey DOS" > \\?\C:\temp\CON del \\?\C:\temp\CON While it can be "trivially" done, it's a hit or miss if any application actually supports it. You effectively need to deal with the NT path prefix \\?\ when dealing with the Win32 APIs to be able to open a handle to the file. The prefix essentially tells Win32 to back off and get the handle directly from the NT object manager.
- icedchai 6y agoDOS had several reserved file names. Back in the BBS days, I remember kids uploading files named "CLOCK$" to lock up BBSes.
- pjmlp 6y agoWindows still exists because no FOSS UNIX clone can get their act together in what means to provide a proper desktop experience both to users, as well as developers. KDE and GNOME are quite close to it, but keep being ostracized, so one is left with Android and ChromeOS where the Linux kernel is an implementation detail.
- guerby 6y agoFUSE allows you to experiment new (or old) features easily nowadays, too bad it isn't mentionned in the article: https://en.wikipedia.org/wiki/Filesystem_in_Userspace https://en.wikipedia.org/wiki/Filesystem_in_Userspace I use borg mount to access my backups for example.
- bityard 6y agoFUSE effectively provides a POSIX-like interface to arbitrary code, whereas the author is lamenting that these features weren't built into Unix/POSIX-like systems to make them widely available in the first place. The "problem" with modern POSIXish systems is that the definition of what is "POSIX" seems to be set in stone. All the various Unix-likes offer POSIX compatibility (or aim toward it) while new features that extend capability end up being implemented in completely different ways across different systems. So for example while Linux has inotify (arguably an implementation of the filesystem traps that Multics had), SGI had FAM, FreeBSD has kqueue, MacOS has FSEvents, and they're all incompatible with each other... a nightmare for developers of portable applications.
- beervirus 6y agoHow did this get to be the top link on HN in under 20 minutes with 0 comments?
- kbenson 6y agoComments usually trail upvotes, and placement is an algorithm of upvotes to age (plus the occasional editorial tweak from admins). My guess is that 5 upvotes in 5 minutes is enough to get a front page placement during certain times, and from there as long as the title seems interesting regular effects of popularity take over.
- solarkraft 6y agoComments are not a ranking factor. Per the FAQ: > The basic algorithm divides points by a power of the time since a story was submitted. ... So what matters is that this story was upvoted quickly.
- saagarjha 6y agoComments are actually usually a downranking factor, to try to prevent flamewar posts from staying on the front page.
- _delirium 6y agoThis is incidentally why the Common Lisp file I/O functions are a bit more complicated than you might expect. You can ignore most of it if you assume something vaguely POSIX-like, but if you want to be portable to e.g. both Unix and VMS (as was once desirable), there are functions like make-pathname that build a representation of a file location from up to 6 parameters (host, device, directory, name, type, and version). The 3 examples in the docs are interesting: http://clhs.lisp.se/Body/f_mk_pn.htm http://clhs.lisp.se/Body/f_mk_pn.htm
- ulzeraj 6y agoAnyone remembers Novel NSS? Even when it got ported to Linux as part of Open Enterprise Server it lacked a lot of POSIX features. The ACLs were the weirdest part since they had to be configured in eDirectory’s LDAP console.
- sillysaurusx 6y agoSurprised no mention of plan9's FS. It always seemed like the core innovation in plan9. Basically, everything is a file. But to a ridiculous extent, far beyond what you'd normally think of as files. It's been years since I looked into it though, so maybe I'm misremembering. Designing it that way has lots of advantages. For example, you can connect computers together via networks using the equivalent of `cat`. (And yes, we have netcat, but it's not quite the same thing as having the abstraction built into the OS.)
- amelius 6y agoSounds cool, but at that point I guess you can stop calling it a file, and call it "object" instead.
- anthk 6y agoIf you can use read and write, it's a file.
- floren 6y agoIf you implement, for example, the network stack as a filesystem, and you have generic facilities which let you bind "files" from one system to another, you can forward your traffic through another host as easily as: import remote-host /net Now your new network connections will go through remote-host, because a network connection is just reads and writes to files in /net. Lots of neat applications when you take a few simple ideas and expand them as far as you can: represent everything as files in a namespace, provide advanced tools to manipulate that namespace, and do it all via a networked protocol.
- mnem 6y agoFascinating to see other possibilities that were out there. Were the capabilities in Hydra the inspiration for what Fuchsia OS has, or are they a long standing concept?
- kentonv 6y agoHydra and Fuchsia are both operating systems that use Capability-based security. Hydra would have been one of the very earliest ones, while Fuchsia is a very new one. There have been many others in between, like KeyKOS or EROS, as well as use of capabilities in "normal" operating systems, like FreeBSD's Capsicum, or arguably even just passing file descriptors over Unix sockets with SCM_RIGHTS on any POSIX system. There have also been capability-based programming languages like E and Pony, and capability-based network protocols like CapTP or (my own) Cap'n Proto. So it's unlikely that the designers of Fuchsia directly based it on Hydra, but there was probably indirect inspiration, yes.
- gumby 6y agoCapability systems have been an area of research since the 60s
- tyingq 6y agoWas surprised not to see the VAX/VMS filesystem on the list. It had both stream and record based files, versioning, and a fairly rich ACL setup.
- protomyth 6y agoIt also allowed for amazing clustering. I would guess there are quite a few filesystems lost in history. My personal favorite will be the Newton's Soups. A modern version with replication would be amazing.
- Gibbon1 6y agoA VMS programmer in the late 80's told me the current direction the industry was moving towards made him sick to his stomach. Microsoft's FAT was one thing. But POSIX was the real trash.
- JdeBP 6y agoThe headlined article is primarily about things that have been forgotten. ODS hasn't been forgotten. Much of it exists pretty much as-is in NTFS, with just some names changed.
- bregma 6y agoVMS had four different file types out of the box: serial with carriage-return carriage control, serial with FORTRAN carriage control, random-access, and ISAM (indexed-sequential access model). Three of those worked as well from a bank of open-reel TM03 devices or TU-58 carts as they did from a cluster of DECNET drives. You could, of course, extend those models by plugging in to the rab$, fab$, and xab$ layers or write your own from scratch using the sys$ layer. Unix, in contrast, had only the random-access file model. You could use just that file model to build any of the other types in userspace, but to talk to a tape drive you had to use the tape archive (tar) utility. It's just the usual tradeoff about where you draw the line between a solution and a tool.
- skissane 6y agoArticle describes Multics’ ability to have a file's directory entry on disk but its contents on tape, so trying to access its contents will cause it to be retrieved from tape, and then asks "Is something like this offered on any POSIX-compatible file system?" What the article is describing is basically just HSM (Hierarchical Storage Management), which is a commercially available technology – e.g. Sun/Oracle SAM-QFS on Solaris, IBM Tivoli Storage Manager on AIX, DFSMShsm on z/OS. Windows NTFS also supports HSM, although the core NTFS itself only provides features necessary to implement HSM (such as FILE_ATTRIBUTE_OFFLINE and reparse points), and you need an add-on to Windows to actually use those features to produce a full HSM solution. (Actually Windows itself used to include such a solution, Remote Storage Service, but Microsoft removed it in Windows Server 2008 onwards; but the underlying functionality is still there in NTFS, and available for third party HSM implementations to exploit.)
- a1369209993 6y agoYou could also implement this pretty easily in FUSE: if the dirent is present on the underlying (ext or whatever) filesystem, just forward the operations, otherwise leave the syscall blocked and hunt down the relevant backup. I don't know that anyone's actually written that, though.
- acdha 6y agoNothing about this is “easily” once you work through the edge cases for performance and reliability. The only places which need HSM have enough data volume and range of applications to stress any simple solution (e.g. with the approach you outlined: what happens when someone runs find on that volume?). One of the more interesting challenges is how to deal with not having quite enough fast storage to batch the slow storage. Your system can appear to work well with one test workload and then fail miserably when two people start running different tasks at the same time. After a couple decades of this, I generally think this class of software is a mistake. Any time you misrepresent one class of storage as another it inevitably leads to very complex software which is still pretty fragile and confuses its users on a regular basis, and the cost savings never deliver to the hoped-for degree.
- mjevans 6y agoIt would be interesting if a "file type" existed to provide these more complex operations, and if those could also be bound to a daemon running with provided parameters. This would allow for a filesystem name linked to a database API / any other daemon or program. I also love the TRAP (related to above) and both directory operations. Quotas can sort of fulfill the space limits thing (but I've never bothered using such a system and haven't been on a system that did). Append Only directories are a far harder beast to slay, but would very nicely fulfill many queue submission systems.
- macintux 6y agoI was fortunate enough to be working for Basho and attended the RICON where mrb gave his distributed systems archaeology talk. It really highlighted for me how so many important ideas are captured in historical whitepapers and often forgotten today. https://speakerdeck.com/mrb/distributed-systems-archaeology https://speakerdeck.com/mrb/distributed-systems-archaeology Regrettably many RICON talks were lost when Basho went out of business, but someone rescued this one. Thanks Alex Ott! https://youtu.be/om_mAaM5sL8 https://youtu.be/om_mAaM5sL8
- garmaine 6y agoStrange that BeFS/BeOS is not on this list.
- nayuki 6y agoA good read: https://arstechnica.com/information-technology/2018/07/the-beos-filesystem/ https://arstechnica.com/information-technology/2018/07/the-b...
- heisenbit 6y agoPortability was important and so more complex capabilities were not sufficiently used to survive and spread. I wonder how the similar struggle ends for advanced cloud features.
- notacoward 6y agoI was expecting another article about the current crop of less than POSIX filesystems, but I was pleasantly surprised to find it was about older systems with features absent in POSIX. Very interesting stuff. Kudos. I wonder if I can find some information about MTS's filesystem, which also had some neat features especially with respect to access control (PKEYs). Might be a worthwhile addition.
- vincent-manis 6y agoThe MTS file system was non hierarchical, and the data model was not stream-of-bytes but rather line-numbered-record file. That said, it had some nice features, including append-only access, and program keys. As I recall, fairly complete MTS documentation can be found at Bitsavers, and one can download a complete MTS distribution to run under emulation, see the Wikipedia article for a link.
- formerly_proven 6y agoThe whole point of the Unix FS was that the developers felt the mainframe approach (~"files are actually sorta like databases") as to complex and heavyweight.
- notacoward 6y agoWhat the OP should teach us is that there wasn't any one mainframe approach. There were many approaches, each involving many components. The process of standardizing what we now know as POSIX involved pruning a lot of unnecessary pieces, but it also inevitably involved leaving out some features that might actually have been useful. Some of them (ahem ACLs) even had to be added back in later versions of POSIX. Just as we should never stop looking for new ideas that can make our lives as programmers and users easier, we should also never forget old ideas whose time might well have come back around. It has happened too many times for the possibility to be ignored except by fools.
- bregma 6y agoEngineering has always been a tug-of-war between parsimony and completeness. The pendulum is constantly swinging.
- amelius 6y agoAll this innovation and meanwhile, when I pull out a USB drive that was not properly unmounted, I risk corrupting it to the point it becomes unusable.
- Animats 6y agoI've wanted somewhat different file system semantics, but close to the POSIX model: - Unit files. The unit of consistency is the entire file. Files are opened for writing, written, and closed. Then then become openable by other programs. Overwriting a file replaces the file as a unit; other readers see the old file until the writer closes and the reader reopens. This is the default and the case for most files. On a crash, the file reverts to the previous good file, if any. Can be memory-mapped as read only. File replacement can be done now through non-portable renaming gyrations. It should just work. - Log files. Append-only mode, enforced. Can't seek back and overwrite. On a crash, the file recovers to some recent good write, with a correct end of file position. It does not tail off into junk. - Temp files. Not persistent over restarts. Not backed up. Can be memory-mapped as read/write. On a crash, disappears. - Managed files. These are for databases. Async I/O available. May have additional locking functions. Separate completion returns for "accepted data" (caller can reuse write buffer) and "committed data" (safely stored), so the database knows when the data has been safely stored. Can be memory-mapped as read/write. Intended for use by programs which are very aware of the file system semantics. On a crash, "committed data" should be intact but data not yet committed may be lost.
- wahern 6y ago> Overwriting a file replaces the file as a unit; other readers see the old file until the writer closes and the reader reopens. This is the semantic rename provides, and the strength of that semantic is a common complaint from many in the anti-POSIX crowd. I suspect you knew that already, but the number of syscalls would be similar in either case except that the latter file would need to be visible before the rename. Linux does provide O_TMPFILE + AT_SYMLINK_FOLLOW which provides the nearly ideal behavior. (Nearly because it still requires /proc/self/fd access, AFAIU.) > Log files. Append-only mode, enforced Both chflags (BSD) and chattr (Linux) provide append-only modes attached to the file/inode (instead of the file descriptor or open file table entry)[1].[2] Neither command nor their options are defined by POSIX, but adding more requirements to POSIX filesystem conformance goes against the grain of prevailing sentiments. > Temp files. Not persistent over restarts. Not backed up. Can be memory-mapped as read/write. On a crash, disappears. Again, the semantics of unlinking deliberately provide this, though it does make it invisible in the normal namespace. But there's a tension between leveraging the namespace for implicit semantics (Plan 9) vs adding a multitude of options and modes attached to each particular file (Windows). It's also not uncommon for temporary filesystems like /tmp to be reformatted on boot if the backing store is persistent. That's one reason to use multiple filesystems for a Unix install. [1] Note that O_APPEND mode is attached to the open file table entry, not the file descriptor, so the mode is inherited across dup and interprocess descriptor passing. Interestingly, on Linux /proc/self/fd (also /dev/fd, which is a symlink) has the semantics of open, not dup, so file table entry state like O_APPEND is lost; whereas on BSD /dev/fd has the semantics of dup so O_APPEND is preserved/enforced. [2] Solaris lacks chattr and chflags, but chmod supports an append-only option.
- LeoPanthera 6y agoAcorn's "ADFS", as used in RISC OS, uses "." as the directory separator. Fully qualified paths look like this: fs::drive_id.$.directory.directory.filename Where "$" means "root directory". (On network filesystems, you can also use "&" which means "home directory".) The top level identifier is the filesystem type, usually "adfs", which is a slightly unusual way of doing it. Bringing in files from other systems, which invariably have filename extensions, involves converting the . to a /, so you end up with filenames like "readme/txt". ADFS stores the file type not as a filename extension, but as a three-character hex ID in the filesystem instead. (Text, for example, is FFF.)
- fanf2 6y agoOn the BBC Micro-era pre-hierarchial DFS, there was a single level of single character directories, so files were called things like “d.file”. $ was the default directory rather than the root. The Acorn/Norcroft ARM C compiler used c and h directories which effectively swapped the file extension around, so sources were named like c.main, and #include would try swapping name and extension for compatibility. It was about as Unix-flavoured as was possible on the Archimedes...
- ChuckMcM 6y agoThis is a good read. I believe that Steve Bourne (of Bourne Shell fame) implemented the file system in UNIX at Bell Labs but I may be mis-remembering that. He was also a big Multics fan. What is unsaid in this article is that nearly all file systems that are in widespread use today, started when "disk space" was a constrained resource. It is a very reasonable thing to ask, "now that stable storage space is much more plentiful, how might we design systems that are better than the current ones?" The ability to scavenge blocks to re-create state is a good example of that. One of the cool things about WAFL (the Write Anywhere File Layout) system that NetApp used (uses?) was that it's very design made snapshots 'trivial' since every write to disk was to un-allocated blocks. What that meant in practice was that the file system on disk was always sane. This was what let you pull the power from the box at any time, and assuming its non-volatile RAM was still available, it could always recover. Something that you could do with Intel Cross Point memory and a bunch of disks. Microsoft research built a number of interesting file systems, some more successful than others, which incorporated many of the ideas from Multics and other OSes. All in all, its a fun place to experiment.
- LargoLasskhyfv 6y agoSounds like [1] https://en.wikipedia.org/wiki/NILFS https://en.wikipedia.org/wiki/NILFS [2] https://nilfs.sourceforge.io/en/ https://nilfs.sourceforge.io/en/ to me. edit: [3] https://github.com/nilfs-dev/ https://github.com/nilfs-dev/
- diegocg 6y agoWAFL trick is COW, which had been popularised massively by ZFS
- mprovost 6y agoOracle paid NetApp a lot of money to settle the patent infringement lawsuit after buying Sun which "borrowed" a lot of WAFL's ideas for ZFS.
- ChuckMcM 6y agoAs you have observed they have an identical design philosophy. There is a not so funny story there where a person interned at NetApp and then went to work at Sun and re-implemented the patented technology as ZFS.
- peter_d_sherman 6y ago>"The very first hierarchical file system was developed for Multics. It is described in the paper A General-Purpose File System For Secondary Storage (1965) by Robert C. Daley and Peter G. Neumann. There are several things that I find astounding about this paper: There were apparently no hierarchical file systems before Multics. The references do no cite any previous work on this and I haven’t found any." (PDS: As an amateur computer historian, I'd be interested in this myself, if there existed any hierarchical file system before Multics, or if one was conceived in any academic paper before the one cited...) [...] >"Directory entries that point to secondary storage. This is a game changer for file system management. More on this below." (PDS: Which, to this day, is one of the great abstractions, one of the great ideas, in computing!)
- nanomonkey 6y agoNo mention of the immutable content addressable hash file system presented in the Artifacts System[]. One of the benefits of a system like this is that you can have multiple versions of a library coexisting without conflict with your requirements system. Each program can reference a different hash, duplicates are easily made unnecessary because they resolve to the same hash, etc. []https://apps.dtic.mil/dtic/tr/fulltext/u2/a276589.pdf https://apps.dtic.mil/dtic/tr/fulltext/u2/a276589.pdf
- narwally 6y agoSounds like having your libraries stored in git repos and then having your programs point to the commit they want to use. Something like this would solve headaches that there are innumerable tools for. A much more elegant solution for the problems things like pyenv or nix are attempting to solve.
- nanomonkey 6y agoYes, exactly, except for local first, and for all your files in your filesystem (documents, libraries, program source code, etc.), and collaborative.
- ruslan 6y agoAnd this guy have not heard about IBM OS/400 and VMS yet :)
- gumby 6y agoHaving used both Multics and the Alto, I can add a few points: Multics as implemented used a slightly different syntax, such as > for path separation. Segments were not file descriptors but were intended to be what might today be called blocks. The original idea was that the entire memory would essentially be a single address space, with segments being blocks of memory that might be in core, on disk, or on tape (called very slow storage if I remember correctly but it’s been decades). Security was at the segment level, so you could for example have (in posix parlance) an suid shared library that an ordinary program could, with appropriate authentication and permissions, call into in a controlled manner. Multics’ permission structure was more fine grained than the binary of suid. You can see how the backup system kind of falls out of this automatically. I trust some multician will step up and correct any memory-corruption-based thinkos in the above. Of course reality didn’t quite match the dream and Multics was cancelled before some of that research could be completed. But if you look at the x86 segment registers they could implement something like that. I think this was also in the intel iapx-430 but fortunately even the name of that ancient dead processor is hazy in my mind. As for the alto filesystem and its descendants: the labels don’t have to be next to the blocks they describe. After all Unix filesystems have multiple copies of their basic breadcrumbs at least. There’s certainly plenty of room for these in modern storage systems; every modern drive, whether spinning or ssd, does a small amount of this in block remapping and wear leveling.
- vermilingua 6y ago> Why can your browser run sudo? ...Suppose only your SSH client had the operation required to use your SSH keys. Perhaps I’m missing something, but why could a malicious application (say, Chrome in the context of a browser running sudo) not also include the capability to access the file system with impunity? It seems that Hydra simply turns the technical problem of managing filesystem protections into a political one, of auditing the applications we use; which may have been practical in the 80’s, but is now demonstrably a failed method of protecting user data.
- venamresm__ 6y agoThis is a great article. For anyone wondering about the current data storage stack on Unix-like systems today, I wrote a simple article that goes over it [1]. [1] https://venam.nixers.net/blog/unix/2017/11/05/unix-filesystem.html https://venam.nixers.net/blog/unix/2017/11/05/unix-filesyste...
- t0astbread 6y agoWhat I really want that isn't there in POSIX is not directly a file system feature but something more general: I really want some sort of "middleware-system" to intercept all sorts of events (like file system access, binary execution, network or device access etc.). There should be multiple intercepting programs that handle one or more events each and can decide to block or pass the event to the next interceptor. They could also log or modify parameters of the event (like redirect a file read or wrap a binary that's about to be executed in another program like script or torsocks). You could even use this system to implement some Unix features as interceptors: Shebangs and even file system permissions could be handled this way. You could also implement containers with this or provide some kind of "switchboard" UI akin to uMatrix for letting the user decide on permissions.
- deepstack 6y ago> When you use a file system through a library instead of going through the operating system there are some extra possibilities. You are no longer required to obey the host operating system’s semantics for filenames. You get to decide if you use / or \ to separate directory components (or something else altogether) This is really about time in 2020 that someone (Apple, MS, Linux, FreeBSD, Plan 9 I'm looking at you) implemented this!
- qubex 6y agoFor anybody who has the desire and opportunity to interact with something currently in production and truly alien-feeling, I warmly suggest IBM’s AS/400 (now iSeries) OS/400’s filesystem. Library versus folders, logical versus physical files, integrated DB2 database is one of the many, many head-scratchers you’ll encounter until you suddenly “_get it_”.
- GuB-42 6y agoMaybe another filesystem worth mentioning would be WinFS. It should have been one of the major features of Windows Longhorn, which later became Vista. Simply put, it was a relational database, with features you typically find in RDBMS like Postgres or Microsoft's own SQL Server, which took advantage of some of the work done on WinFS.