4 ms·

Next in date after FFS, I consider as one of the most innovative and influential filesystems the HPFS (High-Performance File System), which was introduced in No

Next in date after FFS, I consider as one of the most innovative and influential filesystems the HPFS (High-Performance File System), which was introduced in November 1989, together with the OS/2 version 1.2 operating system. Among other new features, HPFS has introduced extended file attributes and B-tree directories (both features are ubiquitous in modern filesystems). Two very important filesystems that have been heavily influenced by HPFS have been Microsoft NTFS and Silicon Graphics XFS, both introduced in 1993. XFS combined ideas from HPFS with some from the earlier Silicon Graphics filesystem, Extent FS, which had been introduced in October 1988 and which had much better performance for big files than the Berkeley FFS (because file blocks were allocated sequentially whenever possible, instead of being addressed indirectly and scattered on the disk). Journaling in filesystems has been introduced by IBM, in JFS (Journaled FS, February 1990). An influential research report was "Beating the I/O Bottleneck: A Case for Log-Structured File Systems", by John K. Ousterhout & Fred Douglis, written in 1988-10 at Berkeley. A paper derived from it was presented at USENIX in 1990-06.

kragen 2y ago
Didn't the Oberon filesystem have B-tree directories before HPFS? And what do you mean by "extended file attributes"? Nowadays I interpret that phrase as meaning "any user-level file metadata that isn't part of 6th Edition Unix", such as ACLs or the immutability bit, but by that definition almost every pre-Unix filesystem had "extended file attributes". https://en.m.wikipedia.org/wiki/Extended_file_attributes https://en.m.wikipedia.org/wiki/Extended_file_attributes has a definition which sounds like it might be what you mean? It also sounds like you could easily implement it as a library, though, instead of including it in the filesystem.
- adrian_b 2y ago
  Extended file attributes can contain arbitrary data associated with the file, whatever the user chooses, unlike file metadata with fixed content, like file name, file size, file owner, time of last modification and so on. I am not aware of any earlier filesystem with extended file attributes. What you have mentioned are just examples of fixed file metadata, which are defined by the operating system and which cannot be defined or extended by the user. The only earlier similar feature is the resource fork of the Mac OS files, but that had a different purpose and it was accessed in a different way than extended file attributes, which are a set of name + value pairs, so they are accessed with a special get/set API, not with file read/write functions. Once a method for storing arbitrary file metadata exists, it can also be used to implement features like access control lists or to implement any of the legacy file metadata. The Oberon file system has been described very summarily in a paper published in 1989-09, i.e. 2 months before the commercial introduction of HPFS by IBM and Microsoft, which must have worked at HPFS for a few years before that. Therefore these 2 filesystems must have been developed completely independently and the idea of using B-tree directories, which both teams have presented as novel, must have occurred independently to them. While the Oberon filesystem has the merit of introducing independently and simultaneously the same improvement in directory implementation, its historical impact has been very reduced in comparison with HPFS, which had a significant user base and which became well known immediately, all over the world, influencing strongly all the filesystems developed after it.
  - kragen 2y ago
    If you just want to store arbitrary name-value pairs associated with a file, it seems like you could use any filesystem to do that. Here's an example implementation: def get(filename, key): try: with open(filename + ".ea") as f: metadata = json.load(f) except FileNotFoundError: raise KeyError(key) return metadata[key] def set(filename, key, value): ... # left as an exercise to the reader A fully robust implementation of the facility requires handling race conditions between concurrent writers, eventual metadata removal when the underlying file is removed, some kind of thinking about how permissions to the file and permissions to the extended attributes relate, and keeping the extended-attribute storage files from cluttering up the usual user interfaces. Still, given how trivial it is to implement the facility if it isn't built into the filesystem, building it into the filesystem seems more like a tradeoff between obvious design alternatives and less like an innovation. There are lots of ways you can do things like this; Windows 95 started stashing long-filename metadata in the filesystem directory as extra invalid hidden system volume-name read-only filename entries, for example, and 4DOS (and Total Commander) stored timestamps and arbitrarily-long file descriptions in a file in every directory called DESCRIPT.ION. (With respect to the concurrency and robustness issues, keep in mind that HPFS would regularly lose data if you lost power, disk drives at the time would regularly suffer physical damage if you lost power unless the disk head was parked, common implementations of Pascal at the time would fail to flush buffered I/O data if you failed to close files, PCs mostly weren't multitasking, and users could remove floppy disks from drives at any time without warning, so even a pretty non-robust implementation would have been adequate.) As for Oberon's filesystem, it wasn't quite the same improvement in directory implementation; in Oberon there is one directory per disk, while in HPFS there is one directory per folder. This may sound like its filesystem was non-hierarchical, but if I recall correctly, it is in fact hierarchical; files are stored under their pathnames in that single directory. I thought that probably, if Oberon was described so late, its predecessor probably did the same thing, but I just checked out Svend Erik Knudsen's dissertation (ETH No. 7346) on Medos-2 4.2 https://oberoncore.ru/_media/library/knudsen_medos-2_a_modula-2_oriented_operating_system_for_the_personal_computer_lilith.pdf https://oberoncore.ru/_media/library/knudsen_medos-2_a_modul... and p. 67, at the tail end of §6.2, describes the file name directory of Medos's DiskSystem. There is no mention at all of B-trees. (There is also a "file directory", but it's confusingly named; what it contains is basically inodes.) It would probably be nice if HPFS had strongly influenced all the filesystems developed after it, but in fact it seems to have had virtually no influence on Sprite-LFS (begun 01990), BSD LFS, ext2fs, FAT32, VFAT, NetApp's WAFL, GFS, HDFS, Ceph, yaffs, or jffs2. I'm sure it had a lot of influence on NTFS. I don't know enough about ZFS, Reiserfs, XFS, HFS+, or exFAT to say; do you? Can you think of any other filesystems that were strongly influenced by HPFS other than NTFS? Filesystem design seems to be largely a matter of standing on giants' feet.

kragen 2y ago

Didn't the Oberon filesystem have B-tree directories before HPFS? And what do you mean by "extended file attributes"? Nowadays I interpret that phrase as meaning "any user-level file metadata that isn't part of 6th Edition Unix", such as ACLs or the immutability bit, but by that definition almost every pre-Unix filesystem had "extended file attributes". https://en.m.wikipedia.org/wiki/Extended_file_attributes https://en.m.wikipedia.org/wiki/Extended_file_attributes has a definition which sounds like it might be what you mean? It also sounds like you could easily implement it as a library, though, instead of including it in the filesystem.

adrian_b 2y ago
Extended file attributes can contain arbitrary data associated with the file, whatever the user chooses, unlike file metadata with fixed content, like file name, file size, file owner, time of last modification and so on. I am not aware of any earlier filesystem with extended file attributes. What you have mentioned are just examples of fixed file metadata, which are defined by the operating system and which cannot be defined or extended by the user. The only earlier similar feature is the resource fork of the Mac OS files, but that had a different purpose and it was accessed in a different way than extended file attributes, which are a set of name + value pairs, so they are accessed with a special get/set API, not with file read/write functions. Once a method for storing arbitrary file metadata exists, it can also be used to implement features like access control lists or to implement any of the legacy file metadata. The Oberon file system has been described very summarily in a paper published in 1989-09, i.e. 2 months before the commercial introduction of HPFS by IBM and Microsoft, which must have worked at HPFS for a few years before that. Therefore these 2 filesystems must have been developed completely independently and the idea of using B-tree directories, which both teams have presented as novel, must have occurred independently to them. While the Oberon filesystem has the merit of introducing independently and simultaneously the same improvement in directory implementation, its historical impact has been very reduced in comparison with HPFS, which had a significant user base and which became well known immediately, all over the world, influencing strongly all the filesystems developed after it.
- kragen 2y ago
  If you just want to store arbitrary name-value pairs associated with a file, it seems like you could use any filesystem to do that. Here's an example implementation: def get(filename, key): try: with open(filename + ".ea") as f: metadata = json.load(f) except FileNotFoundError: raise KeyError(key) return metadata[key] def set(filename, key, value): ... # left as an exercise to the reader A fully robust implementation of the facility requires handling race conditions between concurrent writers, eventual metadata removal when the underlying file is removed, some kind of thinking about how permissions to the file and permissions to the extended attributes relate, and keeping the extended-attribute storage files from cluttering up the usual user interfaces. Still, given how trivial it is to implement the facility if it isn't built into the filesystem, building it into the filesystem seems more like a tradeoff between obvious design alternatives and less like an innovation. There are lots of ways you can do things like this; Windows 95 started stashing long-filename metadata in the filesystem directory as extra invalid hidden system volume-name read-only filename entries, for example, and 4DOS (and Total Commander) stored timestamps and arbitrarily-long file descriptions in a file in every directory called DESCRIPT.ION. (With respect to the concurrency and robustness issues, keep in mind that HPFS would regularly lose data if you lost power, disk drives at the time would regularly suffer physical damage if you lost power unless the disk head was parked, common implementations of Pascal at the time would fail to flush buffered I/O data if you failed to close files, PCs mostly weren't multitasking, and users could remove floppy disks from drives at any time without warning, so even a pretty non-robust implementation would have been adequate.) As for Oberon's filesystem, it wasn't quite the same improvement in directory implementation; in Oberon there is one directory per disk, while in HPFS there is one directory per folder. This may sound like its filesystem was non-hierarchical, but if I recall correctly, it is in fact hierarchical; files are stored under their pathnames in that single directory. I thought that probably, if Oberon was described so late, its predecessor probably did the same thing, but I just checked out Svend Erik Knudsen's dissertation (ETH No. 7346) on Medos-2 4.2 https://oberoncore.ru/_media/library/knudsen_medos-2_a_modula-2_oriented_operating_system_for_the_personal_computer_lilith.pdf https://oberoncore.ru/_media/library/knudsen_medos-2_a_modul... and p. 67, at the tail end of §6.2, describes the file name directory of Medos's DiskSystem. There is no mention at all of B-trees. (There is also a "file directory", but it's confusingly named; what it contains is basically inodes.) It would probably be nice if HPFS had strongly influenced all the filesystems developed after it, but in fact it seems to have had virtually no influence on Sprite-LFS (begun 01990), BSD LFS, ext2fs, FAT32, VFAT, NetApp's WAFL, GFS, HDFS, Ceph, yaffs, or jffs2. I'm sure it had a lot of influence on NTFS. I don't know enough about ZFS, Reiserfs, XFS, HFS+, or exFAT to say; do you? Can you think of any other filesystems that were strongly influenced by HPFS other than NTFS? Filesystem design seems to be largely a matter of standing on giants' feet.