5 ms·

50 Years in Filesystems: 1984

198 points by kaycebasques 2y ago

gjvc 2y ago
As the old saying says: "You can tune a filesystem, but you can't tune a fish."
- sadboots 2y ago
  lol
- catgirlinspace 2y ago
  is there some metaphor i’m missing or is it just funny?
  - CraigJPerry 2y ago
    tuna fish Tune-a-fish EDIT: looks like there’s a bit more to it https://unixhistory.livejournal.com/1808.html https://unixhistory.livejournal.com/1808.html
  - deleted 2y ago
    [deleted]
  - kps 2y ago
    The phrase originates with the ‘BUGS’ section of the 4.2BSD man page for the `tunefs(8)` command, for the Berkeley Fast File System that is the topic of the OP article.
    - gjvc 2y ago
      This, but in true to form HN style, if you don't recognise the reference, it's psychologically safer to down vote what you don't know.
      - macintux 2y ago
        Even if you do recognize it, this isn’t Reddit.
        gjvc 2y ago
        /r/confidentlyincorrect
      - luqtas 2y ago
        karma worth a little on the influencer age
  - jhbadger 2y ago
    It's a reference to a joke by the comedian Groucho Marx (1890-1977) "You can tune a piano but you can't tuna fish" (also a title of a 1978 REO Speedwagon album paying homage to Marx who had recently died). It's a pun on "tune a" and "tuna".
- cozzyd 2y ago
  Sure you can, https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/performance_tuning_guide/chap-tuna#chap-Tuna https://docs.redhat.com/en/documentation/red_hat_enterprise_...
- azinman2 2y ago
  Could it come from this: https://en.wikipedia.org/wiki/You_Can_Tune_a_Piano,_but_You_Can%27t_Tuna_Fish https://en.wikipedia.org/wiki/You_Can_Tune_a_Piano,_but_You_... ?
pjdesno 2y ago
Note that there are some significant non-Unix filesystems along the way. In particular, a few years later: "Reimplementing the Cedar File System Using Logging and Group Commit", Robert Hagmann, Xerox PARC. SIGOPS Operating Systems Review, November 1987. That's the origin of the journaling approach used in post-FFS unix file systems through the 90s. (I don't count ext2 as "post-FFS", as it's basically FFS on a disk with LBA addressing)
- gjvc 2y ago
  Xerox PARC strikes again
  - pjmlp 2y ago
    Still waiting for the Mesa, Cedar, Smalltalk, Interlisp-D set of approaches software development, OSes and systems programming to take over, but I guess we can't have everything at once, and should be happy with what we already got into mainstream out of Xerox PARC.
    - gjvc 2y ago
      They don't need to take over in order to be a wild success. I'm reminded of the 1997 speech from Steve Jobs where he said "We have to let go of this notion that for Apple to win, Microsoft has to lose". Perhaps AIs will recognise their value and convince people! :-)
- convolvatron 2y ago
  wow. I always thought this was due to M. Rosenblum in 1992. Looking at the paper now it doesn't reference this one. I wonder what other group ideation happened in the late 80s around this. I guess it was already well established in databases.
  - Upvoter33 2y ago
    Journaling and log-structured file systems are different (but related) things
bvrmn 2y ago
[flagged]
- deleted 2y ago
  [deleted]
teleforce 2y ago
Fun facts, back in 2021 Linus was asked what is the most special about Linux in an interview for 30 years of Linux, and he pointed to the Linux File System as the fastest in the business [1]. [1] 30 Years of Linux: An Interview With Linus Torvalds: Linux and Git - Part 1: https://www.tag1consulting.com/blog/interview-linus-torvalds-linux-and-git https://www.tag1consulting.com/blog/interview-linus-torvalds...
AdieuToLogic 2y ago
FFS earned its flowers, no doubt, and should be respected for its historic contributions. FreeBSD's ZFS[0] is really awesome and has proven to be a deserving successor in all scenarios when legacy needs[1] are not a concern. 0 - https://docs.freebsd.org/en/books/handbook/zfs/ https://docs.freebsd.org/en/books/handbook/zfs/ 1 - https://docs.freebsd.org/en/books/handbook/filesystems/ https://docs.freebsd.org/en/books/handbook/filesystems/
- pjdesno 2y ago
  There are a couple of generations of filesystems in between FFS and ZFS. And for modern filesystems it's also worth looking at btrfs, Microsoft's Resilient File System, and Apple APFS. Also note that ext2 is basically a re-implementation of FFS, without the interleaving because it's designed for disks with integrated controllers.
  - AdieuToLogic 2y ago
    > There are a couple of generations of filesystems in between FFS and ZFS. Quite true. I did not mean to imply ZFS was the immediate successor to FFS, but instead that it is a superior choice currently. > And for modern filesystems it's also worth looking at btrfs, Microsoft's Resilient File System, and Apple APFS. All good options and which to consider is applicable when running Linux, MS-Windows, and macOS respectively. If FreeBSD+ZFS is a possibility though... :-)
    - deleted 2y ago
      [deleted]
  - adrian_b 2y ago
    Next in date after FFS, I consider as one of the most innovative and influential filesystems the HPFS (High-Performance File System), which was introduced in November 1989, together with the OS/2 version 1.2 operating system. Among other new features, HPFS has introduced extended file attributes and B-tree directories (both features are ubiquitous in modern filesystems). Two very important filesystems that have been heavily influenced by HPFS have been Microsoft NTFS and Silicon Graphics XFS, both introduced in 1993. XFS combined ideas from HPFS with some from the earlier Silicon Graphics filesystem, Extent FS, which had been introduced in October 1988 and which had much better performance for big files than the Berkeley FFS (because file blocks were allocated sequentially whenever possible, instead of being addressed indirectly and scattered on the disk). Journaling in filesystems has been introduced by IBM, in JFS (Journaled FS, February 1990). An influential research report was "Beating the I/O Bottleneck: A Case for Log-Structured File Systems", by John K. Ousterhout & Fred Douglis, written in 1988-10 at Berkeley. A paper derived from it was presented at USENIX in 1990-06.
deleted 2y ago
[deleted]
kazinator 2y ago
This is kind of sparse, pardon the pun. The previous chapter covers one Unix filesystem and that's it, as if that's all there was in 1974-1984. The 1984 chapter is again just one BSD Unix filesystem. How about something lesser known, but excellent?
WaitWaitWha 2y ago
I am so confused. File systems or filesystems? I do not recall anything in my AT&T days referred to as "filesystems". when I read the article I was expecting mentions of CP/M, Files-11, TOPS-10, UFS, VSAM, ISAM, BDAM, FAT? MFS for Macs showed up in 1984, Amiga's OFS followed. I mean, not even UFS, which showed up in the 70s, yet still had impact on ext and NTFS... I was trying to see if this article was about history of file systemS from the 70s to through the 80s, or just a single file system's evolution; alas I could not tell.
- lproven 2y ago
  I think you did not read it closely enough. The article says right at the top that it is part 2 in a series, and that part 1 was 1974, with a link. That Part 1 discussed UFS in depth. The point being that (1) by 1984 UFS was far from state of the art and (2) you need to understand the 50 year old SOTA to understand what was different in the 40YO SOTA.
- unwind 2y ago
  The two spellings are the same. First sentence from the Wikipedia entry [1]: In computing, a file system or filesystem (often abbreviated to FS or fs) governs file organization and access. [1]: https://en.wikipedia.org/wiki/File_system https://en.wikipedia.org/wiki/File_system
michelb 2y ago
What are the latest developments on this front? Still ZFS for stability, XFS for performance? Now that we have NAND storage, would we create filesystems differently? Are there ways to make a step-change in overall performance somewhere? Disclaimer: I know nothing about filesystems..
- mtillman 2y ago
  XFS is a file system and ZFS is both a file system and a volume manager. ZFS also recently added RAIDZ expansion which I was really excited about.
- avar 2y ago
  If you know nothing about filesystems, then you're almost certainly limited by the underlying hardware, not the filesystem. And "performance" is meaningless in this context without specifying what you want to perform. Raw sequential read/write throughput, random metadata operations on directories containing millions of files? Something in between? > Are there ways to make a step-change in > overall performance somewhere? Yes, it's called O_DIRECT (or equivalent), and is very useful to those that actually need it. A general-use filesystem is inherently something that requires you to compromise on performance.
- 6SixTy 2y ago
  A lot of NAND SSDs have controllers that pretty much take away a lot of control the rest of the system has over the actual flash cells. Pretty much any flash specific file systems are designed for embedded applications where there is no controller to perform wear leveling or TRIM operations. Pretty much most of the benefits that could be gained by now are at the application level maybe the CPU, not the OS nor filesystem.
- evil-olive 2y ago
  > Now that we have NAND storage, would we create filesystems differently? Are there ways to make a step-change in overall performance somewhere? you might enjoy Allan Jude's "Scaling ZFS for NVMe" talk [0] from 2022. the tl;dw is that modern NVMe drives have gotten so fast that filesystem developers are having to revisit assumptions that have been true for decades about which operations will be slow vs. fast. flash storage that exposed a SATA interface and pretended to be a superbly fast hard drive was an evolutionary improvement of sorts. NVMe is more of a revolutionary improvement, it does away with the spinning-rust assumptions. for example, SATA limits a device to one command queue with 32 commands; NVMe makes it almost unlimited (64k queues with up to 64k commands) [1] there is also a corner of the NVMe spec [2] that allows you to treat an SSD directly as key-value storage instead of block storage with a filesystem implementing a key-value layer on top of it. this is promising but AFAIK no mainstream filesystem is using it yet, and I'm not sure if support has even trickled down into consumer drives or if it's market-segmented off for datacenter use only. 0: https://www.youtube.com/watch?v=v8sl8gj9UnA https://www.youtube.com/watch?v=v8sl8gj9UnA 1: https://en.wikipedia.org/wiki/NVM_Express#Comparison_with_AHCI https://en.wikipedia.org/wiki/NVM_Express#Comparison_with_AH... 2: https://nvmexpress.org/specification/key-value-command-set-specification/ https://nvmexpress.org/specification/key-value-command-set-s...
curtisszmania 2y ago
[dead]
blacklion 2y ago
Looks like there is 1994 (XFS) but no 2004 (ZFS? btrfs?) or later installments :-(
- yencabulator 2y ago
  https://blog.koehntopp.info/2023/05/15/50-years-in-filesystems-vnodes.html https://blog.koehntopp.info/2023/05/15/50-years-in-filesyste... https://blog.koehntopp.info/2023/05/17/50-years-in-filesystems-towards-2004-lfs.html https://blog.koehntopp.info/2023/05/17/50-years-in-filesyste...
  - isotopp 2y ago
    https://blog.koehntopp.info/2023/05/05/50-years-in-filesystems-1974.html https://blog.koehntopp.info/2023/05/05/50-years-in-filesyste... https://blog.koehntopp.info/2023/05/06/50-years-in-filesystems-1984.html https://blog.koehntopp.info/2023/05/06/50-years-in-filesyste... https://blog.koehntopp.info/2023/05/12/50-years-in-filesystems-1994.html https://blog.koehntopp.info/2023/05/12/50-years-in-filesyste... https://blog.koehntopp.info/2023/05/15/50-years-in-filesystems-vnodes.html https://blog.koehntopp.info/2023/05/15/50-years-in-filesyste... https://blog.koehntopp.info/2023/05/17/50-years-in-filesystems-towards-2004-lfs.html https://blog.koehntopp.info/2023/05/17/50-years-in-filesyste...
  - blacklion 2y ago
    Oh, I missed 2004, thank you.

198 points by kaycebasques 2y ago

gjvc 2y ago

As the old saying says: "You can tune a filesystem, but you can't tune a fish."

sadboots 2y ago
lol
catgirlinspace 2y ago
is there some metaphor i’m missing or is it just funny?
- CraigJPerry 2y ago
  tuna fish Tune-a-fish EDIT: looks like there’s a bit more to it https://unixhistory.livejournal.com/1808.html https://unixhistory.livejournal.com/1808.html
- deleted 2y ago
  [deleted]
- kps 2y ago
  The phrase originates with the ‘BUGS’ section of the 4.2BSD man page for the `tunefs(8)` command, for the Berkeley Fast File System that is the topic of the OP article.
  - gjvc 2y ago
    This, but in true to form HN style, if you don't recognise the reference, it's psychologically safer to down vote what you don't know.
    - macintux 2y ago
      Even if you do recognize it, this isn’t Reddit.
      - gjvc 2y ago
        /r/confidentlyincorrect
    - luqtas 2y ago
      karma worth a little on the influencer age
- jhbadger 2y ago
  It's a reference to a joke by the comedian Groucho Marx (1890-1977) "You can tune a piano but you can't tuna fish" (also a title of a 1978 REO Speedwagon album paying homage to Marx who had recently died). It's a pun on "tune a" and "tuna".
cozzyd 2y ago
Sure you can, https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/performance_tuning_guide/chap-tuna#chap-Tuna https://docs.redhat.com/en/documentation/red_hat_enterprise_...
azinman2 2y ago
Could it come from this: https://en.wikipedia.org/wiki/You_Can_Tune_a_Piano,_but_You_Can%27t_Tuna_Fish https://en.wikipedia.org/wiki/You_Can_Tune_a_Piano,_but_You_... ?

pjdesno 2y ago

Note that there are some significant non-Unix filesystems along the way. In particular, a few years later: "Reimplementing the Cedar File System Using Logging and Group Commit", Robert Hagmann, Xerox PARC. SIGOPS Operating Systems Review, November 1987. That's the origin of the journaling approach used in post-FFS unix file systems through the 90s. (I don't count ext2 as "post-FFS", as it's basically FFS on a disk with LBA addressing)

gjvc 2y ago
Xerox PARC strikes again
- pjmlp 2y ago
  Still waiting for the Mesa, Cedar, Smalltalk, Interlisp-D set of approaches software development, OSes and systems programming to take over, but I guess we can't have everything at once, and should be happy with what we already got into mainstream out of Xerox PARC.
  - gjvc 2y ago
    They don't need to take over in order to be a wild success. I'm reminded of the 1997 speech from Steve Jobs where he said "We have to let go of this notion that for Apple to win, Microsoft has to lose". Perhaps AIs will recognise their value and convince people! :-)
convolvatron 2y ago
wow. I always thought this was due to M. Rosenblum in 1992. Looking at the paper now it doesn't reference this one. I wonder what other group ideation happened in the late 80s around this. I guess it was already well established in databases.
- Upvoter33 2y ago
  Journaling and log-structured file systems are different (but related) things

bvrmn 2y ago

[flagged]

deleted 2y ago
[deleted]

teleforce 2y ago

Fun facts, back in 2021 Linus was asked what is the most special about Linux in an interview for 30 years of Linux, and he pointed to the Linux File System as the fastest in the business [1]. [1] 30 Years of Linux: An Interview With Linus Torvalds: Linux and Git - Part 1: https://www.tag1consulting.com/blog/interview-linus-torvalds-linux-and-git https://www.tag1consulting.com/blog/interview-linus-torvalds...

AdieuToLogic 2y ago

FFS earned its flowers, no doubt, and should be respected for its historic contributions. FreeBSD's ZFS[0] is really awesome and has proven to be a deserving successor in all scenarios when legacy needs[1] are not a concern. 0 - https://docs.freebsd.org/en/books/handbook/zfs/ https://docs.freebsd.org/en/books/handbook/zfs/ 1 - https://docs.freebsd.org/en/books/handbook/filesystems/ https://docs.freebsd.org/en/books/handbook/filesystems/

pjdesno 2y ago
There are a couple of generations of filesystems in between FFS and ZFS. And for modern filesystems it's also worth looking at btrfs, Microsoft's Resilient File System, and Apple APFS. Also note that ext2 is basically a re-implementation of FFS, without the interleaving because it's designed for disks with integrated controllers.
- AdieuToLogic 2y ago
  > There are a couple of generations of filesystems in between FFS and ZFS. Quite true. I did not mean to imply ZFS was the immediate successor to FFS, but instead that it is a superior choice currently. > And for modern filesystems it's also worth looking at btrfs, Microsoft's Resilient File System, and Apple APFS. All good options and which to consider is applicable when running Linux, MS-Windows, and macOS respectively. If FreeBSD+ZFS is a possibility though... :-)
  - deleted 2y ago
    [deleted]
- adrian_b 2y ago
  Next in date after FFS, I consider as one of the most innovative and influential filesystems the HPFS (High-Performance File System), which was introduced in November 1989, together with the OS/2 version 1.2 operating system. Among other new features, HPFS has introduced extended file attributes and B-tree directories (both features are ubiquitous in modern filesystems). Two very important filesystems that have been heavily influenced by HPFS have been Microsoft NTFS and Silicon Graphics XFS, both introduced in 1993. XFS combined ideas from HPFS with some from the earlier Silicon Graphics filesystem, Extent FS, which had been introduced in October 1988 and which had much better performance for big files than the Berkeley FFS (because file blocks were allocated sequentially whenever possible, instead of being addressed indirectly and scattered on the disk). Journaling in filesystems has been introduced by IBM, in JFS (Journaled FS, February 1990). An influential research report was "Beating the I/O Bottleneck: A Case for Log-Structured File Systems", by John K. Ousterhout & Fred Douglis, written in 1988-10 at Berkeley. A paper derived from it was presented at USENIX in 1990-06.

deleted 2y ago

[deleted]

kazinator 2y ago

This is kind of sparse, pardon the pun. The previous chapter covers one Unix filesystem and that's it, as if that's all there was in 1974-1984. The 1984 chapter is again just one BSD Unix filesystem. How about something lesser known, but excellent?

WaitWaitWha 2y ago

I am so confused. File systems or filesystems? I do not recall anything in my AT&T days referred to as "filesystems". when I read the article I was expecting mentions of CP/M, Files-11, TOPS-10, UFS, VSAM, ISAM, BDAM, FAT? MFS for Macs showed up in 1984, Amiga's OFS followed. I mean, not even UFS, which showed up in the 70s, yet still had impact on ext and NTFS... I was trying to see if this article was about history of file systemS from the 70s to through the 80s, or just a single file system's evolution; alas I could not tell.

lproven 2y ago
I think you did not read it closely enough. The article says right at the top that it is part 2 in a series, and that part 1 was 1974, with a link. That Part 1 discussed UFS in depth. The point being that (1) by 1984 UFS was far from state of the art and (2) you need to understand the 50 year old SOTA to understand what was different in the 40YO SOTA.
unwind 2y ago
The two spellings are the same. First sentence from the Wikipedia entry [1]: In computing, a file system or filesystem (often abbreviated to FS or fs) governs file organization and access. [1]: https://en.wikipedia.org/wiki/File_system https://en.wikipedia.org/wiki/File_system

michelb 2y ago

What are the latest developments on this front? Still ZFS for stability, XFS for performance? Now that we have NAND storage, would we create filesystems differently? Are there ways to make a step-change in overall performance somewhere? Disclaimer: I know nothing about filesystems..

mtillman 2y ago
XFS is a file system and ZFS is both a file system and a volume manager. ZFS also recently added RAIDZ expansion which I was really excited about.
avar 2y ago
If you know nothing about filesystems, then you're almost certainly limited by the underlying hardware, not the filesystem. And "performance" is meaningless in this context without specifying what you want to perform. Raw sequential read/write throughput, random metadata operations on directories containing millions of files? Something in between? > Are there ways to make a step-change in > overall performance somewhere? Yes, it's called O_DIRECT (or equivalent), and is very useful to those that actually need it. A general-use filesystem is inherently something that requires you to compromise on performance.
6SixTy 2y ago
A lot of NAND SSDs have controllers that pretty much take away a lot of control the rest of the system has over the actual flash cells. Pretty much any flash specific file systems are designed for embedded applications where there is no controller to perform wear leveling or TRIM operations. Pretty much most of the benefits that could be gained by now are at the application level maybe the CPU, not the OS nor filesystem.
evil-olive 2y ago
> Now that we have NAND storage, would we create filesystems differently? Are there ways to make a step-change in overall performance somewhere? you might enjoy Allan Jude's "Scaling ZFS for NVMe" talk [0] from 2022. the tl;dw is that modern NVMe drives have gotten so fast that filesystem developers are having to revisit assumptions that have been true for decades about which operations will be slow vs. fast. flash storage that exposed a SATA interface and pretended to be a superbly fast hard drive was an evolutionary improvement of sorts. NVMe is more of a revolutionary improvement, it does away with the spinning-rust assumptions. for example, SATA limits a device to one command queue with 32 commands; NVMe makes it almost unlimited (64k queues with up to 64k commands) [1] there is also a corner of the NVMe spec [2] that allows you to treat an SSD directly as key-value storage instead of block storage with a filesystem implementing a key-value layer on top of it. this is promising but AFAIK no mainstream filesystem is using it yet, and I'm not sure if support has even trickled down into consumer drives or if it's market-segmented off for datacenter use only. 0: https://www.youtube.com/watch?v=v8sl8gj9UnA https://www.youtube.com/watch?v=v8sl8gj9UnA 1: https://en.wikipedia.org/wiki/NVM_Express#Comparison_with_AHCI https://en.wikipedia.org/wiki/NVM_Express#Comparison_with_AH... 2: https://nvmexpress.org/specification/key-value-command-set-specification/ https://nvmexpress.org/specification/key-value-command-set-s...

curtisszmania 2y ago

[dead]

blacklion 2y ago

Looks like there is 1994 (XFS) but no 2004 (ZFS? btrfs?) or later installments :-(

yencabulator 2y ago
https://blog.koehntopp.info/2023/05/15/50-years-in-filesystems-vnodes.html https://blog.koehntopp.info/2023/05/15/50-years-in-filesyste... https://blog.koehntopp.info/2023/05/17/50-years-in-filesystems-towards-2004-lfs.html https://blog.koehntopp.info/2023/05/17/50-years-in-filesyste...
- isotopp 2y ago
  https://blog.koehntopp.info/2023/05/05/50-years-in-filesystems-1974.html https://blog.koehntopp.info/2023/05/05/50-years-in-filesyste... https://blog.koehntopp.info/2023/05/06/50-years-in-filesystems-1984.html https://blog.koehntopp.info/2023/05/06/50-years-in-filesyste... https://blog.koehntopp.info/2023/05/12/50-years-in-filesystems-1994.html https://blog.koehntopp.info/2023/05/12/50-years-in-filesyste... https://blog.koehntopp.info/2023/05/15/50-years-in-filesystems-vnodes.html https://blog.koehntopp.info/2023/05/15/50-years-in-filesyste... https://blog.koehntopp.info/2023/05/17/50-years-in-filesystems-towards-2004-lfs.html https://blog.koehntopp.info/2023/05/17/50-years-in-filesyste...
- blacklion 2y ago
  Oh, I missed 2004, thank you.