6 ms·

Practical File System Design: The Be File System (1999) [pdf]

101 points by e-sushi 10y ago

jhugg 10y ago
This is a fantastic book, but do we have the rights to it? What is this URL?
- adamnemecek 10y ago
  It's the author's website, I assume yes.
- edu 10y ago
  Yes, the author published openly a few years ago, when it got out of print. Actually, the link is to the authors site: http://www.nobius.org/~dbg/index.html http://www.nobius.org/~dbg/index.html
appleflaxen 10y ago
I remember reading about the BeFS, and how advanced it was for its time. Can anyone who does filesystems help me remember why it was so powerful? Is anything out there now comparable? Have we surpassed BeFS? Any residual functionality that contemporary filesystems lack?
- jonhohle 10y ago
  It had a lot of features for the time (64-bit, journaling, etc.), but one of the most used features in BeOS was that arbitrary metadata could be attached to files, exposed in Tracker or other applications, and could be used to search for files. One of the examples of the time were scripts to take MP3 ID3 tags and store them as metadata. Email was stored as individual files on the file system, and the email client was just a Tracker window with columns displaying email relevant metadata. Spotlight on OS X supports similar metadata storage and querying, but it's built on top of HFS+, not directly into the file system (and surprise, was built by Dominic Giampaolo, who worked on BeFS, author of the linked book). WinFS claimed a lot of similar features shortly after Be was acquired, but to my knowledge, no file system has provided similar features _and_ had spextensive application support to leverage them.
  - geocar 10y ago
    I think the clever bit is when a user sees their email in the file browser[1] they can mentally connect with that and start using their file system as the database that it is, building their own dynamic todo list[2] that isn't managed by any application! Almost every filesystem "supports metadata", and while I agree that application support could be better, I think the real key is getting users to use that space themselves. [1]: https://birdhouse.org/beos/refugee/bemail.jpg https://birdhouse.org/beos/refugee/bemail.jpg [2]: https://birdhouse.org/beos/refugee/trackerbase.gif https://birdhouse.org/beos/refugee/trackerbase.gif
    - PeCaN 10y ago
      BeFS's metadata support was much different from usual—it was a key/value map attached to each file. NTFS supports extended attributes and alternate data streams, which you could use to do the same thing, but none of the *nix filesystems have anything like that. What sort of metadata do you mean with “almost every filesystem ‘supports metadata’”?
      - acdha 10y ago
        JFS, XFS, ReiserFS, ext3/4, and btrfs all support extended attributes – that's become a bigger deal for things like ACLs – but along with varying size limits the share the same limitations as NTFS, HFS+, etc. of not indexing those values. That means there's no way to do a query like "file type = audio/mp3" without either walking the filesystem or querying a separate index, which removes a lot of the benefits.
        PeCaN 10y ago
        I think NTFS can index alternate data streams¹, but yeah, I agree no filesystems are on the level that BeFS was of easy querying and versatility. ¹ I am not sure about this, but https://msdn.microsoft.com/en-us/library/windows/desktop/aa364404(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... seems to suggest it
donatj 10y ago
As I seem to recall the author now works for Apple and worked on their new file system.
- misframer 10y ago
  Yep. According to Adam Leventhal, he was the lead engineer of APFS at Apple [0]. [0] http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1/
userbinator 10y ago
It's funny to see a book on "practical" filesystem design make absolutely no mention of FAT, even in its chapter on other filesystems, despite it being probably the most widely supported read-write filesystem --- or family of filesystems --- in existence.
- kalleboo 10y ago
  Because FAT is a very impractical file system (if you read the section on "Other file systems" it's clear they're picked for being interesting case studies - I imagine FAT is simply not interesting to discuss)
  - userbinator 10y ago
    Because FAT is a very impractical file system That apparently doesn't seem to matter as it is supported almost everywhere... they've picked some of the most featureful ones, but I think there is much to be said for simplicity and ubiquity since that is what really enables data interchange. Then again, the fact that it calls BeFS "practical" reminds me of that old joke about maths textbook titles.
    - deleted 10y ago
      [deleted]
    - chriswarbo 10y ago
      > > Because FAT is a very impractical file system > That apparently doesn't seem to matter as it is supported almost everywhere... they've picked some of the most featureful ones, but I think there is much to be said for simplicity and ubiquity since that is what really enables data interchange. That's true for filesystem choice, but the ubiquity of FAT has nothing to do with "Practical [or impractical] Filesystem Design". You can't sit down and design a filesystem with "installed everywhere" as a feature; things which are everywhere have already been designed, new designs aren't installed anywhere to begin with.
- PeCaN 10y ago
  FAT is not very interesting from a filesystem design perspective (ask someone who doesn't know much about filesystems to try to design one; chances are it will look like FAT¹), so that hardly surprises me. Anyone who buys a book about filesystem design and implementation will already by quite familiar with FAT. ¹ Which is why FAT is so ubiquitous—it's dead simple.
joshka 10y ago
Looks like this has been posted a few times over the years. I wonder how much is new. Previous discussions: https://news.ycombinator.com/from?site=nobius.org https://news.ycombinator.com/from?site=nobius.org
- edu 10y ago
  I read the book 10 years ago, so not much. But in any case it's a great book.

101 points by e-sushi 10y ago

jhugg 10y ago

This is a fantastic book, but do we have the rights to it? What is this URL?

adamnemecek 10y ago
It's the author's website, I assume yes.
edu 10y ago
Yes, the author published openly a few years ago, when it got out of print. Actually, the link is to the authors site: http://www.nobius.org/~dbg/index.html http://www.nobius.org/~dbg/index.html

appleflaxen 10y ago

I remember reading about the BeFS, and how advanced it was for its time. Can anyone who does filesystems help me remember why it was so powerful? Is anything out there now comparable? Have we surpassed BeFS? Any residual functionality that contemporary filesystems lack?

jonhohle 10y ago
It had a lot of features for the time (64-bit, journaling, etc.), but one of the most used features in BeOS was that arbitrary metadata could be attached to files, exposed in Tracker or other applications, and could be used to search for files. One of the examples of the time were scripts to take MP3 ID3 tags and store them as metadata. Email was stored as individual files on the file system, and the email client was just a Tracker window with columns displaying email relevant metadata. Spotlight on OS X supports similar metadata storage and querying, but it's built on top of HFS+, not directly into the file system (and surprise, was built by Dominic Giampaolo, who worked on BeFS, author of the linked book). WinFS claimed a lot of similar features shortly after Be was acquired, but to my knowledge, no file system has provided similar features _and_ had spextensive application support to leverage them.
- geocar 10y ago
  I think the clever bit is when a user sees their email in the file browser[1] they can mentally connect with that and start using their file system as the database that it is, building their own dynamic todo list[2] that isn't managed by any application! Almost every filesystem "supports metadata", and while I agree that application support could be better, I think the real key is getting users to use that space themselves. [1]: https://birdhouse.org/beos/refugee/bemail.jpg https://birdhouse.org/beos/refugee/bemail.jpg [2]: https://birdhouse.org/beos/refugee/trackerbase.gif https://birdhouse.org/beos/refugee/trackerbase.gif
  - PeCaN 10y ago
    BeFS's metadata support was much different from usual—it was a key/value map attached to each file. NTFS supports extended attributes and alternate data streams, which you could use to do the same thing, but none of the *nix filesystems have anything like that. What sort of metadata do you mean with “almost every filesystem ‘supports metadata’”?
    - acdha 10y ago
      JFS, XFS, ReiserFS, ext3/4, and btrfs all support extended attributes – that's become a bigger deal for things like ACLs – but along with varying size limits the share the same limitations as NTFS, HFS+, etc. of not indexing those values. That means there's no way to do a query like "file type = audio/mp3" without either walking the filesystem or querying a separate index, which removes a lot of the benefits.
      - PeCaN 10y ago
        I think NTFS can index alternate data streams¹, but yeah, I agree no filesystems are on the level that BeFS was of easy querying and versatility. ¹ I am not sure about this, but https://msdn.microsoft.com/en-us/library/windows/desktop/aa364404(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... seems to suggest it

donatj 10y ago

As I seem to recall the author now works for Apple and worked on their new file system.

misframer 10y ago
Yep. According to Adam Leventhal, he was the lead engineer of APFS at Apple [0]. [0] http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1/

userbinator 10y ago

It's funny to see a book on "practical" filesystem design make absolutely no mention of FAT, even in its chapter on other filesystems, despite it being probably the most widely supported read-write filesystem --- or family of filesystems --- in existence.

kalleboo 10y ago
Because FAT is a very impractical file system (if you read the section on "Other file systems" it's clear they're picked for being interesting case studies - I imagine FAT is simply not interesting to discuss)
- userbinator 10y ago
  Because FAT is a very impractical file system That apparently doesn't seem to matter as it is supported almost everywhere... they've picked some of the most featureful ones, but I think there is much to be said for simplicity and ubiquity since that is what really enables data interchange. Then again, the fact that it calls BeFS "practical" reminds me of that old joke about maths textbook titles.
  - deleted 10y ago
    [deleted]
  - chriswarbo 10y ago
    > > Because FAT is a very impractical file system > That apparently doesn't seem to matter as it is supported almost everywhere... they've picked some of the most featureful ones, but I think there is much to be said for simplicity and ubiquity since that is what really enables data interchange. That's true for filesystem choice, but the ubiquity of FAT has nothing to do with "Practical [or impractical] Filesystem Design". You can't sit down and design a filesystem with "installed everywhere" as a feature; things which are everywhere have already been designed, new designs aren't installed anywhere to begin with.
PeCaN 10y ago
FAT is not very interesting from a filesystem design perspective (ask someone who doesn't know much about filesystems to try to design one; chances are it will look like FAT¹), so that hardly surprises me. Anyone who buys a book about filesystem design and implementation will already by quite familiar with FAT. ¹ Which is why FAT is so ubiquitous—it's dead simple.

joshka 10y ago

Looks like this has been posted a few times over the years. I wonder how much is new. Previous discussions: https://news.ycombinator.com/from?site=nobius.org https://news.ycombinator.com/from?site=nobius.org

edu 10y ago
I read the book 10 years ago, so not much. But in any case it's a great book.