3 ms·

Huge kudos to ZFS and btrfs, but I am very disappointed with one detail of next generations file systems: Why, oh why, do we still not have a file-type metadata

Huge kudos to ZFS and btrfs, but I am very disappointed with one detail of next generations file systems: Why, oh why, do we still not have a file-type metadata field? We are still using silly file name extensions and magic mime-type detection. In the Age of Types (OOP and FP strong type systems) it only makes sense for the file system to do the same.

chrismonsanto 13y ago
I personally think the biggest problem with current file systems is lack of ACID semantics. It is a tragedy that you can't rollback a set of file system changes if a shell script fails halfway. Why should I have to bring out SQLite if I want transactions, but don't need relations? (Also I agree--yes MIME types in filesystem please)
- stouset 13y ago
  Even if such a thing were to exist, it would probably by necessity be built on some lower-level system that doesn't provide such guarantees… like a filesystem.
  - wmf 13y ago
    IIRC in the old Tandem OS transactions were a basic service, even lower level than the file system. So their file system could use the transaction service to perform arbitrary transactions without much complexity. These days all the world's a VAX though.
- omh 13y ago
  Depending on exactly what you're doing, ZFS (and probably btrfs?) provide at least the Consistency & Durability of ACID, and maybe Isolation as well. But yes, it seems like it would be relatively straightforward to build Atomicity guarantees on top of a COW-based filesystem. I guess it's unlikely that Oracle would add that to ZFS now though - they don't tend to like discouraging the use of relational databases!
  - thwarted 13y ago
    I remember reading some blog post that talked about implementing an InnoDB storage engine using a zfs backend, and using zfs snapshot operations. Seemed to be more of an idea than an actual implementation though.
- blibble 13y ago
  NTFS supports it: http://en.wikipedia.org/wiki/Transactional_NTFS http://en.wikipedia.org/wiki/Transactional_NTFS
- binarycrusader 13y ago
  While ZFS doesn't provide ACID semantics directly, you can achieve similar functionality using zfs snapshots. Take a snapshot before you make changes in a script, then run the script, if the script fails, just rollback!
pauldino 13y ago
I think pretty much all modern file systems (including ntrfs and ZFS) have the facility to support this but it's more a matter of applications and operating systems taking advantage of it. On Linux for example you can set the user.mime_type xattr and it's possible to have Apache use that for it's mime-type.
thwarted 13y ago
Not sure why the filesystems are getting blamed for this, many filesystems on Linux support xattrs for storing related data about the file. But the applications and the tools need to support storing and reading and doing something with that (meta)data.
simias 13y ago
This could easily be implemented on top of any existing filesystem, why would it benefit to be implemented at the fs level? Versioning makes sense for performance but adding metadata sounds like a huge can of worms with no obvious benefits for a low level implementation IMHO. Besides your metadata wouldn't carry well to other filesystems (most USB drives still use FAT...) so it would be a bit of a headache to get right. Look at the mess that are file permissions and ACL already.
dllthomas 13y ago
"Why, oh why, do we still not have a file-type metadata field?" Because you haven't added it. Extended attributes can store arbitrary data. Propose something, experiment, see how it works, and maybe you'll come to something we can standardize on.
ars 13y ago
> oh why, do we still not have a file-type metadata field? Because it's a bad idea. Apple tried it and it didn't work. See: http://en.wikipedia.org/wiki/Creator_code http://en.wikipedia.org/wiki/Creator_code and http://en.wikipedia.org/wiki/Type_code http://en.wikipedia.org/wiki/Type_code

chrismonsanto 13y ago

I personally think the biggest problem with current file systems is lack of ACID semantics. It is a tragedy that you can't rollback a set of file system changes if a shell script fails halfway. Why should I have to bring out SQLite if I want transactions, but don't need relations? (Also I agree--yes MIME types in filesystem please)

stouset 13y ago
Even if such a thing were to exist, it would probably by necessity be built on some lower-level system that doesn't provide such guarantees… like a filesystem.
- wmf 13y ago
  IIRC in the old Tandem OS transactions were a basic service, even lower level than the file system. So their file system could use the transaction service to perform arbitrary transactions without much complexity. These days all the world's a VAX though.
omh 13y ago
Depending on exactly what you're doing, ZFS (and probably btrfs?) provide at least the Consistency & Durability of ACID, and maybe Isolation as well. But yes, it seems like it would be relatively straightforward to build Atomicity guarantees on top of a COW-based filesystem. I guess it's unlikely that Oracle would add that to ZFS now though - they don't tend to like discouraging the use of relational databases!
- thwarted 13y ago
  I remember reading some blog post that talked about implementing an InnoDB storage engine using a zfs backend, and using zfs snapshot operations. Seemed to be more of an idea than an actual implementation though.
blibble 13y ago
NTFS supports it: http://en.wikipedia.org/wiki/Transactional_NTFS http://en.wikipedia.org/wiki/Transactional_NTFS
binarycrusader 13y ago
While ZFS doesn't provide ACID semantics directly, you can achieve similar functionality using zfs snapshots. Take a snapshot before you make changes in a script, then run the script, if the script fails, just rollback!

pauldino 13y ago

I think pretty much all modern file systems (including ntrfs and ZFS) have the facility to support this but it's more a matter of applications and operating systems taking advantage of it. On Linux for example you can set the user.mime_type xattr and it's possible to have Apache use that for it's mime-type.

thwarted 13y ago

Not sure why the filesystems are getting blamed for this, many filesystems on Linux support xattrs for storing related data about the file. But the applications and the tools need to support storing and reading and doing something with that (meta)data.

simias 13y ago

This could easily be implemented on top of any existing filesystem, why would it benefit to be implemented at the fs level? Versioning makes sense for performance but adding metadata sounds like a huge can of worms with no obvious benefits for a low level implementation IMHO. Besides your metadata wouldn't carry well to other filesystems (most USB drives still use FAT...) so it would be a bit of a headache to get right. Look at the mess that are file permissions and ACL already.

dllthomas 13y ago

"Why, oh why, do we still not have a file-type metadata field?" Because you haven't added it. Extended attributes can store arbitrary data. Propose something, experiment, see how it works, and maybe you'll come to something we can standardize on.

ars 13y ago

> oh why, do we still not have a file-type metadata field? Because it's a bad idea. Apple tried it and it didn't work. See: http://en.wikipedia.org/wiki/Creator_code http://en.wikipedia.org/wiki/Creator_code and http://en.wikipedia.org/wiki/Type_code http://en.wikipedia.org/wiki/Type_code