4 ms·

I wish zfs had a 'tag' type of feature like Apple style fs. I have thought about making an object store to get some of the metadata on the fs level, rather tha

by chaxor 3y ago

I wish zfs had a 'tag' type of feature like Apple style fs. I have thought about making an object store to get some of the metadata on the fs level, rather than having to make some cobbles together solution of an SQLite DB of metadata with the files or something, but many of the object store solutions seem to be for fairly large files, rather than a few thousand pictures of family and such. Unsure what a good solution using zfs would be.

adrian_b 3y ago
I am not sure what kind of "tag" you mean, but in any modern file system which supports extended file attributes you can attach arbitrary metadata to any file. You just have to define whatever kind of tags you want and choose names for the xattrs in which to store them and then you should use them in your scripts or applications. I am using all the time xattrs for things like file checksums and content classification, on XFS and on FreeBSD UFS, but almost all modern file systems support xattrs, except Linux tmpfs (which has only a partial support that is useless for those who use custom xattrs, so a file with xattrs cannot be copied to /tmp if tmpfs is used).
- vetinari 3y ago
  > except Linux tmpfs (which has only a partial support that is useless for those who use custom xattrs, so a file with xattrs cannot be copied to /tmp if tmpfs is used). tmpfs is getting user xattrs in kernel 6.6, together with quotas. > You just have to define whatever kind of tags you want and choose names for the xattrs in which to store them and then you should use them in your scripts or applications. It is still just a oob store, you have to traverse the directory hierarchy old-style; the old BeOS did (and Haiku today does) something more: it indexed the selected xattrs and allowed live queries against those indices. This allowed for very interesting uses.
  - adrian_b 3y ago
    > tmpfs is getting user xattrs in kernel 6.6 I am aware of this, but the so-called user xattrs support most likely remains too bad to be of any use. The kernel list message: "Add support for user xattrs. While tmpfs already supports security xattrs (security.*) and POSIX ACLs for a long time it lacked support for user xattrs (user.*). With this pull request tmpfs will be able to support a limited number of user xattrs." I am waiting to see which is the meaning of "a limited number", but based on the history of tmpfs, I suspect that this does not mean that the number of extended file attributes is limited to some value, but it means that there is a small list of names of user xattrs that will preserved (which are those used by some application that the tmpfs devs love, while they are hating the other applications), while the other xattrs will be lost, like now. If "a limited number" is really a number, that would be much better, but this might still mean that the number is excessively low, e.g. ten attributes or less, instead of being high enough to not be a serious limitation, e.g. one hundred or one thousand. > It is still just a oob store, you have to traverse the directory hierarchy old-style; the old BeOS did (and Haiku today does) something more: it indexed the selected xattrs and allowed live queries against those indices. You are right and it would be nice if the file system itself would allow faster queries by xattrs than what can be achieved by testing the attributes of files selected by their names. However, it is not difficult to create a database file indexing the files by attributes, but it may need frequent updates, to remain in sync with the indexed file system. I use such databases, but mainly for file system parts where I keep files for long-term storage, like books, research papers, handbooks, product documentation or older software projects.
    - DHowett 3y ago
      Looking over the landed patch (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ecd7db20474c3859d4d01f34aaabf41bd28c7d84 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...), I cannot find a hardcoded list of user.* xattrs that will be accepted. It looks like it will store as many xattrs as there is space for, based on their contents.
      - adrian_b 3y ago
        That would be nice, but then the "a limited number" expression does not make much sense. All the file systems that support extended file attributes have some limit for the size of the attributes attached to a file, for instance of 64 kB. After searching through the patch, it seems that the maximum number of xattrs is 128. There is also a size limit of 128 kB, which might be the limit for the sum of the sizes of all attributes. If that is the only reason for saying "a limited number", then this is very good, the limits are high enough to not cause any problems in most applications, so tmpfs will finally no longer strip the attributes of the files copied through it.

by chaxor 3y ago

adrian_b 3y ago

I am not sure what kind of "tag" you mean, but in any modern file system which supports extended file attributes you can attach arbitrary metadata to any file. You just have to define whatever kind of tags you want and choose names for the xattrs in which to store them and then you should use them in your scripts or applications. I am using all the time xattrs for things like file checksums and content classification, on XFS and on FreeBSD UFS, but almost all modern file systems support xattrs, except Linux tmpfs (which has only a partial support that is useless for those who use custom xattrs, so a file with xattrs cannot be copied to /tmp if tmpfs is used).

vetinari 3y ago
> except Linux tmpfs (which has only a partial support that is useless for those who use custom xattrs, so a file with xattrs cannot be copied to /tmp if tmpfs is used). tmpfs is getting user xattrs in kernel 6.6, together with quotas. > You just have to define whatever kind of tags you want and choose names for the xattrs in which to store them and then you should use them in your scripts or applications. It is still just a oob store, you have to traverse the directory hierarchy old-style; the old BeOS did (and Haiku today does) something more: it indexed the selected xattrs and allowed live queries against those indices. This allowed for very interesting uses.
- adrian_b 3y ago
  > tmpfs is getting user xattrs in kernel 6.6 I am aware of this, but the so-called user xattrs support most likely remains too bad to be of any use. The kernel list message: "Add support for user xattrs. While tmpfs already supports security xattrs (security.*) and POSIX ACLs for a long time it lacked support for user xattrs (user.*). With this pull request tmpfs will be able to support a limited number of user xattrs." I am waiting to see which is the meaning of "a limited number", but based on the history of tmpfs, I suspect that this does not mean that the number of extended file attributes is limited to some value, but it means that there is a small list of names of user xattrs that will preserved (which are those used by some application that the tmpfs devs love, while they are hating the other applications), while the other xattrs will be lost, like now. If "a limited number" is really a number, that would be much better, but this might still mean that the number is excessively low, e.g. ten attributes or less, instead of being high enough to not be a serious limitation, e.g. one hundred or one thousand. > It is still just a oob store, you have to traverse the directory hierarchy old-style; the old BeOS did (and Haiku today does) something more: it indexed the selected xattrs and allowed live queries against those indices. You are right and it would be nice if the file system itself would allow faster queries by xattrs than what can be achieved by testing the attributes of files selected by their names. However, it is not difficult to create a database file indexing the files by attributes, but it may need frequent updates, to remain in sync with the indexed file system. I use such databases, but mainly for file system parts where I keep files for long-term storage, like books, research papers, handbooks, product documentation or older software projects.
  - DHowett 3y ago
    Looking over the landed patch (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ecd7db20474c3859d4d01f34aaabf41bd28c7d84 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...), I cannot find a hardcoded list of user.* xattrs that will be accepted. It looks like it will store as many xattrs as there is space for, based on their contents.
    - adrian_b 3y ago
      That would be nice, but then the "a limited number" expression does not make much sense. All the file systems that support extended file attributes have some limit for the size of the attributes attached to a file, for instance of 64 kB. After searching through the patch, it seems that the maximum number of xattrs is 128. There is also a size limit of 128 kB, which might be the limit for the sum of the sizes of all attributes. If that is the only reason for saying "a limited number", then this is very good, the limits are high enough to not cause any problems in most applications, so tmpfs will finally no longer strip the attributes of the files copied through it.