5 ms·

Most people seem to interpret this as "don't hide things". I interpret this as "file systems need to have a flexible metadata mechanism instead of ad-hoc hacks

Most people seem to interpret this as "don't hide things". I interpret this as "file systems need to have a flexible metadata mechanism instead of ad-hoc hacks". Of course, at this point in time no such thing will happen, because it's easier to invent 100 new databases than change some assumptions about our core architecture.

jl6 4y ago
I guess one could mention xattrs but there isn’t a lot of consensus on how they should be used.
- marcosdumay 4y ago
  Or if they should be used at all. Distros still default into disabling them. What is quite interesting, because it's all a matter of user interface. It's a change that doesn't even impact on software compatibility. Any distro that decides that a file is hidden due to an attr can just push that change, and nothing will break.
deleted 4y ago
[deleted]
klabb3 4y ago
> I interpret this as "file systems need to have a flexible metadata mechanism instead of ad-hoc hacks". Pike isn't even talking about that, only the typical config files in the home dir that clutter and affect performance of file name ops in the home dir. > (For those who object that dot files serve a purpose, I don't dispute that but counter that it's the files that serve the purpose, not the convention for their names. They could just as easily be in $HOME/cfg or $HOME/lib, which is what we did in Plan 9, which had no dot files. Lessons can be learned.) I have to agree. I can't think of any situation where hiding really helps. If the reason is clutter, the solution should be a dir.
- BrainVirus 4y ago
  >I can't think of any situation where hiding really helps. The irony of dot-files is that they became used for metadata, like .git directories, .htcasses files and so on. So, a solution that fakes metadata by abusing names is now used to indicate metadata at another level of the system. There is clearly something missing in file system design.
  - klabb3 4y ago
    I agree, sometimes dot files are used for metadata. But say a typical code repo: lots of dirs (vendored deps, compiled libs, generated code) are in regular dirs but git uses a dot-dir. Is there a strong reason for this I'm not getting? You're not supposed to modify either of them, heck even listing can be a shit-show. I can see the use-case of "when zipping or sending a dir to a different machine, omit the metadata", but this doesn't hold true for code repos, and is sometimes even the opposite of what you want. It'd probably be better with dir name conventions like "cache", "secrets", "config" or similar.
    - BrainVirus 4y ago
      I think what you're pointing out is an example of a real problem. I'm not sure the problem has a name or is noticed by most people. But something is off. In some cases it might be that common conventions (like dotfiles) aren't really well-designed. But I think the more general issue is that the current set of abstractions (file/directory/symlink) are not sufficient to take good care of practical use cases we have for data these days, not matter how cleverly we use them. Should .git directory reside in the same directory as your source code? Maybe not. Fossil has an interesting compromise in that regard. Should compiled artefacts reside in the same directory as your source code? Almost certainly not. This seems like a bad idea from some hacky implementation decades ago. Should tools be able to distinguish between "real" files and metadata files? Almost certainly yes, except there are different types of "metadata" files, as you pointed out. I think naming conventions are simply not good enough to handle this properly. This is where FS metadata could come into play. Generally, I think if someone without prejudice examined the common problems we have with data these days and designed a "file" system from scratch, they would end up with something vastly different from our current files/folders paradigm.
GekkePrutser 4y ago
Inventing new databases doesn't stop the old architecture from working, and everything that depends on it. Changing that architecture will. This is simply why.
akira2501 4y ago
> "file systems need to have a flexible metadata mechanism instead of ad-hoc hacks" xattr's require a system call for each path you want to look up, and then another system call for each value you actually want to retrieve. Plus all the error handling that goes along with those calls. Oh, and they're all strings, so even more work if you want to have an integer attribute. Also, which quota should the space used by the attributes accrue to? Should we split that into "user" and "system" attributes? Whoops, looks like we need an "ad-hoc" namespace here too... this time with actual security implications! The mechanism exists, but it's more cumbersome than it's worth. That's certainly the way I've felt about it every time I have to muck around with a system that "mistakenly" decided to use them instead of just a database. In that light, the dot is not a historical mistake. It's an obvious optimization of a particularly popular use case.

jl6 4y ago

I guess one could mention xattrs but there isn’t a lot of consensus on how they should be used.

marcosdumay 4y ago
Or if they should be used at all. Distros still default into disabling them. What is quite interesting, because it's all a matter of user interface. It's a change that doesn't even impact on software compatibility. Any distro that decides that a file is hidden due to an attr can just push that change, and nothing will break.

deleted 4y ago

[deleted]

klabb3 4y ago

> I interpret this as "file systems need to have a flexible metadata mechanism instead of ad-hoc hacks". Pike isn't even talking about that, only the typical config files in the home dir that clutter and affect performance of file name ops in the home dir. > (For those who object that dot files serve a purpose, I don't dispute that but counter that it's the files that serve the purpose, not the convention for their names. They could just as easily be in $HOME/cfg or $HOME/lib, which is what we did in Plan 9, which had no dot files. Lessons can be learned.) I have to agree. I can't think of any situation where hiding really helps. If the reason is clutter, the solution should be a dir.

BrainVirus 4y ago
>I can't think of any situation where hiding really helps. The irony of dot-files is that they became used for metadata, like .git directories, .htcasses files and so on. So, a solution that fakes metadata by abusing names is now used to indicate metadata at another level of the system. There is clearly something missing in file system design.
- klabb3 4y ago
  I agree, sometimes dot files are used for metadata. But say a typical code repo: lots of dirs (vendored deps, compiled libs, generated code) are in regular dirs but git uses a dot-dir. Is there a strong reason for this I'm not getting? You're not supposed to modify either of them, heck even listing can be a shit-show. I can see the use-case of "when zipping or sending a dir to a different machine, omit the metadata", but this doesn't hold true for code repos, and is sometimes even the opposite of what you want. It'd probably be better with dir name conventions like "cache", "secrets", "config" or similar.
  - BrainVirus 4y ago
    I think what you're pointing out is an example of a real problem. I'm not sure the problem has a name or is noticed by most people. But something is off. In some cases it might be that common conventions (like dotfiles) aren't really well-designed. But I think the more general issue is that the current set of abstractions (file/directory/symlink) are not sufficient to take good care of practical use cases we have for data these days, not matter how cleverly we use them. Should .git directory reside in the same directory as your source code? Maybe not. Fossil has an interesting compromise in that regard. Should compiled artefacts reside in the same directory as your source code? Almost certainly not. This seems like a bad idea from some hacky implementation decades ago. Should tools be able to distinguish between "real" files and metadata files? Almost certainly yes, except there are different types of "metadata" files, as you pointed out. I think naming conventions are simply not good enough to handle this properly. This is where FS metadata could come into play. Generally, I think if someone without prejudice examined the common problems we have with data these days and designed a "file" system from scratch, they would end up with something vastly different from our current files/folders paradigm.

GekkePrutser 4y ago

Inventing new databases doesn't stop the old architecture from working, and everything that depends on it. Changing that architecture will. This is simply why.

akira2501 4y ago

> "file systems need to have a flexible metadata mechanism instead of ad-hoc hacks" xattr's require a system call for each path you want to look up, and then another system call for each value you actually want to retrieve. Plus all the error handling that goes along with those calls. Oh, and they're all strings, so even more work if you want to have an integer attribute. Also, which quota should the space used by the attributes accrue to? Should we split that into "user" and "system" attributes? Whoops, looks like we need an "ad-hoc" namespace here too... this time with actual security implications! The mechanism exists, but it's more cumbersome than it's worth. That's certainly the way I've felt about it every time I have to muck around with a system that "mistakenly" decided to use them instead of just a database. In that light, the dot is not a historical mistake. It's an obvious optimization of a particularly popular use case.