7 ms·
A general tagging mechanism, reasonably supported across filesystems, would be great to have. Not only for media collections. It's not easy to implement nicely
by nine_k 8y ago
A general tagging mechanism, reasonably supported across filesystems, would be great to have. Not only for media collections.
It's not easy to implement nicely, though.
- netsec_burn 8y agoWhy wouldn't it to be easy to implement? You could probably make a global LD_PRELOAD library that hooks into all open() CREAT calls and tags the inode's xattr with the program that created the file. You could use an attribute like, user.created_by.
- nine_k 8y agoAdding an interface to access the tag store won't be very hard. It would be harder to make it so that all the tag-oblivious programs, like mv or vi, to say nothing of rsync, would preserve the attributes when moving or modifying a file.
- netsec_burn 8y agoSounds like you might need to hook rename() if you'd like to add that feature too, but I would still say it's straightforward.
- nine_k 8y agoThe way most editors save a file is like this: f = open("the_file.new") write(f, new_contents); close(f); rename("the_file", "the_file~"); rename("the_file.new", "the_file"); So the old file is never modified, it's renamed to the backup copy, and an entirely new file is created to take its place. This has a number of advantages, but does not play well with any extended info the old file used to have, unless they are copied explicitly. And in a tagging-oblivious program, they won't be.
- netsec_burn 8y agoThanks for teaching me this! I wasn't aware. So, the original xattr (and the program that created it) would be deleted and replaced by vi for example. Still technically right, but I'm unsure how you'd keep a history.
- 6c696e7578 8y agoI see a lot of saves work like this: f = open( "the_file.new.$$" ); write( f, new_contents ); close( f ); rename( "the_file.new.$$", "the_file.new" ) If not implemented like this, then something that attempts to read "the_file.new" may get partial contents, normally truncated along the file system block size. Backups often take place, too, like you mentioned, depending on editor.
- notyourday 8y agoIf you do not rename files while holding their open file descriptors there's no guarantee that you are renaming the file you just wrote to.
- JdeBP 8y agoThere is no guarantee of that if one does retain the open file descriptor.
- notyourday 8y agobefore rename: fstat the fd, get st_dev and st_ino. rename stat the new name. Compare st_dev and st_ino. If the value matches, you renamed the right file. If it does not match, you renamed a wrong file. Without holding the fd, it is impossible to know if it is the right file.
- JdeBP 8y ago"you renamed a wrong file" shows that "there's no guarantee that you are renaming the file you just wrote to".
- sneak 8y agoSidecar files might work, and would also be filesystem-portable.
- netsec_burn 8y agoThis is what Mac does to track download locations, if I remember correctly (._file).
- amaccuish 8y agoPlease no. This article is literally "Dotfile madness", don't make it worse! I have to deal already with these on file shares, specifically for Apple: .DS_Store, .Trashes and .AppleDouble, or for Windows: Thumbs.db, $RECYCLE.BIN (for some reason Windows sometimes ignores the fact I've disabled the recycle bin on a share and creates this instead) and desktop.ini. Please don't drop crap around directories where there exist a multitude of tidier alternatives.
- sneak 8y agoHow do you have to “deal with” them? What are the portable tidier alternatives?
- featherrust 8y agoA daemon with a CLI and a programmatic interface, backed by a SQLite store, hooked into Linux audit and perhaps an LD_PRELOAD?
- amaccuish 8y agoBecause when you have multiple OSs accessing the same shared folder, they all create their own crap, which is then visible to the other OSs, and fills up directories with stuff that confuses normal users.
- sneak 8y agoNormal users don’t run multiple OSes.
- cma 8y agoPrograms can use programs like 'touch' to create the files so you'd have to get meta into a chain of process ownership or something while exempting the user's shell.
- netsec_burn 8y agoYou're right. I can see anything like scripts getting difficult. It wouldn't work for all use cases.
- ksherlock 8y agoHave you seen actual programs (not shell scripts) that do that? I've heard of code smells, but that's a programmer smell with there.
- netsec_burn 8y agoI think touch is a generic example, but not a practical one. The idea is that program A can launch program B, and you'd have to recursively search for the parent. But it's not easy to find which parent is the true owner, maybe an exception can be made for init but shells can't have that exception as they are sometimes the owner.
- larkeith 8y agoCouldn't you just record the creating process and all ancestors?
- megous 8y agoWhy such a hack? This is a solved problem already. Try Linux audit API. You can set it up for tracking various actions, and you'll get a log with all the information including a process name. https://linux-audit.com/configuring-and-auditing-linux-systems-with-audit-daemon/ https://linux-audit.com/configuring-and-auditing-linux-syste...
- netsec_burn 8y agoI believe the question was asking for a method of storing the metadata, I was just suggesting xattrs. This will work at the expense of running a daemon.
- bennofs 8y agoCan audit be configured to only log file creation, not modification or access? Last time I didn't find a way to do that and I don't want to write a log file entry for every file access.
- fwip 8y agoYou can at least ignore read events, not sure about separating write from creation.
- ecnahc515 8y agoMacOS does something sorta like this. It uses extended attributes to mark where a file came from in some cases, such as when an executable is downloaded from a browser, it will mark it as "unsafe" and when you attempt to run it, this causes it to fail to run and you have to whitelist it via security preferences.
- lgeorget 8y agoThere's an entire research field dedicated to the information flow control which could solve just that (if it were actually used). In my research team, we used a tainting tracing mechanism to understand the behavior of malware. Basically, we installed a malware on a clean phone and we then traced all information flow originating from the APK to processes, to files, to sockets, etc. It helped reverse-engineering the malware.
- 0db532a0 8y agoAny interesting links to papers?
- lgeorget 8y agoThere's the work by Myers and Liskov which is quite central to the whole field: https://dl.acm.org/citation.cfm?id=266669 https://dl.acm.org/citation.cfm?id=266669, there's Flume: https://dl.acm.org/citation.cfm?id=1294293 https://dl.acm.org/citation.cfm?id=1294293, DStar https://www.usenix.org/event/nsdi08/tech/full_papers/zeldovich/zeldovich.pdf.. https://www.usenix.org/event/nsdi08/tech/full_papers/zeldovi.... More recently, on Android: https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_nadkarni.pdf https://www.usenix.org/system/files/conference/usenixsecurit.... And this (disclaimer: I'm the main author): https://link.springer.com/chapter/10.1007/978-3-319-66197-1_1 https://link.springer.com/chapter/10.1007/978-3-319-66197-1_....
- 0db532a0 8y agoThanks.
- TeMPOraL 8y agoI'd love something like this on program level. Given contents of a variable, I'd like to know where they came from - which pieces of code contributed to the result. I'd also love to be able to mark a piece of data, and see what code touched it or derived new data from it. A programming equivalent of injecting radioisotopes into the body.
- darkpuma 8y agoAssociating tags with files is straight forward. What's not quite so straight forward is how you query the system. The worst implementations of file tagging only implement retrieving a list of all files that have a given tag. Slightly better than this are tagging systems that will return the intersection between two or more tags. Most tag systems never go beyond this. Going slightly further, tag exclusions are powerful and are sometimes implemented (given a set of files from some subquery, exclude all files that have a given tag.) What you rarely see are systems that allow you to exclude one subquery from another. However what you almost never see implemented is a system for preferential tags; given a subquery, reorder the results such that files with preferred tags are raised to the top, ordered by how many of the preferred tags they have. Once you implement this, the system's UX changes dramatically because the user no longer has to make strong assumptions about how well their files have been tagged. Many files might be missing relevant tags and the user may not be sure if the file they're after is one of these incompletely tagged files. When using a system with preferential tag querying, the user will receive files in their result list that don't match all the tags listed, but most of them. This is a bigger deal than it may sound, since the main drawback of using tags for file management is incomplete tagging. By addressing this drawback, you stand to bring file tagging to the next conceptual level, which is rendering hierarchical file management obsolete. Consider that hierarchical file management is a strict subset of file tagging. You can model file hierarchies inside a file tagging system. To demonstrate this by example, consider /home/joesixpack/Documents/seventh-novel.pdf For each level of the hierarchy, we can create a new tag, such that this document has the tags: '/home/', '/home/joesixpack/', '/home/joesixpack/Documents/'. But because we're using file tagging, we can also automatically tag that file with things like 'pdf' or maybe even 'Documents'. Now before I go on, there is something to be said about the number of tags in the system exploding when you model a tree as a tagset. In practice this probably isn't the approach any real system should take, if only because most of those tags will be useless and because there is a great deal of redundancy in trees that wouldn't exist in a native filetagging system. Consider /home/joesixpack/Documents/ and /home/johnsmith/Documents/. We have two different Documents directories because they exist in different parts of the hierarchy. However in a file tagging system we'd ideally only have a single Documents tag and one tag per user, such that querying the intersection between 'joesixpack' and 'Documents' returns the files that would otherwise be in /home/joesixpack/Documents. Some tags, such as the user tag, could be implicit; such that if joesixpack simply queries 'Documents' the tag 'joesixpack' is automatically intersected with the results presented to him. With that out of the way, consider how we can go further if we apply straight forward statistics to the tag system. Suppose 99% of joesixpack's 'Documents' that are 'pdf' are also 'novel'. When joesixpack creates a new file that's in the tags 'Documents' and 'pdf', what are the chances that file should also be tagged with 'novel'? We could analyze the contents of the file if the system has a semantic understanding of what 'novel' means, but let's not go there. Tag systems created by the likes of Facebook and Google do this sort of analysis, but it's heavy, tricky, goes wrong in ways that create PR disasters, etc. We can get pretty good results by ignoring the contents of the file and looking instead at merely what that file is already tagged with. If a file is tagged with 'joesixpack', 'Documents', and 'pdf', it _probably_ should be tagged with 'novel' as well. Probably, but not necessarily. So the system can expose to joesixpack the suggestion that he tag that particular file with 'novel'. By presenting suggestions like that to the user, you greatly reduce the UX friction needed to tag files well and therefore increase the usability of the tagging system, while at the same time improving the quality of the tag suggestions in the future. What we create here is a 'virtuous cycle' of sorts that gives back to the user more as it's used more. Such a tag suggester can be implemented as multi-label classification using naive bayes; it's fairly straightforward. Going further; if we have such a tag suggester, and an appropriate caching scheme, tag suggestions can be used by queries. If joesixpack queries for 'pdf','novel', the system could return to him the intersection of 'pdf','novel', but also return to him files that are tagged with 'pdf' but not tagged with 'novel', in cases where the tag suggester indicates there is a high likelihood that the file should be tagged with 'novel'. That last paragraph may or may not play out well, I haven't experimented with it extensively yet. But getting back to my original point: if you're going to implement file tagging in a filesystem, you should think carefully about how the users will query that system, and whether the query system will be extensible in userspace to facilitate new powerful methods of querying it. It would be a total tragedy if the system only supported basic tag intersections and the only way to extend it was to implement a new kernel module.
- Pxtl 8y agoThat's a killer idea. I hope somebody will reise to the challenge of a filesystem with such a feature.
- smallstepforman 8y agoBeFileSystem (Haiku) HFS+ and APFS Even NTFS has attributes. The problem is file manager. Tracker/Finder are several steps ahead of Explorer. Under Linux, its a greater mess even though there are no technical limitations.
- Pxtl 8y agoI was mostly going for a morbid pun, but it's nice to hear that the FS does support this. Surprising Explorer is so bad at it since they're good at getting EXIF data and ID3 tags into the Properties tab in Explorer.
- agumonkey 8y agofor non portable, I found it nice that wget or curl adds the original url to the saved file as xattrs