5 ms·

Can anyone tell me if anything like this exists : A program that runs in the background on your machine and watches processes and the file handles they open. T

Can anyone tell me if anything like this exists : A program that runs in the background on your machine and watches processes and the file handles they open. Then, keeps a database of associations between files and programs. E.g. you open a pdf with adobe acrobat, and now it adds an entry with a relation between that specific file and acrobat, and between pdf file extensions and acrobat. Then, this information is used to provide an intuitive interface into the file system. Searching for that .docx file you made last year and cant remember for the life of you where you put it? Just search for .docx, or files associated with "Word" This to me is the holy grail of filesystem search, and it doesn't seem that hard to implement, but I've never found anything like it.

epilys 4y ago
I haven't come across of any project like that, but in case anyone wants to implement this and doesn't know where to start, here's a way to do it on a freedesktop-compatible linux: Make a userspace daemon process that adds eBPF tracepoints[0] to open{,_at} etc syscalls which match files of your user directories with specific extensions (e.g. .docx). Associate PIDs that open those files with their .desktop entries[1] Store results in some database like sqlite3.[2] Search this database with your favorite interface, like a CLI script or a GNOME shell search provider[3]. I have seen this Rust project on HN which does something similar but with file attribute syscalls, you can use it as reference: https://github.com/javierhonduco/sweeper https://github.com/javierhonduco/sweeper [0]: https://github.com/iovisor/bpftrace https://github.com/iovisor/bpftrace [1]: https://www.freedesktop.org/wiki/Specifications/desktop-entry-spec/ https://www.freedesktop.org/wiki/Specifications/desktop-entr... [2]: https://sqlite.org/ https://sqlite.org/ [3]: https://developer.gnome.org/documentation/tutorials/search-provider.html https://developer.gnome.org/documentation/tutorials/search-p...
- DontchaKnowit 4y ago
  This is wonderful, thank you. I may try this out. Originally wanted to do it for windows but I eventually concluded it would take some prodigious hackery to get it do work
  - epilys 4y ago
    I wish I could help you but I haven't touched windows in decades. Perhaps someone else can chime in.
    - DontchaKnowit 4y ago
      So as far as I can tell the only way a user can access this type of info on windows is via procmon. Problem is its just a gui application, no programatic access afaik. I did find a project where someone reverse engineered it to some extent and it allowed you to make calls to it programatically. Anyway, it turned out there was a loooot of heavy lifting that'd be required to get it to behave the way I needed it. Anywho, I'm happy for the info on linux. I was looking for a project that would require me to up my bash skills anyway. This sounds perfect
      - Joker_vD 4y ago
        I'm pretty sure the latter (Win8 and on?) versions of Windows had some sort of framework for writing FS filter drivers or something, that the AVs are supposed to use?
        p_l 4y ago
        Every NT version, in fact. Driver filters including filesystem filters are core NT feature. So you'd have to write a driver that hooked itself at appropriate step in IO Request processing, then it could inspect and gather data just like eBPF. Procmon uses diagnostic interfaces from outside kernel, might require debug permissions.
giantrobot 4y ago
Spotlight on the Mac has done this for like 15 years. You can search for files by content and metadata attributes[0]. You can even create "smart folders" that save the search parameters and show an up to date instance of the search when opened. The Spotlight search is also exposed on the command line with the mdfind command. Spotlight uses the LaunchServices database of file type associations to give friendly names to file types but has a hierarchy of type identifiers[1]. [0] https://support.apple.com/guide/mac-help/narrow-search-results-mh15155/mac https://support.apple.com/guide/mac-help/narrow-search-resul... [1] https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/understanding_utis/understand_utis_conc/understand_utis_conc.html https://developer.apple.com/library/archive/documentation/Fi...
MisterTea 4y ago
Think about this solution for a second. You want a process that spies on other processes to build a database of what files they touch so you can remember where you put that file last year. This is a problem that can be solved simply by better organizing your files. No security nightmare Rube Goldberg contraption needed. Ive been in that boat before and it just boils down to organizing your stuff better.
- Joker_vD 4y ago
  > You want a process that spies on other processes to build a database of what files they touch It's basically an extension to the VFS that would maintain atime and some additional metadata to it, what's so horrible about it?
- DontchaKnowit 4y ago
  I mean I would write the damn thing myself, Im not worried about security. I just think windows tools for navigating the file system are straight garbage and think it could be done a million times better by just indexing interactions between processes and files. But who knows maybe im barkin up the wrong tree. Also- organize your files better : first off easier said than done and second off, my shit is already a total mess and I am not willing to take off work for 3 weeks to organize everythibng. I just want a file viewer UI that is actually useful in allowing my to find files. Imo the folder hierarchy paradigm shouldnt have ever been exposed to users as the primary mechanism for accessing files. It makes no sense for 99 percent of use cases for the average user.
mtVessel 4y ago
I'm not following your use cases. Why do you need an association database to look for files named .docx or .pdf? I could see it possibly being useful for apps like VLC which open a number of different file types, but then if I used multiple apps to open videos I'd have to remember all the apps I used in order to do that search.
- DontchaKnowit 4y ago
  Basically windows search is slpw as all fuck if you search by file extension. Itd be a lot faster if you just had a DB of associations. But also in the case that I do not know the extension of a file, or want to find all files involed in configuring a program, etc, it would allow you to find that info quickly.
eviks 4y ago
(Windows) File associations database is part of the OS, no? How is it different what you need? Search is best done via Everything by Voidtools, which is instant, though it doesn't connect to this database, but instead uses manually entered lists of extensions But you could write integration that extracts the list from the OS database and dumps it in Everything's lists Or you could integrate Everything in you file manager and then add the OS file associations integration there

epilys 4y ago

I haven't come across of any project like that, but in case anyone wants to implement this and doesn't know where to start, here's a way to do it on a freedesktop-compatible linux: Make a userspace daemon process that adds eBPF tracepoints[0] to open{,_at} etc syscalls which match files of your user directories with specific extensions (e.g. .docx). Associate PIDs that open those files with their .desktop entries[1] Store results in some database like sqlite3.[2] Search this database with your favorite interface, like a CLI script or a GNOME shell search provider[3]. I have seen this Rust project on HN which does something similar but with file attribute syscalls, you can use it as reference: https://github.com/javierhonduco/sweeper https://github.com/javierhonduco/sweeper [0]: https://github.com/iovisor/bpftrace https://github.com/iovisor/bpftrace [1]: https://www.freedesktop.org/wiki/Specifications/desktop-entry-spec/ https://www.freedesktop.org/wiki/Specifications/desktop-entr... [2]: https://sqlite.org/ https://sqlite.org/ [3]: https://developer.gnome.org/documentation/tutorials/search-provider.html https://developer.gnome.org/documentation/tutorials/search-p...

DontchaKnowit 4y ago
This is wonderful, thank you. I may try this out. Originally wanted to do it for windows but I eventually concluded it would take some prodigious hackery to get it do work
- epilys 4y ago
  I wish I could help you but I haven't touched windows in decades. Perhaps someone else can chime in.
  - DontchaKnowit 4y ago
    So as far as I can tell the only way a user can access this type of info on windows is via procmon. Problem is its just a gui application, no programatic access afaik. I did find a project where someone reverse engineered it to some extent and it allowed you to make calls to it programatically. Anyway, it turned out there was a loooot of heavy lifting that'd be required to get it to behave the way I needed it. Anywho, I'm happy for the info on linux. I was looking for a project that would require me to up my bash skills anyway. This sounds perfect
    - Joker_vD 4y ago
      I'm pretty sure the latter (Win8 and on?) versions of Windows had some sort of framework for writing FS filter drivers or something, that the AVs are supposed to use?
      - p_l 4y ago
        Every NT version, in fact. Driver filters including filesystem filters are core NT feature. So you'd have to write a driver that hooked itself at appropriate step in IO Request processing, then it could inspect and gather data just like eBPF. Procmon uses diagnostic interfaces from outside kernel, might require debug permissions.

giantrobot 4y ago

Spotlight on the Mac has done this for like 15 years. You can search for files by content and metadata attributes[0]. You can even create "smart folders" that save the search parameters and show an up to date instance of the search when opened. The Spotlight search is also exposed on the command line with the mdfind command. Spotlight uses the LaunchServices database of file type associations to give friendly names to file types but has a hierarchy of type identifiers[1]. [0] https://support.apple.com/guide/mac-help/narrow-search-results-mh15155/mac https://support.apple.com/guide/mac-help/narrow-search-resul... [1] https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/understanding_utis/understand_utis_conc/understand_utis_conc.html https://developer.apple.com/library/archive/documentation/Fi...

MisterTea 4y ago

Think about this solution for a second. You want a process that spies on other processes to build a database of what files they touch so you can remember where you put that file last year. This is a problem that can be solved simply by better organizing your files. No security nightmare Rube Goldberg contraption needed. Ive been in that boat before and it just boils down to organizing your stuff better.

Joker_vD 4y ago
> You want a process that spies on other processes to build a database of what files they touch It's basically an extension to the VFS that would maintain atime and some additional metadata to it, what's so horrible about it?
DontchaKnowit 4y ago
I mean I would write the damn thing myself, Im not worried about security. I just think windows tools for navigating the file system are straight garbage and think it could be done a million times better by just indexing interactions between processes and files. But who knows maybe im barkin up the wrong tree. Also- organize your files better : first off easier said than done and second off, my shit is already a total mess and I am not willing to take off work for 3 weeks to organize everythibng. I just want a file viewer UI that is actually useful in allowing my to find files. Imo the folder hierarchy paradigm shouldnt have ever been exposed to users as the primary mechanism for accessing files. It makes no sense for 99 percent of use cases for the average user.

mtVessel 4y ago

I'm not following your use cases. Why do you need an association database to look for files named .docx or .pdf? I could see it possibly being useful for apps like VLC which open a number of different file types, but then if I used multiple apps to open videos I'd have to remember all the apps I used in order to do that search.

DontchaKnowit 4y ago
Basically windows search is slpw as all fuck if you search by file extension. Itd be a lot faster if you just had a DB of associations. But also in the case that I do not know the extension of a file, or want to find all files involed in configuring a program, etc, it would allow you to find that info quickly.

eviks 4y ago

(Windows) File associations database is part of the OS, no? How is it different what you need? Search is best done via Everything by Voidtools, which is instant, though it doesn't connect to this database, but instead uses manually entered lists of extensions But you could write integration that extracts the list from the OS database and dumps it in Everything's lists Or you could integrate Everything in you file manager and then add the OS file associations integration there