5 ms·

I just want my entire file system stored in Sqlite so I can query it myself

by reactjavascript 4y ago

samatman 4y ago
Oh hey look what I just found https://github.com/narumatt/sqlitefs https://github.com/narumatt/sqlitefs
- reactjavascript 4y ago
  >> "mount a sqlite database file as a normal filesystem" I'm looking to do the opposite. Given a location, recurse it, stat() each file and create the database.
  - my69thaccount 4y ago
    You could also make it as a Sqlite virtual table so it's dynamic. Also file data is available in osquery: https://osquery.io/schema/5.2.3/#file https://osquery.io/schema/5.2.3/#file
  - samatman 4y ago
    It's still not clear to me whether you want the files inside the database or just the metadata. The stat part suggests the latter? I guess I can see that, but now you have cache invalidation (someone else linked to Spotlight, which does this as a background process). SQLite files can be larger than any physical medium you can purchase so why not go the distance?
  - neura 4y ago
    If you happen to be using macOS, there's mdfind, which uses the spotlight database which is always kept up to date, unlike locate/updatedb, where updatedb is expensive to run, even if you've run it recently. I have yet to find a good solution for linux CLI... something that uses an internal database that is kept up to date with all directory structure changes. Maybe someone else has seen something cool for this? :D
jeffbee 4y ago
Sounds like macOS would be right up your alley then.
- reactjavascript 4y ago
  I have MacOS, where is the sqlite file?
  - jeffbee 4y ago
    /private/var/db/Spotlight-V100/...
  - ashton314 4y ago
    Is the parent post referring to `mdfind` perhaps?
    - neura 4y ago
      Believe so.
mi_lk 4y ago
updatedb/locate?
- smoothgrammer 4y ago
  I'm shocked so many people forget this
  - cb321 4y ago
    https://plocate.sesse.net/ https://plocate.sesse.net/ is even better
- neura 4y ago
  updatedb is expensive to run and is usually run by a cron job, whereas something like mdfind on macos uses spotlight, which is kept up to date with all filesystem changes (or at least reasonably fast). Anybody know of something like that for linux?
mekster 4y ago
https://github.com/kashav/fsql https://github.com/kashav/fsql

I just want my entire file system stored in Sqlite so I can query it myself

samatman 4y ago

Oh hey look what I just found https://github.com/narumatt/sqlitefs https://github.com/narumatt/sqlitefs

reactjavascript 4y ago
>> "mount a sqlite database file as a normal filesystem" I'm looking to do the opposite. Given a location, recurse it, stat() each file and create the database.
- my69thaccount 4y ago
  You could also make it as a Sqlite virtual table so it's dynamic. Also file data is available in osquery: https://osquery.io/schema/5.2.3/#file https://osquery.io/schema/5.2.3/#file
- samatman 4y ago
  It's still not clear to me whether you want the files inside the database or just the metadata. The stat part suggests the latter? I guess I can see that, but now you have cache invalidation (someone else linked to Spotlight, which does this as a background process). SQLite files can be larger than any physical medium you can purchase so why not go the distance?
- neura 4y ago
  If you happen to be using macOS, there's mdfind, which uses the spotlight database which is always kept up to date, unlike locate/updatedb, where updatedb is expensive to run, even if you've run it recently. I have yet to find a good solution for linux CLI... something that uses an internal database that is kept up to date with all directory structure changes. Maybe someone else has seen something cool for this? :D

jeffbee 4y ago

Sounds like macOS would be right up your alley then.

reactjavascript 4y ago
I have MacOS, where is the sqlite file?
- jeffbee 4y ago
  /private/var/db/Spotlight-V100/...
- ashton314 4y ago
  Is the parent post referring to `mdfind` perhaps?
  - neura 4y ago
    Believe so.

mi_lk 4y ago

updatedb/locate?

smoothgrammer 4y ago
I'm shocked so many people forget this
- cb321 4y ago
  https://plocate.sesse.net/ https://plocate.sesse.net/ is even better
neura 4y ago
updatedb is expensive to run and is usually run by a cron job, whereas something like mdfind on macos uses spotlight, which is kept up to date with all filesystem changes (or at least reasonably fast). Anybody know of something like that for linux?

mekster 4y ago

https://github.com/kashav/fsql https://github.com/kashav/fsql