5 ms·
Sqlitefs: SQLite as a Filesystem
- szundi 2y agoFilesystems are DB like structures these days anyway.
- jeltz 2y agoAre they? From my experience from databases and some limited from filesystems I would disagree. Sure, there are some similarities but not that much.
- pdimitar 2y agoWhere do you get that from? In my practice this was never true and still is not.
- chipdart 2y ago> Filesystems are DB like structures these days anyway. I think you're confusing the fact that an application uses data structures with that application being like a database.
- lupire 2y agoOther way around. Applications use databases instead of the filesystem.
- tom1337 2y agoOk so if we take this repo and then the headline from https://news.ycombinator.com/item?id=41085376 https://news.ycombinator.com/item?id=41085376, we could get an infinitely fast file system by just putting another SQLite filesystem on top.
- bhaney 2y agoThere's a recursive CTE joke in here somewhere
- deleted 2y ago[deleted]
- bilekas 2y agoI was just thinking the same thing.. Im supposed to be on holidays but damn it I need to test this.
- bee_rider 2y agoIMO in this fantasy it should go like (1-.35)^n, not 1-.35n. We must keep our harebrained schemes plausible.
- punnerud 2y agoIf SQLite is faster than a normal file system, how fast is SQLite running on “SQLite as a filesystem”?
- zoky 2y agoIt turns out if you run SQLiteFS on SQLiteFS, performance increases exponentially, and as the number of SQLiteFS layers increases, seek and read times asymptotically approach zero. In other words, this is the world’s first O(0) file system. At this rate, with only a few performance tweaks, SQLiteFS should be able to retrieve files before you even ask for them.
- earthboundkid 2y agoThe problem is that it sends you files before you need them, which clogs up throughput as you try to figure out what the files are supposed to be for. It's more efficient to slow everything back down to NVMe speeds.
- smitty1e 2y agoLess humorously, if one were calculating a few files for compilation, or caching for a service, a SqliteFS using :memory: might be just the thing.
- bilekas 2y agoJokes aside, if I understand correctly it's still assumed the performance increase from the sqlLite is down to the reduced number of OPEN and CLOSE calls. So if that's the case, then it will be as slow as however many sqllites you're running? Meaning it should in theory lose performance?!
- lupire 2y agoObviously adding redundant layers will reduce performance.
- jefebromden 2y agoWill anyone mention the repo hasn't been updated on 4 years?
- willcipriano 2y agoWhen have you last updated your filesystem?
- endorphine 2y agoA fair analogy would be: "when was upstream code of the filesystem was last updated?". To that question, I doubt one would find a polular filesystem that was updated longer than 1 year ago (maybe I'm too liberal).
- Jach 2y ago# zpool history | grep upgrade Looks like last May.
- yjftsjthsd-h 2y agoThat's the on-disk format, you probably updated the actual driver code a lot more recently than that
- Jach 2y agoThey didn't specify what kind of update, but in this case no, I'm on zfs-2.2.3, which was released in February. I'd have to go check out when it was marked stable by Gentoo and I actually updated. (I do occasionally have to rebuild whatever version I'm on when upgrading kernel versions.)
- willcipriano 2y agoWhat was in that update that you are worried about this missing?
- Jach 2y ago
- alberth 2y agoReminds me of Microsoft WinFS. Which was essentially SQL Server as a filesystem. Project was ultimately cancelled. https://en.m.wikipedia.org/wiki/WinFS https://en.m.wikipedia.org/wiki/WinFS
- krylon 2y agoI'm not a big fan of Windows, but I'm a bit sad that the project didn't go anywhere forever and then was cancelled. They had some pretty ambitious and interesting ideas for it. E.g. the only public beta of Vista that had it enabled used the database as a backing store for Outlook contacts. Each contact was an object in the database. So were photos. And you could link - as I recall, I only read about it back then - people to photos. And then search on that data, of course. Like, "give me all emails, documents, and photos that have to do with person X, from 2022-2023". But IIRC, the performance was atrocious. They might have been able to fix that, but instead it ended on the junk heap of unrealized ideas.
- HeckFeck 2y agoI remember those Longhorn betas - yes, it was coming apart under feature creep, but that only gave us more to play with and anticipate in the leaks. A shame that MS is no longer in the mindset of ambition like this. Instead we get adverts and clickbait in the Start Menu, and the worst filesystem search on any OS I've experienced.
- naikrovek 2y agoI feel like I need to inform you that the people doing the “recommendations” in the Windows start menu are not the same people who write filesystems. The venn diagram of those two sets of people is two non-touching, non-intersecting circles. There are interesting things going on, I’m sure, but we won’t see any of it until this AI baloney is behind us. Nerds create a virtual moron that gets a large percentage of stuff wrong and execs demand that it be inserted into every product made immediately. An absolute genius move only an MBA can explain.
- greenthrow 2y agoWhy would you bother with this instead of just making an in memory drive? I remember mounting in memory drives even in MS-DOS back in the early 90s. I did it to move whatever game I wanted to play into the RAM Drive so my load times would be faster.
- BrandoElFollito 2y agoI use this today on my Raspberry Pis to load the logs and other active IO to memory and avoid wearing out the SD card.
- greenthrow 2y agoOk but why not just use a memory drive directly? What is gained by adding SQLite as a layer?
- throwaway984393 2y ago[dead]
- BrandoElFollito 2y agoI am commenting on your "I remember mounting in memory drives even in MS-DOS back in the early 90s.". Just to say this this is still a thing today. I also mounted drives in memory in early 90's :)
- bee_rider 2y agoVideogames have gotten so ridiculously huge nowadays, we’ll be at the prosumer/workstation cusp to make this work nowadays.
- gbonc2 2y agoI wrote this one, I tested with PostgresDB and Oracle , at time: https://github.com/gbonacini/dbfs https://github.com/gbonacini/dbfs
- deleted 2y ago[deleted]
- adius 2y agoRelated: SQLiteDAV - WebDAV server that maps an SQLite database to directories/files. https://github.com/Airsequel/SQLiteDAV https://github.com/Airsequel/SQLiteDAV
- deleted 2y ago[deleted]
- DemocracyFTW2 2y agoIt works (on Linux)! Should add that I only got to do what it does after following the advice expressed in the error message: fusermount: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf It did not work using `sudo`
- sillywalk 2y agoFor the bored, I was searching for 'database filesystem' and found one by Oracle. I gather it's a way to make editing BLOBs easier. That could make for some interesting recursive things. It's "a file system interface placed on top of files and directories that are stored in database tables."[0] [0] https://docs.oracle.com/en/database/oracle/oracle-database/19/adlob/introducing-database-file-system.html https://docs.oracle.com/en/database/oracle/oracle-database/1...