5 ms·

I'd like to go the other way, and have a filesystem that leverages a proper database for its metadata. I can search millions of records in a split second in SQ

I'd like to go the other way, and have a filesystem that leverages a proper database for its metadata. I can search millions of records in a split second in SQL. But searching records on my hard disk takes enormously more time. The solutions out there for speeding it up (like Windows Indexing service) rely on asynchronous indexing that's patched on top of the filesystem instead of integrated realtime within it, and are subsequently brittle (in my experience), slow down your computer at inconvenient times, and too often aren't instantly up date. Being able to have stored procedures execute when changes trigger would be another interesting aspect that could take the place of event filters and file watchers (especially if critical portions could be configured to happen sequentially & atomically, akin to IRQ handlers). I feel like databases have gotten lots of love over the decades and are super optimized (and by this point are pretty dang reliable). Filesystems feel like they lost their time in the spotlight and have floundered with less innovation. Hardware that calculates CRC's of blocks in realtime would also be awesome (imagine what it would do for sync). The ironic thing is your HDD already does this, there's just no way to tap into that information at the software / filesystem driver level.

jasode 3y ago
>I'd like to go the other way, and have a filesystem that leverages a proper database for its metadata. Microsoft made an attempt back in 2003 at creating a "smarter" file system with rich metadata based on a real database engine (MS SQL Server) ... but eventually abandoned the project: https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS WinFS wasn't necessarily going to completely replace NTFS but Microsoft did have ambitious plans for its integration into everything. It's interesting that other platforms like Unix/Linux/Apple that were designing "next generation" file systems like ZFS/ext4/APFS after the failed WinFS project had a chance to try adding their own WinFS smart metadata features to but they didn't. The later fragmentation of user data being spread around in the cloud such as AWS objects, Apple iCloud, Backblaze backups, etc instead of every file saved on home computers makes a "universal rich metadata index stored in a database engine" to be a cross-platform standard by all operating systems less realistic today than back in 2003. Therefore, the core os file systems remain a "dumb blob of bytes" with a separate layer of metadata index db in vendor-specific schemes like Apple Spotlight db, Windows Search in ".edb" file, etc.
- rkagerer 3y ago
  Yeah they totally missed the mark with WinFS, in my opinion. It was too exotic/foreign with no clear adoption path. All I want is dumb files with something like SQLite replacing or augmenting the allocation table. Then sprinkle on some new capabilities incrementally.
- didgetmaster 3y ago
  When WinFS was canceled, I started my own pet project to build a new kind of file system that totally changed the way file metadata was stored and managed. I built an object based system that could easily handle hundreds of millions of files within a single container (what I called a pod) and do searches much like how you would search a large database table. The 'file table' contains a 64 byte record for each object and has some very unique characteristics that make searches incredibly fast (e.g. it can find all the photos in under a second, even if there are 20 million of them among 200 million files). You can also attach various metadata 'tags' to each object and find matching objects based on them. The project is in open beta and anyone can download it and try it out on their own computer. https://www.didgets.com https://www.didgets.com
nvm0n2 3y ago
The reason indexing services don't work synchronously is because it would cause file IO to get far slower than apps expect, and that can break user interfaces and cause unexpected slow downs. There's also no real reason for it to be synchronous in most cases.
EvanAnderson 3y ago
You'd probably enjoy reading about the Be File System: https://en.wikipedia.org/wiki/Be_File_System https://en.wikipedia.org/wiki/Be_File_System It has relational database-like semantics for metadata.

jasode 3y ago

>I'd like to go the other way, and have a filesystem that leverages a proper database for its metadata. Microsoft made an attempt back in 2003 at creating a "smarter" file system with rich metadata based on a real database engine (MS SQL Server) ... but eventually abandoned the project: https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS WinFS wasn't necessarily going to completely replace NTFS but Microsoft did have ambitious plans for its integration into everything. It's interesting that other platforms like Unix/Linux/Apple that were designing "next generation" file systems like ZFS/ext4/APFS after the failed WinFS project had a chance to try adding their own WinFS smart metadata features to but they didn't. The later fragmentation of user data being spread around in the cloud such as AWS objects, Apple iCloud, Backblaze backups, etc instead of every file saved on home computers makes a "universal rich metadata index stored in a database engine" to be a cross-platform standard by all operating systems less realistic today than back in 2003. Therefore, the core os file systems remain a "dumb blob of bytes" with a separate layer of metadata index db in vendor-specific schemes like Apple Spotlight db, Windows Search in ".edb" file, etc.

rkagerer 3y ago
Yeah they totally missed the mark with WinFS, in my opinion. It was too exotic/foreign with no clear adoption path. All I want is dumb files with something like SQLite replacing or augmenting the allocation table. Then sprinkle on some new capabilities incrementally.
didgetmaster 3y ago
When WinFS was canceled, I started my own pet project to build a new kind of file system that totally changed the way file metadata was stored and managed. I built an object based system that could easily handle hundreds of millions of files within a single container (what I called a pod) and do searches much like how you would search a large database table. The 'file table' contains a 64 byte record for each object and has some very unique characteristics that make searches incredibly fast (e.g. it can find all the photos in under a second, even if there are 20 million of them among 200 million files). You can also attach various metadata 'tags' to each object and find matching objects based on them. The project is in open beta and anyone can download it and try it out on their own computer. https://www.didgets.com https://www.didgets.com

nvm0n2 3y ago

The reason indexing services don't work synchronously is because it would cause file IO to get far slower than apps expect, and that can break user interfaces and cause unexpected slow downs. There's also no real reason for it to be synchronous in most cases.

EvanAnderson 3y ago

You'd probably enjoy reading about the Be File System: https://en.wikipedia.org/wiki/Be_File_System https://en.wikipedia.org/wiki/Be_File_System It has relational database-like semantics for metadata.