3 ms·

they could have gone the other way, throw everything into a DB, and then wrote a fuse plugin to access it all through traditional file system This is t

they could have gone the other way, throw everything into a DB, and then wrote a fuse plugin to access it all through traditional file system This is the Camlistore strategy! Of course, there are other problems with that approach Could you elaborate more on these? I've never worked with FUSE.

kbenson 10y ago
The other problems I was alluding to weren't really with FUSE, but one that does pertain to FUSE is speed, since FUSE imposes overhead through a daemon running in user space, and associated context level switches because of that. From just looking into is again, this may have been mitigated to some larger or smaller degree with some FUSE performance enhancements in 2012. Specifically, I was referring to the different off the shelf database systems which could be used. Each will have it's own benefits and drawbacks to storing large chunks of data per-record. Benefits might include (relatively) easy sharding or replication. Drawbacks might include not being space efficient for removed files, not being as resilient to corruption due to crashes or corruption affecting more than the files in use, or overly aggressive use of memory to function efficiently. If a custom database was developed, you could tailor to your exact needs, but then you have much more work to do, and a period of immaturity. Off the top of my head, if I were designing a general purpose system for tagging files where people were expected to use it as a regular file system and some overhead from FUSE was acceptable, I think I would leverage the file system but in a different way. I would set up a specialized directory for the files themselves, and store then hashed within it, and have a BerkelyDB database relate filename to hash and tags, and use FUSE to do direct file access. But that's my 5 minute assessment, so I reserve the right to change it completely given someone pointing out the obvious problems. :)

kbenson 10y ago

The other problems I was alluding to weren't really with FUSE, but one that does pertain to FUSE is speed, since FUSE imposes overhead through a daemon running in user space, and associated context level switches because of that. From just looking into is again, this may have been mitigated to some larger or smaller degree with some FUSE performance enhancements in 2012. Specifically, I was referring to the different off the shelf database systems which could be used. Each will have it's own benefits and drawbacks to storing large chunks of data per-record. Benefits might include (relatively) easy sharding or replication. Drawbacks might include not being space efficient for removed files, not being as resilient to corruption due to crashes or corruption affecting more than the files in use, or overly aggressive use of memory to function efficiently. If a custom database was developed, you could tailor to your exact needs, but then you have much more work to do, and a period of immaturity. Off the top of my head, if I were designing a general purpose system for tagging files where people were expected to use it as a regular file system and some overhead from FUSE was acceptable, I think I would leverage the file system but in a different way. I would set up a specialized directory for the files themselves, and store then hashed within it, and have a BerkelyDB database relate filename to hash and tags, and use FUSE to do direct file access. But that's my 5 minute assessment, so I reserve the right to change it completely given someone pointing out the obvious problems. :)