10 ms·
Freqfs: In-memory filesystem cache for Rust
- amelius 5y ago> freqfs automatically caches the most frequently-used files and backs up the others to disk. This allows the developer to create and update large collections of data purely in-memory without explicitly sync’ing to disk, while still retaining the flexibility to run on a host with extremely limited memory. Why not let the OS take care of this?
- haydnv 5y agoOne advantage is consistency across host platforms, but the main advantage is that the file data can be accessed (and mutated) in memory in a deserialized format. If you let the OS take care of it, you would still have the overhead of serializing & deserializing a file every time it's accessed.
- dfranke 5y agoThat's what mmap is for.
- deleted 5y ago[deleted]
- haydnv 5y agoIt might be possible to replace freqfs with mmap on a POSIX OS, but a) you would still have to implement your own read-write lock, and b) you would (I think probably?) lose some consistency in behavior across different host operating systems.
- gpderetta 5y agommaps for read, explicit API for writing, a-la LMDB. Buggy readers can read inconsistent data but cannot corrupt the os.
- vlovich123 5y agoWhich OSes does this run on that doesn’t have some kind of mmap operation?
- alexruf 5y agoPersonally I don’t see a scenario for myself, but I can imagine that there are some where this might be useful. But isn’t there a extremely high risk of data loss an inconsistency when adding an extra layer on top of OS file system handling?
- haydnv 5y agoIf there is any concurrent access to cached files not through freqfs, there is a risk of inconsistency and crashes.
- ericbarrett 5y agoFreqfs seems like a shim you'd add to an existing project for a quick optimization. Whereas mmap et al. are "better" the same way any specific, built-to-purpose code will be "better" than just bolting a framework on. Sometimes it's the right call to do the extra work; sometimes it's 100% more effort (both development and maintenance) for an extra 10% gain.
- jdeaton 5y agoMy thought exactly.
- shakna 5y agoPresumably for a similar usecase as SQLite [0]. Performance. You can beat the OS, and by a noticeable margin, by doing things in memory and avoiding the I/O bottleneck. [0] https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html
- cornstalks 5y agoI think GP's point is that the OS usually has a file system cache in RAM.
- topspin 5y agoI think the P's point, supported by evidence, is that the OS cache is not optimal for all use cases.
- edoceo 5y agoNo cache is optimal for all use cases. That's an impossible goal.
- fsckboy 5y agothe actual question is whether the person making the choice is up to the task of measuring/proving that their manual caching is optimal for their use case. A lot of the time the answer is simply "no". For example, the people who say "just buy enough RAM and turn virtual memory off" for the most part do not understand the implications of what they are talking about.
- BiteCode_dev 5y agoYou can pick and chose. Maybe your caching strat of you OS isn't best for your use case. Also, you may use a network file system, or several types of FS, and want your cache warm up to be tuned up and consistent.
- the8472 5y ago> Maybe your caching strat of you OS isn't best for your use case. On the other hand the OS does know about memory pressure from IO and from heap memory for the whole system. This crate will only know about cache pressure within a single process. > Also, you may use a network file system Which can also be set to do aggressive caching, at the expense of consistency. > and want your cache warm up to be tuned up and consistent. the description doesn't say that it's doing cache warmup any more eagerly as regular reads would
- kbenson 5y agoThe benefit of bringing something in process is as always more control, and usually at the expense of having to make decisions with less data about the rest of the system than an OS level service would have. Sometimes you need very explicit control over when things are read from cache and when they aren't. This can be hard with network file systems. Especially when you have two different use cases on the same filesystem, which isn't that odd, even within a single application.
- Svetlitski 5y agoUpon seeing this, I can’t help but think of “So What’s Wrong with 1975 Programming” from the author of Varnish [1]. [1] https://varnish-cache.org/docs/trunk/phk/notes.html https://varnish-cache.org/docs/trunk/phk/notes.html
- munro 5y agoMusing: sometimes I wish file systems & databases were the unified. I'm imaging just a single fast db engine sitting on my storage—and your traditional file system structure would just be tables in there. I kinda just treat SQLite like that, but it's not as transparently optimized as it could be for large files. Why? I don't want to mentally jump around technologies. I want to query my FS like a DB, and I want to store files in my DB like a FS. The reality is though that there isn't a one size fits all DB that exists. And more on topic: tokio-uring is really fast [1], and I'm really loving tokio in general. [1] https://gist.github.com/munro/14219f9a671484a8fe820eb35d26bb80 https://gist.github.com/munro/14219f9a671484a8fe820eb35d26bb...
- arghwhat 5y agoYour filesystem is a database. It's just a document-oriented database, rather than relational SQL.
- cogman10 5y agoIt even has a lot of the same features as a full fledged DB. For example, most file systems today are journaling. Which is exactly how most databases handle atomic, consistent, and durability in ACID. About the only thing it's missing is automatic document locking (though most file systems support explicit locks). That said, there are often some pretty hard limits on the number of objects in a table (directory). Depending on the file system you can be looking at anywhere from 10k to 1 billion files per directory. There are also some unfortunate storage characteristics. Most file systems have a minimum file size of around 4kb, mostly to optimize for disk access. DBs often pack things together much more tightly. But hey, if you can spin using the FS as a DB... Do it. Particular for a read heavy application, the FS is nearly perfect for such operations.
- gpderetta 5y agoThe biggest problem is the lack of good transactional facilities.
- axegon_ 5y agoWhy hasn't anyone told me about this?!?!? I love you so much for posting, I needed something like this for a personal project I'm fiddling around with in my spare time.
- deleted 5y ago[deleted]
- mountainboy 5y agorequires tokio / async. ugh. I'm out.
- nonameiguess 5y agoI honestly don't think I like this at the application level. You're removing a degree of freedom from operators and users. I have a ton of memory on all of my home devices, and usually just take the working directories for frequently used applications and mount them as tmpfs. I do the same thing for application working directories of applications I deploy at work as well, where we have complete freedom to deploy memory-optimized servers with lots of RAM. Putting an extra in-memory cache on top of the OS filesystem that is already in-memory is an unnecessary extra step and doubling the memory use of each file that can't be turned off without patching and recompiling your application. The OS is already smart enough not to add a cache on top of tmpfs.
- haydnv 5y agoI don't know that it's fair to say it's "doubling" the memory use of each file because the OS cache memory is still "free" from the perspective of an application. Where it comes in handy is an applications like databases or training an ML model where there are hot spots that get accessed/updated extremely frequently--then the application doesn't have to incur serialization overhead in order to read/write the data that the file encodes (although as another poster pointed out it might also be possible to do this with mmap).
- __s 5y agoUnfortunately this doesn't meet my use case (which they list as an intended use case): serving static assets over http. I currently use an in-memory cache without eviction. It doesn't meet my requirements because I store the in-memory content precompressed https://github.com/serprex/openEtG/blob/master/src/rs/server/src/handleget.rs https://github.com/serprex/openEtG/blob/master/src/rs/server... edit: seems it can. Nice
- jagged-chisel 5y agoI think I don't understand the problem. Precompressed files are still files and can be cached.
- WJW 5y agoI think what they mean is that they want the file on disk to be uncompressed but the file in memory to be compressed.
- haydnv 5y agoI think if you call your precompression function in FileLoad::load it should do what you need--please file an issue if this is not the case: https://github.com/haydnv/freqfs/issues https://github.com/haydnv/freqfs/issues
- sagichmal 5y agoDon't filesystems already do this? Like, really really good?
- hexo 5y agoIs this some kind of excercise or what? I mean, my OS already has filesystem cache why would I need another one? More buffer bloat?
- tedunangst 5y ago> This crate assumes that file paths are valid Unicode and may panic if it encounters a file path which is not valid Unicode. Love it when a program could simply work, but chooses to fail because it doesn't like my life choices.
- habibur 5y agoThis will hardly increase performance if any. Note that OS caches frequently used files in-memory. When you are using this, you are basically competing with the OS for in-memory file cache. This library might have other uses that I am not aware of.
- fsckboy 5y agoare there embedded systems, including potentially realtime, that are running OSes or on CPUs that don't provide caches or page fault support?
- brandmeyer 5y agoIn those cases, the filesystem provides its own block cache.
- haydnv 5y agoI realized based on several of the comments here that I should have included a comparison with OS filesystem caching in the documentation for freqfs. I will update this in the next release. The major advantage of freqfs over just letting the OS handle file caching is that with freqfs you can read and mutate the data that your file represents purely in memory. For example, if you implement a BTree node as a struct, you can just borrow the struct mutably and update it, and it will only be synchronized with the filesystem in the event that it's evicted (or you explicitly call `sync`). This avoids a lot of (de)serialization overhead and defensive coding against an out-of-memory error. Again, I will update the documentation to clarify.
- spullara 5y agoThis is called mmap?
- haydnv 5y agommap does map file contents into memory but does not provide read-write locking. The main limitation is that a file is not necessarily the same thing as the Rust data structure that it encodes. For example, if you store a Rust Vec in a file, loading the file with mmap won't allow you to `push` a new item onto the Vec.
- minroot 5y agoWhy did you link to doc.rs instead of to the source repository?