4 ms·
Not entirely related (except the block size), but I am considering making and standardizing a system-wide content-based cache with default block size 16KB. The
by lostmsu 2y ago
Not entirely related (except the block size), but I am considering making and standardizing a system-wide content-based cache with default block size 16KB.
The idea is that you'd have a system-wide (or not) service that can do two or three things:
- read 16KB block by its SHA256 (also return length that can be <16KB), if cached
- write a block to cache
- maybe pin a block (e.g. make it non-evictable)
I would be like a block-level file content dedup + eviction to keep the size limited.
Should reduce storage used by various things due to dedup functionality, but may require internet for corresponding apps to work properly.
With a peer-to-peer sharing system on top of it may significantly reduce storage requirements.
The only disadvantage is the same as with shared website caches prior to cache isolation introduction: apps can poke what you have in your cache and deduce some information about you from it.
- monocasa 2y agoI'd probably pick a size greater than 16KB for that. Windows doesn't expose translations less than 64KB in their version of mmap, and internally their file cache works in increments of 256KB. And these were numbers they picked back in the 90s.
- treyd 2y agoI would go for higher than 16K. I believe BitTorrent's default minimum chunk size is 64K, for example. It really depends on the use case in question though, if you're doing random writes then larger chunk sizes quickly waste a ton of bandwidth, especially if you're doing recursive rewrites of a tree structure. Would a variable chunk size be acceptable for whatever it is you're building?
- lostmsu 2y agoI could feasibly do 2x partitioning. E.g. have caches for 16KB, 32KB, etc provided there's some mechanism to automatically combine/split pieces