4 ms·
PlentyFS: PoC read-only filesystem with random contents generated on demand
- rwmj 6y agoFor something similar at the block level try: nbdkit random 1G nbd-client localhost /dev/nbd0 Any size can be used and there is a seed= parameter to get the same data between runs. You might need "modprobe nbd". As in this article, we also use this for testing.
- baq 6y agondb is also quite fun in that you can rate-limit easily, so you can check what happens with your app on slow storage. don't make it too slow, the Linux kernel doesn't like that and can hang completely.
- dredmorbius 6y agohttps://en.wikipedia.org/wiki/Network_block_device https://en.wikipedia.org/wiki/Network_block_device If I'm understanding correctly, this still requires one instance with real (vs. virtual) storage on it? The advantage of PlentyFS is that it effectively requires no storage anywhere. I didn't write PlentyFS, but was participating (mostly not helpfully) in Lars Wirzenius's thread on Mastodon. My idea was to look at an HTTP or WebDAV solution using FUSE. The server would "create' the content (thinking along the line of HTTP honeypots), the FUSE module would present the filesystem-based access. There's an existing WebFS fuse modele, also a WebDAV one IIRC. Unfortunately, HHTP doesn't offer directory presentation, making full filesystem semantics difficult. I'm looking at other possible options as well, the concept is interesting.
- monocasa 6y agoNeat. Reminds me of Jorge Luis Borges's short story "The Library of Babel". https://en.wikipedia.org/wiki/The_Library_of_Babel https://en.wikipedia.org/wiki/The_Library_of_Babel
- coolspot 6y agoSee also: pifs - https://github.com/philipl/pifs https://github.com/philipl/pifs
- dredmorbius 6y agoThe inspiration for this was wanting to develop and test file access and backup tools on many large files, without actually requiring the physical storage for origin data, and minimal memory or system overheaad. Files are regular files (not FIFO's, named pipes, sockets, or symlinks), and data are random (not simply ASCII null as with /dev/zero). Access is read only. To be clear, this is a test rig, not for actual data storage.
- rwmj 6y agoSome comments: The seed should be an optional parameter (I see there's a comment in the code mentioning this). This is very important for CI testing where a bug only happens for specific file content and you need to reproduce the bug locally. Why use a cryptographic hash? For nbdkit-random-plugin we use a fast PRNG (xoshiro256-starstar 1.0 as it happens, but the specific one is not important). I doubt a cryptographically secure hash is worth the extra computational cost to users.
- dredmorbius 6y agoI'm really just a bystanderhere. File an issue w/ the project.
- minoru 6y agoHi, the author here. Thanks for taking a look at the code! I picked SHA-1 because it was the simplest way to generate pseudorandom data: I knew there's a Rust crate for it which I used before. Works fine for a proof-of-concept :) I'm now reading up on other hashes and ciphers. My tentative plan was to replace SHA-1 with AES-NI in counter mode, but if I can find something that works on all CPUs while giving me a comparable performance, I'll go for that. (Should probably look at PRNGs as well; if I concatenate the file seed with the counter and use that as a seed for a PRNG, then ask it for a value -- it should be good enough too.) I doubt Lars has requirements for the quality of randomness, but we'll see.