3 ms·
The inspiration for this was wanting to develop and test file access and backup tools on many large files, without actually requiring the physical storage for o
by dredmorbius 6y ago
The inspiration for this was wanting to develop and test file access and backup tools on many large files, without actually requiring the physical storage for origin data, and minimal memory or system overheaad.
Files are regular files (not FIFO's, named pipes, sockets, or symlinks), and data are random (not simply ASCII null as with /dev/zero).
Access is read only.
To be clear, this is a test rig, not for actual data storage.
- rwmj 6y agoSome comments: The seed should be an optional parameter (I see there's a comment in the code mentioning this). This is very important for CI testing where a bug only happens for specific file content and you need to reproduce the bug locally. Why use a cryptographic hash? For nbdkit-random-plugin we use a fast PRNG (xoshiro256-starstar 1.0 as it happens, but the specific one is not important). I doubt a cryptographically secure hash is worth the extra computational cost to users.
- dredmorbius 6y agoI'm really just a bystanderhere. File an issue w/ the project.
- minoru 6y agoHi, the author here. Thanks for taking a look at the code! I picked SHA-1 because it was the simplest way to generate pseudorandom data: I knew there's a Rust crate for it which I used before. Works fine for a proof-of-concept :) I'm now reading up on other hashes and ciphers. My tentative plan was to replace SHA-1 with AES-NI in counter mode, but if I can find something that works on all CPUs while giving me a comparable performance, I'll go for that. (Should probably look at PRNGs as well; if I concatenate the file seed with the counter and use that as a seed for a PRNG, then ask it for a value -- it should be good enough too.) I doubt Lars has requirements for the quality of randomness, but we'll see.