5 ms·
I do a lot of manual proptesting in Rust that all look something like: let mut rng = rand::thread_rng(); for action_count in 1..4 { for _ in 0.
by gamegoblin 2y ago
I do a lot of manual proptesting in Rust that all look something like:
let mut rng = rand::thread_rng();
for action_count in 1..4 {
for _ in 0..10_000 {
let seed = rng.gen::<u64>();
eprintln!("let seed = {seed};");
let mut rng = ChaChaRng::seed_from_u64(seed);
i.e. top level true randomness, then a bunch of nested loops (only 2 here, but some tests have more) to go from low-complexity cases to high-complexity
then generate a seed to seed a deterministic PRNG, and print it out so if the test fails, I just copy and paste the error seed to replay the error case
I have found doing this manual proptesting to be faster, more flexible, and generally less fuss than using any frameworks or libraries
That said, for really robust concurrency testing, I cannot recommend enough the AWS Shuttle library (https://github.com/awslabs/shuttle https://github.com/awslabs/shuttle) which can find insanely complicated race conditions. I wrote a little tutorial on it here: https://grantslatton.com/shuttle https://grantslatton.com/shuttle
We used it at AWS to verify the custom filesystem we wrote to power AWS S3.
- nextaccountic 2y agoHow do you do shrinking? IMO that's the property testing killer feature
- gamegoblin 2y agoThat's why my outer loop goes from low-complexity cases to high-complexity cases, it has basically the same effect as shrinking (without actually having to do any work)
- cpeterso 2y agoLike a breath-first search instead of depth-first so you hit the shorter path first?
- gamegoblin 2y agoYes exactly, though not exhaustive, so more like beam search actually
- josephg 2y agoI do this sort of thing too. I don’t have an automated way to shrink my input - but that’s usually fine in practice. For example, say I’m testing a data structure. I’ll have an outer loop that picks a seed and an inner loop that does ~100 mutations of a data structure instance, testing assertions each time. If there’s a failure, I’ll try a bunch of seeds to look for one that fails the fastest. (In the fewest possible inner loop iterations). It’s not a perfect system, but for most bugs I can usually get a test case that only needs 5-10 steps before a crash occurs. And if the reduction step finds a different bug? That’s no problem at all. I fix what I found and go back to trying more seeds.
- sshine 2y agoI gave up on proptest and did this, too. What I miss is: Shrinking, deterministic replay.
- hansvm 2y agoYou are talking about an FS powering S3, not s3fs, right?