5 ms·

Property testing is awesome, but it does significantly slow down test suites. Are there any standard practices for striking a balance of getting the added confi

Property testing is awesome, but it does significantly slow down test suites. Are there any standard practices for striking a balance of getting the added confidence/safety of using property tests without introducing large delays into your CI pipeline?

mullr 6y ago
We like to run the tests as part of CI with a relatively small number of iterations, and then turn the knob way up in a nightly or weekly scheduled test job.
- Smaug123 6y ago
  Yup. We've got some stuff set up so that in the IDE your tests finish in ~100ms; in precommit testing pipelines you get several seconds; in nightly tests you get two hours.
k__ 6y ago
You could fan-out tests to multiple FaaS instances. Cloudflare Workers, for example, have no cold-starts. But they are JS/WASM only
- amw-zero 6y ago
  That’s a lot of buzzwords you got there
hinkley 6y ago
My understanding, shallow as it is, is that property testing goes great with pure functional code, because without side effects you can run tests in parallel, taking advantage of all of these cores and build servers we have. If your tests are coupled, you're already in a bad way whether you know it or not. Dumping property testing on top of that without addressing the underlying cause sounds like a recipe for misery. It's probably a great stick and carrot if you're pushing a tech debt reduction agenda though.
Jtsummers 6y ago
Item 5 in TFA is one that I found related to performance most. Using filters to toss out bad input means you're potentially generating thousands of inputs for dozens of tests. Aggregate this across all your tests and it's an incredible waste of time. Constructing proper inputs directly can get you closer to the minimum time required to execute the tests and help achieve that desired performance threshold. (NB: Not used professionally, only got one office to even entertain them but they wouldn't take it and run with it even after I showed them a dozen errors I'd found in an hour of using them. Consequently my experience is still limited.)

mullr 6y ago

We like to run the tests as part of CI with a relatively small number of iterations, and then turn the knob way up in a nightly or weekly scheduled test job.

Smaug123 6y ago
Yup. We've got some stuff set up so that in the IDE your tests finish in ~100ms; in precommit testing pipelines you get several seconds; in nightly tests you get two hours.

k__ 6y ago

You could fan-out tests to multiple FaaS instances. Cloudflare Workers, for example, have no cold-starts. But they are JS/WASM only

amw-zero 6y ago
That’s a lot of buzzwords you got there

hinkley 6y ago

My understanding, shallow as it is, is that property testing goes great with pure functional code, because without side effects you can run tests in parallel, taking advantage of all of these cores and build servers we have. If your tests are coupled, you're already in a bad way whether you know it or not. Dumping property testing on top of that without addressing the underlying cause sounds like a recipe for misery. It's probably a great stick and carrot if you're pushing a tech debt reduction agenda though.

Jtsummers 6y ago

Item 5 in TFA is one that I found related to performance most. Using filters to toss out bad input means you're potentially generating thousands of inputs for dozens of tests. Aggregate this across all your tests and it's an incredible waste of time. Constructing proper inputs directly can get you closer to the minimum time required to execute the tests and help achieve that desired performance threshold. (NB: Not used professionally, only got one office to even entertain them but they wouldn't take it and run with it even after I showed them a dozen errors I'd found in an hour of using them. Consequently my experience is still limited.)