6 ms·

I find fuzzing fascinating but it seems to only be used in protocols and tools such as compilers, networking protocol implementations etc. Does anyone have expe

I find fuzzing fascinating but it seems to only be used in protocols and tools such as compilers, networking protocol implementations etc. Does anyone have experience using it for business logic?

alexpotato 6y ago
Having worked in FinTech (big bank, HFT, prop shop, start up hedge fund etc) for 10+ years we have several types of fuzzers for business logic. We call them: - market data feeds - orders feeds - users - developers YMMV /sarcasm ;-)
- yjftsjthsd-h 6y ago
  Although the ways that developers and users will find to abuse your systems are often extraordinary, they still pale in comparison to what a good fuzzer can do.
aaron695 6y ago
Not sure of your exact definition of fuzzing but our staging environment had real data in it. I'd always run end to end tests on random real data records. I don't see many people doing this, not sure? There seems to be kickback because tests might fail one run and pass another. People don't seem to like this.
- solarkraft 6y ago
  > tests might fail one run and pass another. People don't seem to like this I have also encountered this attitude and haven't understood it so far. What stops you from recording the test cases to reproduce failed ones?
- zxcmx 6y ago
  I think as long as it's reproducible it's Ok... but should be a team decision. The data has to be non-sensitive of course, very often it's not ok to pull data out of prod. Fuzzers in general tend to work hard to make sure crashes are reproducible (usually with a seed at the start of the run, and by saving crashing inputs). If you can't repro a crash a month or a week later, it's not worth it IMHO. Usually you can get most of the benefits with data generation, based on a deterministic seed. Mainly though, you have to ask about the purpose of the tests. Most developers want a test suite that tells them with high confidence that what they just worked on didn't create a new bug, or a regression. That lets them stay focused on their work rather than chasing through the codebase for an unrelated latent issue which might have been introduced by someone else, years ago. There is often value in separating "exploratory" or "stochastic" tests which might uncover new bugs (previously unknown) from regular tests. To make it really work as part of default workflow you need a culture which understands a feature might take additional time because the team stopped to fix a latent issue to get back to green. To put it bluntly, letting random-ish testing break your pipeline is making a statement of business priority (we care so much about random bugs that we will stop all other work until they are fixed) which might not align with reality. "Is this the most important thing for me to be working on for the success of the business?" I think the "right" way to add fuzzing / random testing to a pipeline is sell the value to the business and have an initiative to rigorously fuzz the snot out of the software. Crucially, have resources dedicated to triage and fix the identified issues. It should be a non-breaking pipeline stage right up to the point where everyone feels confident that any test failures are the result of new code. The worst case is that an opinionated developer adds stochastic tests, with bad reproducibility, without consulting the team and randomly breaks builds in an organisation that only rewards or understands feature delivery and ticket punching. That is basically going to make their co-workers life hell and is IMHO not the right way to go about it.
- arthurjj 6y ago
  I've done this by manually grabbing and anonymizing specific records and then creating unit tests. But never using a random sample of prod data. Definitely using this idea. Thanks

I find fuzzing fascinating but it seems to only be used in protocols and tools such as compilers, networking protocol implementations etc. Does anyone have experience using it for business logic?

alexpotato 6y ago

Having worked in FinTech (big bank, HFT, prop shop, start up hedge fund etc) for 10+ years we have several types of fuzzers for business logic. We call them: - market data feeds - orders feeds - users - developers YMMV /sarcasm ;-)

yjftsjthsd-h 6y ago
Although the ways that developers and users will find to abuse your systems are often extraordinary, they still pale in comparison to what a good fuzzer can do.

aaron695 6y ago

Not sure of your exact definition of fuzzing but our staging environment had real data in it. I'd always run end to end tests on random real data records. I don't see many people doing this, not sure? There seems to be kickback because tests might fail one run and pass another. People don't seem to like this.

solarkraft 6y ago
> tests might fail one run and pass another. People don't seem to like this I have also encountered this attitude and haven't understood it so far. What stops you from recording the test cases to reproduce failed ones?
zxcmx 6y ago
I think as long as it's reproducible it's Ok... but should be a team decision. The data has to be non-sensitive of course, very often it's not ok to pull data out of prod. Fuzzers in general tend to work hard to make sure crashes are reproducible (usually with a seed at the start of the run, and by saving crashing inputs). If you can't repro a crash a month or a week later, it's not worth it IMHO. Usually you can get most of the benefits with data generation, based on a deterministic seed. Mainly though, you have to ask about the purpose of the tests. Most developers want a test suite that tells them with high confidence that what they just worked on didn't create a new bug, or a regression. That lets them stay focused on their work rather than chasing through the codebase for an unrelated latent issue which might have been introduced by someone else, years ago. There is often value in separating "exploratory" or "stochastic" tests which might uncover new bugs (previously unknown) from regular tests. To make it really work as part of default workflow you need a culture which understands a feature might take additional time because the team stopped to fix a latent issue to get back to green. To put it bluntly, letting random-ish testing break your pipeline is making a statement of business priority (we care so much about random bugs that we will stop all other work until they are fixed) which might not align with reality. "Is this the most important thing for me to be working on for the success of the business?" I think the "right" way to add fuzzing / random testing to a pipeline is sell the value to the business and have an initiative to rigorously fuzz the snot out of the software. Crucially, have resources dedicated to triage and fix the identified issues. It should be a non-breaking pipeline stage right up to the point where everyone feels confident that any test failures are the result of new code. The worst case is that an opinionated developer adds stochastic tests, with bad reproducibility, without consulting the team and randomly breaks builds in an organisation that only rewards or understands feature delivery and ticket punching. That is basically going to make their co-workers life hell and is IMHO not the right way to go about it.
arthurjj 6y ago
I've done this by manually grabbing and anonymizing specific records and then creating unit tests. But never using a random sample of prod data. Definitely using this idea. Thanks