4 ms·

(not OP but I would be surprised if the answer wasn't) yes, because you're changing the order in which the random draws are interpreted. But this isn't a proble

by tybug 2y ago

(not OP but I would be surprised if the answer wasn't) yes, because you're changing the order in which the random draws are interpreted. But this isn't a problem in practice because you generally aren't changing the generator in the middle of debugging a failure.

parentheses 2y ago
So, this means refactoring becomes potentially difficult. While the gem is still a great accomplishment and very useful, I'd have to engineer my way around this issue before using it with things like a Rails Model which could have changing shape. @OP: I wonder if the README (and possibly runner) should suggest writing a test-case that doesn't rely on PBT when the user wants to preserve a case for future testing. The issue here is that if you're saving a singular example and it represents a weird corner case, it's totally conceivable that a small change will result in an invisible change to that test case. Another idea: it'd be great if the test could simply take examples that are failing and add them to a `failing_examples.rb` or some such. I know I'd use a feature like this quite a bit.
- deleted 2y ago
  [deleted]
- ohbarye 2y ago
  I think there are usually three actions a programmer can take when PBT fails. - Create a test case that doesn't depend on PBT as you suggest. - Fix the production code being tested since its failure is an unexpected bug. - Fix the PBT itself. This means that the programmer has had wrong assumption for the test target. I think it's difficult for the tool to know which choice is the best on a failure. But if there's any good idea, I'd like to incorporate it. :)
ohbarye 2y ago
>yes, because you're changing the order in which the random draws are interpreted. But this isn't a problem in practice because you generally aren't changing the generator in the middle of debugging a failure. Correct. The test inputs are determined by a seed and generators (including the order of generators).

by tybug 2y ago

parentheses 2y ago

So, this means refactoring becomes potentially difficult. While the gem is still a great accomplishment and very useful, I'd have to engineer my way around this issue before using it with things like a Rails Model which could have changing shape. @OP: I wonder if the README (and possibly runner) should suggest writing a test-case that doesn't rely on PBT when the user wants to preserve a case for future testing. The issue here is that if you're saving a singular example and it represents a weird corner case, it's totally conceivable that a small change will result in an invisible change to that test case. Another idea: it'd be great if the test could simply take examples that are failing and add them to a `failing_examples.rb` or some such. I know I'd use a feature like this quite a bit.

deleted 2y ago
[deleted]
ohbarye 2y ago
I think there are usually three actions a programmer can take when PBT fails. - Create a test case that doesn't depend on PBT as you suggest. - Fix the production code being tested since its failure is an unexpected bug. - Fix the PBT itself. This means that the programmer has had wrong assumption for the test target. I think it's difficult for the tool to know which choice is the best on a failure. But if there's any good idea, I'd like to incorporate it. :)

ohbarye 2y ago

>yes, because you're changing the order in which the random draws are interpreted. But this isn't a problem in practice because you generally aren't changing the generator in the middle of debugging a failure. Correct. The test inputs are determined by a seed and generators (including the order of generators).