13 ms·

That generator function illustrates exactly the problem I'm talking about. The maximum length of a string in Java is 2^31-1 code points. If user is an 'arbitrar

That generator function illustrates exactly the problem I'm talking about. The maximum length of a string in Java is 2^31-1 code points. If user is an 'arbitrary string', then it could be 2^31-1 code points long. If domain is also an arbitrary string, then it can also be 2^31-1 code points long. When you concatenate them and exceed the maximum string length, you will cause a failure in the test code. There are almost always constraints within the test data, but they're complex to properly express, so they aren't specified. Then one day, the generator violates those unstated constraints, causing the test to fail.

chriswarbo 6y ago
> one day, the generator violates those unstated constraints, causing the test to fail Good, that's exactly the sort of assumption I'd like to have exposed. As a bonus, we only need to can fix this in the generators, and all the tests will benefit. I've hit exactly this sort of issue with overflow before, where I made the mistaken assumption that 'n.abs' would be non-negative. In this case Scalacheck will actually start off generating small/empty strings, and try longer and longer strings up to length 100. This is because 'arbitrary[String]' uses 'Gen.stringOf': https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Arbitrary.scala#L148 https://github.com/typelevel/scalacheck/blob/master/src/main... 'Gen.stringOf' uses the generator's current "size" parameter for the length: https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Gen.scala#L913 https://github.com/typelevel/scalacheck/blob/master/src/main... https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Gen.scala#L1254 https://github.com/typelevel/scalacheck/blob/master/src/main... The "size" of a generator starts at 'minSize' and grows to 'maxSize' as tests are performed (this ensures we check "small" values first, although generators are free to ignore the size if they like): https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Gen.scala#L275 https://github.com/typelevel/scalacheck/blob/master/src/main... We can set these manually, e.g. via a config file or commandline args, but the default is minSize = 0 https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Test.scala#L198 https://github.com/typelevel/scalacheck/blob/master/src/main... and maxSize = 100 https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Test.scala#L199 https://github.com/typelevel/scalacheck/blob/master/src/main... https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Gen.scala#L338 https://github.com/typelevel/scalacheck/blob/master/src/main...

chriswarbo 6y ago

> one day, the generator violates those unstated constraints, causing the test to fail Good, that's exactly the sort of assumption I'd like to have exposed. As a bonus, we only need to can fix this in the generators, and all the tests will benefit. I've hit exactly this sort of issue with overflow before, where I made the mistaken assumption that 'n.abs' would be non-negative. In this case Scalacheck will actually start off generating small/empty strings, and try longer and longer strings up to length 100. This is because 'arbitrary[String]' uses 'Gen.stringOf': https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Arbitrary.scala#L148 https://github.com/typelevel/scalacheck/blob/master/src/main... 'Gen.stringOf' uses the generator's current "size" parameter for the length: https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Gen.scala#L913 https://github.com/typelevel/scalacheck/blob/master/src/main... https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Gen.scala#L1254 https://github.com/typelevel/scalacheck/blob/master/src/main... The "size" of a generator starts at 'minSize' and grows to 'maxSize' as tests are performed (this ensures we check "small" values first, although generators are free to ignore the size if they like): https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Gen.scala#L275 https://github.com/typelevel/scalacheck/blob/master/src/main... We can set these manually, e.g. via a config file or commandline args, but the default is minSize = 0 https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Test.scala#L198 https://github.com/typelevel/scalacheck/blob/master/src/main... and maxSize = 100 https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Test.scala#L199 https://github.com/typelevel/scalacheck/blob/master/src/main... https://github.com/typelevel/scalacheck/blob/master/src/main/scala/org/scalacheck/Gen.scala#L338 https://github.com/typelevel/scalacheck/blob/master/src/main...