4 ms·

Public tests are good but relying on public tests only encourages kludges to make tools pass the tests. A form of overfiting. That's why some compression tests

Public tests are good but relying on public tests only encourages kludges to make tools pass the tests. A form of overfiting. That's why some compression tests do not publish their test corpora.

arcticbull 5y ago
Do you think this applies if a corpus contains both affirmative and negative tests? As in, including not just conforming JSON but also a set of JSON that should be rejected due to non-conformance? I agree it could be challenging for instance for compression - where there is a more challenging definition of 'wrong.' I'm just wondering if this idea has legs, and appreciate your thoughts.
- marcodiego 5y ago
  I can't answer with certainty, but I think any fixed set of tests can be cheated with overfiting.
  - arcticbull 5y ago
    I do think that's a material risk - however broadly, do you think that such a scheme would make ecosystem better or worse? If you made a hypothetical JSON implementation in your language of choice, would you use it?
    - marcodiego 5y ago
      > I do think that's a material risk - however broadly, do you think that such a scheme would make ecosystem better or worse? I don't think it would be easy to cheat if: - tested implementation is open source: it would make cheating too obvious, - tests are constantly updated: it would make cheating too cumbersome and - tests include a randomization: it would not always work. So, satisfying these points would drastically increase trust on the test corpus and tested program. > If you made a hypothetical JSON implementation in your language of choice, would you use it? On my machine? I use my own hacked kernel on my machine! In production? Only if tests indicate my implementation is as good as the best ones available.
staticassertion 5y ago
That feels like an issue to be solved by the tests. If a program conforms to the public test suite the program "works" or the tests aren't covering the specification.
- johncolanduoni 5y ago
  For compression tests it's a little different, since the problem is often underdefined even for a fixed algorithm and different implementations may produce encodings with different efficiencies (space, compression time, decompression time) for different inputs. Compression implementations can overfit on some inputs and produce subpar results on average even if they produce valid outputs for all inputs. However I doubt this applies to coreutils' tests, which I suspect are more about conformance.
  - staticassertion 5y ago
    That makes sense if the public tests are "compress these bytes, expect this output" but I'd expect instead to have a lot of specific components with their own individual tests.
johncolanduoni 5y ago
But compression tests only have that issue because there are degrees of success for those tests, even for the same compression algorithm. I don't think they're hiding the test vectors because they're worried about gaming the tests by purposely failing to process valid inputs to achieve better metrics, just writing overly specific heuristics. For straightforward corner case acceptance tests (which I would assume covers most of the coreutils test suite) there's not really a danger of overfitting unless the developers are literally writing if statements that match a single input from the test and provide the correct output.

Public tests are good but relying on public tests only encourages kludges to make tools pass the tests. A form of overfiting. That's why some compression tests do not publish their test corpora.

arcticbull 5y ago

Do you think this applies if a corpus contains both affirmative and negative tests? As in, including not just conforming JSON but also a set of JSON that should be rejected due to non-conformance? I agree it could be challenging for instance for compression - where there is a more challenging definition of 'wrong.' I'm just wondering if this idea has legs, and appreciate your thoughts.

marcodiego 5y ago
I can't answer with certainty, but I think any fixed set of tests can be cheated with overfiting.
- arcticbull 5y ago
  I do think that's a material risk - however broadly, do you think that such a scheme would make ecosystem better or worse? If you made a hypothetical JSON implementation in your language of choice, would you use it?
  - marcodiego 5y ago
    > I do think that's a material risk - however broadly, do you think that such a scheme would make ecosystem better or worse? I don't think it would be easy to cheat if: - tested implementation is open source: it would make cheating too obvious, - tests are constantly updated: it would make cheating too cumbersome and - tests include a randomization: it would not always work. So, satisfying these points would drastically increase trust on the test corpus and tested program. > If you made a hypothetical JSON implementation in your language of choice, would you use it? On my machine? I use my own hacked kernel on my machine! In production? Only if tests indicate my implementation is as good as the best ones available.

staticassertion 5y ago

That feels like an issue to be solved by the tests. If a program conforms to the public test suite the program "works" or the tests aren't covering the specification.

johncolanduoni 5y ago
For compression tests it's a little different, since the problem is often underdefined even for a fixed algorithm and different implementations may produce encodings with different efficiencies (space, compression time, decompression time) for different inputs. Compression implementations can overfit on some inputs and produce subpar results on average even if they produce valid outputs for all inputs. However I doubt this applies to coreutils' tests, which I suspect are more about conformance.
- staticassertion 5y ago
  That makes sense if the public tests are "compress these bytes, expect this output" but I'd expect instead to have a lot of specific components with their own individual tests.

johncolanduoni 5y ago

But compression tests only have that issue because there are degrees of success for those tests, even for the same compression algorithm. I don't think they're hiding the test vectors because they're worried about gaming the tests by purposely failing to process valid inputs to achieve better metrics, just writing overly specific heuristics. For straightforward corner case acceptance tests (which I would assume covers most of the coreutils test suite) there's not really a danger of overfitting unless the developers are literally writing if statements that match a single input from the test and provide the correct output.