2 ms·
Do you think this applies if a corpus contains both affirmative and negative tests? As in, including not just conforming JSON but also a set of JSON that should
by arcticbull 5y ago
Do you think this applies if a corpus contains both affirmative and negative tests? As in, including not just conforming JSON but also a set of JSON that should be rejected due to non-conformance? I agree it could be challenging for instance for compression - where there is a more challenging definition of 'wrong.' I'm just wondering if this idea has legs, and appreciate your thoughts.
- marcodiego 5y agoI can't answer with certainty, but I think any fixed set of tests can be cheated with overfiting.
- arcticbull 5y agoI do think that's a material risk - however broadly, do you think that such a scheme would make ecosystem better or worse? If you made a hypothetical JSON implementation in your language of choice, would you use it?
- marcodiego 5y ago> I do think that's a material risk - however broadly, do you think that such a scheme would make ecosystem better or worse? I don't think it would be easy to cheat if: - tested implementation is open source: it would make cheating too obvious, - tests are constantly updated: it would make cheating too cumbersome and - tests include a randomization: it would not always work. So, satisfying these points would drastically increase trust on the test corpus and tested program. > If you made a hypothetical JSON implementation in your language of choice, would you use it? On my machine? I use my own hacked kernel on my machine! In production? Only if tests indicate my implementation is as good as the best ones available.