4 ms·
I had a conversation with a coworker a few days ago where I jokingly suggested building an AI system that you give a failing test suite and it uses stackoverflo
by ajmurmann 11y ago
I had a conversation with a coworker a few days ago where I jokingly suggested building an AI system that you give a failing test suite and it uses stackoverflow answers to make your tests pass. Sounds like we are one step closer to making that happen.
- Matetricks 11y agoHave you heart of StackSort? https://gkoberger.github.io/stacksort/ https://gkoberger.github.io/stacksort/
- conjectures 11y agoThat actually sounds realistic. A well defined test suite is probably a good target for AI. Main issue is no partial wins, which are needed for training, so you'd need to break all functionality down into absolutely minimal units.
- chillacy 11y agoAlso if you overfit, your code is going to look like: int add(int a, int b) { if (a == 1 && b == 1) return 2; if (a == -1 && b == 0) return -1; if (a == 13 && b == 7) return 20; return 0; } But at least it passes my unit test suite!
- firethief 11y agoJust rerun it with a lower target code complexity. You'll know you set it too low when it starts monkey-patching the test framework to avoid actually doing anything.
- eru 11y agoThat's why you use property based testing like Python's Hypothesis or QuickCheck; instead of example based unit testing.
- Houshalter 11y agoThis can be avoided by optimizing for the simplest code that works. Checking many cases in multiple lines is always going to be longer and more complex than just a single operation on a single line. Which is how AIs avoid overfitting, in general, by penalizing complexity.