3 ms·

A program like function add(a,b) { return 4 } Passes the test

• falcor84 2y ago
  I suppose you're alluding to xkcd's joke about this [0], which is indeed a good one, but what test does this actually pass? The approach I was thinking of is that assuming we start with the Java program: public class Addition { public static int add(int a, int b) { return a + b; } } We can semi-automatically generate a basic test runner with something like this, generating some example inputs automatically: public class Addition { public static int add(int a, int b) { return a + b; } public static class AdditionAssert { private int a; private int b; public AdditionAssert a(int a) { this.a = a; return this; } public AdditionAssert b(int b) { this.b = b; return this; } public void assertExpected(int expected) { int result = add(a, b); assert result == expected : "Expected " + expected + " but got " + result; System.out.println("Assertion passed for " + a + " + " + b + " = " + result); } } public static void main(String[] args) { new AdditionAssert().a(5).b(3).assertExpected(8); new AdditionAssert().a(-1).b(4).assertExpected(3); new AdditionAssert().a(0).b(0).assertExpected(0); System.out.println("All test cases passed."); } } Another bit of automated preparation would then automatically translate the test cases to Python, and then the actual LLM would need to generate a python function until it passes all the translated test cases: def add(a, b): return 4 def addition_assert(a, b, expected): result = add(a, b) assert result == expected, f"Expected {expected} but got {result}" addition_assert(a=5, b=3, expected=8) addition_assert(a=-1, b=4, expected=3) addition_assert(a=0, b=0, expected=0) It might not be perfect, but I think it's very feasible and can get us close to there. [0] https://xkcd.com/221/ https://xkcd.com/221/