To those still missing the point, it's easy to come up with a metric that's guaranteed to be a huge jump. Eg. imagine a big corpus of arithmetic operations (sa

To those still missing the point, it's easy to come up with a metric that's guaranteed to be a huge jump. Eg. imagine a big corpus of arithmetic operations (say 2500 additions, 2500 subtractions, 2500 multiplications and 2500 divisions): you can test when a model passes computing all of those correctly, and it's probably obvious that you'll have a long list of "didn't pass" until you suddenly get it passing.