4 ms·
Putting on my machine learning PhD student hat, the way to do this was to leave 10% of the tests out as a ‘test’ set and then once the port was done, bring them
by QuadmasterXLII 3mo ago
Putting on my machine learning PhD student hat, the way to do this was to leave 10% of the tests out as a ‘test’ set and then once the port was done, bring them back in and find out how good the port is. The port may genuinely be good but because they spent 100k of compute hill climbing the whole test suite, “the test suite passes” now provides far less evidence that the port is good. Its weird that at Anthropic, a very ML phd company, no one pointed this out.
- StriverGuy 3mo agoHuh? This approach makes sense for non-deterministic problems. Not engineering problems that have deterministic end results.
- QuadmasterXLII 3mo agoI’d want to estimate P(a random test passes) where the existing suite of tests is taken to be sampled from a distribution.
- geraneum 3mo ago> non-deterministic problems I'm guessing you mean probabilistic? Nevertheless, you have an indeterministic variable here which is what the LLM generates.