3 ms·
If you are testing a system, you can't simultaneously tune the system - or at least, if you're establishing a new testing approach you can't once do. Maybe the
by joe_the_user 3y ago
If you are testing a system, you can't simultaneously tune the system - or at least, if you're establishing a new testing approach you can't once do. Maybe the testing approach is established, someone could experiment with ways to improve the system on the benchmark. But if you're establishing a benchmark, improving at the same time is going to make such a benchmark kind of meaningless.
- kromem 3y agoYou tune it first. A better faith version of this should have been the experimenters creating a creative writing pipeline they found created the best output and then testing that. Using naive zero shot prompts in a single pass with no CoT doesn't even reflect real world production usage outside of the casual user of ChatGPT who discovered it a week ago. So yes, surprise surprise - casual and naive use of a tool isn't comparable to experts without the tool. And I'm sure my using a hammer to build a deck would perform much worse than an expert carpenter without a hammer. But that's saying less about the utility of the hammer than it is my ability to use the hammer.