3 ms·
Fwiw if they're serious scientists, taking a known method and baseline and improving it is good science. Extensions to get state of the art are probably possibl
by hiddencost 3y ago
Fwiw if they're serious scientists, taking a known method and baseline and improving it is good science. Extensions to get state of the art are probably possible, but their goal is to measure just the impact of their change in a simple setting. Let the engineers do the munged system combinations and get SoTA.
- YetAnotherNick 3y agoI am not talking about SoTA. I am talking about deliberate poor baseline. GSM8k consists of two things: solving the problem and getting the output format correct. Getting the output format corrects gives 30% accuracy for the same model where they got 11%. SoTA is 97%.