3 ms·
The article suggested that GPF-4 FAILED between 42 and 86 percent of the time. So its success rate would be 58 to 14 percent, which compared to the novel NN's
by randcraw 3y ago
The article suggested that GPF-4 FAILED between 42 and 86 percent of the time. So its success rate would be 58 to 14 percent, which compared to the novel NN's 80 percent, seems significant.
- famouswaffles 3y agoYes I'm aware. My point is that 1. The level of variance displayed here is very atypical for a language model unless you're making significant changes to how the information is presented. This alone is cause for more elaboration. 2. With this big a variance already showing, it's hard to say for sure that they hit the top end. Maybe it seems strange but most LLM evaluation papers still don't actually bother to tie in even some extremely well known/basic "output boosters" like chain of thought and so on. But at least in these instances, we know how they presented it. a "this is how we presented the task, this is how it performed" would have been very nice.