5 ms·
Does anyone know what the units are on the "performance improvement over SOTA" chart?
by lukasb 5y ago
Does anyone know what the units are on the "performance improvement over SOTA" chart?
- lukasb 5y agoTurns out it's a composite of "normalized task-specific metrics", details in the paper. Shrug. Numbers go up!
- r-zip 5y agoI was wondering the same. Without better y-axis labeling, it's not that informative of a graphic.
- whymauri 5y agoPoetic that the top post right now is (partially) about how science communication over-simplifying figures results in a popular misunderstanding of science, leading readers to believe that conducting research is easier than it actually is.
- EvgeniyZh 5y agoIf you're talking about fig. 4, then it's some units scaled so that random performance is 0 and perfect performance is 100 (depending on task it may be accuracy or something else). Since the models are so large, good benchmarks are diverse, and different tasks require different metrics.