4 ms·
> Drops compute required for training by ~20%. This is not true. Authors claim that w.r.t. training, their method adds negigible overhead for AttnRes with no m
by dvt 7mo ago
> Drops compute required for training by ~20%.
This is not true. Authors claim that w.r.t. training, their method adds negigible overhead for AttnRes with no memory impact (but is way more complicated for Block AttnRes since we need to use pipelining for larger models, hence the O(Ld) & O(Nd) figures, with N ≪ L).
> WAY lower bandwidth requirements for inference.
Also not true. Paper has nothing to do with inference, apart from the benchmarks. If you're looking at the graph about "compute advantage," it's about training compute. They do some interpolation to get to the 1.25x number, basically answering the question "if non-AttnRes architecture were trained, how much compute would it take to get to the same loss as AttnRes?" (The answer being ~20% more compute.) It's an interesting claim, but there's all kinds of weird and unexpected convergence that can happen, so take it with a grain of salt.
- observationist 7mo agoI think what they're getting at is that for a given unit of compute, this method achieves 125% performance. If model A reaches performance level 100 using 100 units of compute using old methods, and you train model B using AttnRes, aiming at performance level 100, it costs you 80 units of compute. It probably doesn't map precisely, but that's where people are diverging from the claim - it doesn't explicitly say anything about reduced inference or training time, but that's the implicit value of these sorts of things. Less compute to equivalent performance can be a huge win for platforms at scale as well as for local models.
- dvt 7mo ago> I think what they're getting at is that for a given unit of compute, this method achieves 125% performance. This is not what they're getting at; I explained exactly what they're getting at. I mean, your equivalence of "loss" (what authors actually measured) and "performance" is just bizarre. We use benchmarks to measure performance, and the numbers there were like 1-5% better (apart from the GPQA-Diamond outlier). Do people even read these papers?
- jszymborski 7mo ago> Do people even read these papers? Overwhelmingly, no. You may have mistaken this for a lab's reading group, but most people here just skim the README, maybe read the abstract or figures. Expecting them to do more is uh... a bit strange? But also you can forgive people for equating loss with performance, which are admittedly different but related ideas.