4 ms·
Cool, but poorly written. Why on page 4 spend half of it on standard attention (which should be assumed knowledge to a reader at this point) but then not explai
by godelski 3y ago
Cool, but poorly written. Why on page 4 spend half of it on standard attention (which should be assumed knowledge to a reader at this point) but then not explain equation 10? what is L? Why is there an identity matrix? I don't need equations 6-9 but I sure do need more information on 10-14. I hope there's code
What a weird line in Figure 2
> Note: L_LM represents L_LM, and L_Repeat represents L_Repeat, and Loss represents L.
Tautology is not helpful here.
And what's everyone's aversion to log plots? Figure 3 is unreadable but would be perfect with log.
And where's the Appendix? It's referenced from the paper.... I'm also unconvinced it's lossless
- Solvency 3y agoThey spend that much time reiterating well-trodden established information because that's the easiest to redundantly speak to. All of the novel stuff is suspiciously vague and detail is left to the imagination. Because the author is fudging things.
- summarity 3y agoIndeed, this is also one of the worst abstracts to a paper ever. 90% of it is a generic introduction, and the rest goes out of its way to establish any quantitative results.
- godelski 3y agoThis is actually something I really hate about ML research and why I get upset when people say "don't need math." I see a lot of unnecessary equations in papers (dear god, we don't have to write attention in every ViT paper or the density function in every diffusion paper) but then no math around where it matters. It feels like a strategy to get people to just turn on autopilot, since that's what you'll naturally do. It's also weird when you get reviewers asking for this. I've had a reviewer reject a work, saying that I should include the equation for attention... in 2023... There's a reason I don't value conferences and why I tell people they aren't good signals for quality. Hell, we're doing more peer review here than most papers get. I'm not unconvinced the work isn't novel (but novelty is a function of reader's background knowledge) but I think their claims are too strong. Were I the reviewer I would reject for sure based on the unsubstantiated claim of lossless alone, without explanation. Even if a work was great and useful (though conferences make this difficult to make such types of fixes). The method does seem interesting and there is evidence for its utility, but I do think it needs a lot of additional clarity and support of the claims. In fact, I'll go ahead and say what's happening here is how I'd love to see modern peer review work, because we should be updating works and collaborating like this. Which isn't just shitting on the paper. It's constructive. Edit: Sorry, got a bit ranty. CVPR reviews are out. I'll just say I'm so glad they made a big deal about LLMs to "protect the integrity of reviews" because... LLMs were the problems....