5 ms·

Reimplementation from paper is pretty common, though, no?

• ein0p 2y ago
  Yes, that’s why it’ll take time. There’s so much stuff competing for researchers’ attention, and experimentation with this takes so much time and $$$, that if it wasn’t for Sepp Hochreiter on the list of authors this could get ignored entirely. IOW it’s not the seller’s market for novel architectures right now.
  • htrp 2y ago
    You can't outspend the industry labs given the compute inflation in transformer architectures (unless you are ridiculously well connected in the venture/sovereign funding communities). And realistically, do we need another GPT4 evaluation paper?
    • ein0p 2y ago
      That is by far not the only thing industry labs are working on currently. I work in one. My group might be unusual, but I can’t name a single currently active project here that is not a departure from Transformers one way or another. I expect a ton of such efficiency oriented work in the next 4-5 years. We can’t be burning money as inefficiently as we do it right now.