3 ms·
Yannic Kilcher's opinion on this is likely correct. Similar parameter count, but trained for longer. The particulars of their instruction tuning/whatever-else-t
by ShamelessC 4y ago
Yannic Kilcher's opinion on this is likely correct. Similar parameter count, but trained for longer. The particulars of their instruction tuning/whatever-else-they-did are the real secret sauce.
- redox99 4y agoDon't forget about a more efficient attention that let's them get 32k tokens of context.
- bitL 4y agoIt's still much worse than 1M context on 16GB VRAM with Reformer, but at the cost of inference speed. And you can use FlashAttention in your own models to get a more efficient/sparse attention now as well.
- meghan_rain 4y agoHow could one apply the mentioned technologies to llama/alpaca?
- Tenoke 4y agoThe quality with reformer is much much worse, it's not really comparable.
- bitL 4y agoYeah, but it fits on a single GPU. Now imagine it scaled across 1000 GPUs.
- Tenoke 4y agoI finetuned one in 2020[0] to play around with and the results still seemed a bit worse than a gpt of comparable size. 0. https://svilentodorov.xyz/blog/reformer-99m/ https://svilentodorov.xyz/blog/reformer-99m/