3 ms·
Amazing. But at what cost? Have there been any perplexity benchmarks for ELX2? Its annoying that Facebook made llama v2 70B instead of 65B, even with the memo
by brucethemoose2 3y ago
Amazing.
But at what cost? Have there been any perplexity benchmarks for ELX2?
Its annoying that Facebook made llama v2 70B instead of 65B, even with the memory saving changes... 5B less parameters, and the squeeze would be far easier.
- Palmik 3y agoGood point -- hopefully the quality impact is still worth it, remains to be seen. Agree on the size -- hopefully something they will keep in mind for future models.
- brucethemoose2 3y agoIf its better than the equivalent 30B model, that's still a huge achievement. Llama.cpp's Q2_K quant is 2.5625 bpw with perplexity just barely better than the next step down: https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 But subjectively, the Q2 quant "feels" worse than its high wikitext perplexity would suggest. That's apples to oranges, as this quantization is different than Q2_K, but I just hope the quality hit in practice isn't so bad.