2 ms·
Not at all. I suggest ways to measure it here: https://news.ycombinator.com/item?id=36855881 https://news.ycombinator.com/item?id=36855881 but the TL;DR is to
by nl 3y ago
Not at all.
I suggest ways to measure it here: https://news.ycombinator.com/item?id=36855881 https://news.ycombinator.com/item?id=36855881 but the TL;DR is to choose a metric and compare the reduction in performance for quantized versions of the LM compared to the same LM without the modified Softmax.
- moffkalast 3y ago> we'd expect little difference I think that also likely holds for the quants, the difference could very well be within the error bars. Anyway, it's been posted to r/locallama so I'm sure someone will try it within the hour and report back soon :P