3 ms·
Taalas is interesting. 16,000 TPS for Llama on a chip. https://taalas.com/ https://taalas.com/
by nl 8mo ago
Taalas is interesting. 16,000 TPS for Llama on a chip.
https://taalas.com/ https://taalas.com/
- micw 8mo agoOn a very old model, it's more like 16.000 garbage words/s
- nl 8mo agoLlama 3.1 8B is pretty useful for some thing. I use it to generate SQL pretty reliably for example. They are doing an updated model in a month or so anyway, then a frontier level one "by summer".
- numeri 8mo agobut Taalas had to quantize Llama 3.1 8B to death to get it to fit. It can't produce coherent non-English text at all.
- patapong 8mo agoI do wonder if there are tasks where 16k garbage words/s are more useful than 200 good words per second. Does anyone have any ideas? Data extraction perhaps?
- pnocera 8mo agoA politician communication agent maybe...
- DeathArrow 8mo agoI wonder how many token per seconds can they get if they put Mercury 2 on a chip.
- replete 8mo agoIts exciting to see, but look at the die size for only an 8b model
- Nihilartikel 8mo agoNeat! I had been wondering if anyone was trying to implement a model in silico. We're getting closer to having chatty talking toasters every day now!
- empath75 8mo ago"What is my purpose..." https://www.youtube.com/watch?v=sa9MpLXuLs0 https://www.youtube.com/watch?v=sa9MpLXuLs0
- Nihilartikel 8mo agoEven more on the nose: https://youtu.be/LRq_SAuQDec?si=CAe210GZ_lKcc6_Y https://youtu.be/LRq_SAuQDec?si=CAe210GZ_lKcc6_Y