3 ms·
That's really fast. But it mostly seems to be because they made a custom chip. I want to see an LLM that is so highly optimized that it runs at this speed on mo
by orenlindsey 3y ago
That's really fast. But it mostly seems to be because they made a custom chip.
I want to see an LLM that is so highly optimized that it runs at this speed on more normal hardware.
- stavros 3y agoBut the point is that they made a custom chip. I want to see buy their custom chip so I can have an "LLM box" in my house. I'd pay quite a bit of money to have a Mixtral box at home, then we'd all have our own, local assistant/helper/partner/whatever. Basically, the plot of the movie Her.
- agildehaus 3y agoThat'd be nice, but we could also just make this hardware normal.