4 ms·
Don't forget about Macs. Top models can run even fairly large LLMs (although time to first token is... not great). But even midline can certainly run LLMs in th
by int_19h 1y ago
Don't forget about Macs. Top models can run even fairly large LLMs (although time to first token is... not great). But even midline can certainly run LLMs in the ballpark of ~30B params very well, which is where things start to get interesting IMO.
- dabockster 1y ago> although time to first token is... not great The token cache works on CUDA too. So yeah, the initial loadout sucks, but almost everything from then on is solid.
- int_19h 1y agoThis works for scenarios like a simple chat. But the moment you start throwing large documents on it, or lots of code, it's back to waiting.