3 ms·
It’s a very barebones link, but I’m curious. Is this CPU or GPU? What are some of the salient features of this particular project besides the language?
by ComputerGuru 3y ago
It’s a very barebones link, but I’m curious. Is this CPU or GPU? What are some of the salient features of this particular project besides the language?
- molli 3y agohi, it's cpu, makes full use of simd on multiple cores and i got 3 tok/s. it's also memory mapped, so it stays in ram once it's loaded. the weights are 8 bit quantized. it can run as a server and is the backend of https://x.com/paulpapers https://x.com/paulpapers - a x bot, that reads all llm arxiv papers and shares the ones that are about running llms more efficiently.