3 ms·
Llama.cpp and other “inference at the edge” tools are a really amazing pieces of engineering.
by carterschonwald 3y ago
Llama.cpp and other “inference at the edge” tools are a really amazing pieces of engineering.
- andy99 3y agoI agree of course though I also think there is/was a lot of "low hanging fruit" in inference. Pytorch is really a model training framework and pytorch/python have not traditionally been well suited to fast inference, e.g. with lacking quantization but also lacking optimized inference code. That's changing, for example this recent post: https://pytorch.org/blog/accelerating-generative-ai-2/ https://pytorch.org/blog/accelerating-generative-ai-2/ But "AI" is still very research focused (training) and not application focused (inference). Llama.cpp et al are filling in this gap.
- carterschonwald 3y agoI honestly would love to full time focus on making high performance tools for ml / applied math, but its a hard area to break into, or at least I’ve been going about it wrong.