5 ms·
Is this for educational purposes only? Based on the success of llama.cpp and this one it appears that the industry is going in a direction of separate source co
by eclectic29 3y ago
Is this for educational purposes only? Based on the success of llama.cpp and this one it appears that the industry is going in a direction of separate source code for every model that is released instead of general purpose frameworks like pytorch/tensorflow/onnxruntime?
- coder543 3y agoYes, this appears to be entirely educational. No. Despite the name, llama.cpp supports more than just llama. It also isn’t an entirely bespoke thing as you indicate, since it is built on the more general purpose “ggml” tensor library/framework.
- slimsag 3y agoI am very confused; so llama.cpp supports other non-llama models.. but is also based on the general-purpose ggml library? so llama.cpp is actually 'generic LLM framework' while ggml is 'generic ML framework'?
- coder543 3y ago> so llama.cpp is actually 'generic LLM framework' while ggml is 'generic ML framework'? That seems like a reasonable description to me, but I’m not an expert, just someone who is interested in this stuff.
- sanxiyn 3y agoYes. You can consider ggml akin to PyTorch, and llama.cpp like Transformers (by Hugging Face).
- cjbprime 3y agoYes, since it's single-threaded.
- immibis 3y agoEven in a framework there is separate source code for every model, as they are custom code based on the primitives in the framework, and not purely made using the framework. That's the nature of exploratory research. Having said that, once you find a model that works well, it tends to gets its advances incorporated into the next versions of the frameworks (so Tensorflow now has primitives like CNN, GRU and TransformerEncoder), as well as getting specific hardware implementations optimized for speed at the expense of generality (like this one).