3 ms·
I'm aware, I don't need that amount of sophistication yet. Python seems to be the way to go deeper though. Is there a good reason I should be aware of to pick
by verdverm 3y ago
I'm aware, I don't need that amount of sophistication yet.
Python seems to be the way to go deeper though. Is there a good reason I should be aware of to pick llama.cpp over python?
- jameshart 3y agoPython’s as good a choice as any for the application layer. You’re either going to be using PyTorch or llama-cpp-python to get the CUDA stuff working - both rely on native compiled C/C++ code to access GPUs and manage memory at the scale needed for LLMs. I’m not actually up to speed on the current state of the game there but my understanding is that llama.cpp’s less generic approach has allowed it to focus on specifically optimizing performance of llama-style LLMs.
- verdverm 3y agoI've seen more of the model fiddling, like logits restrictions and layer dropping, implemented in python, which is why I ask Most of AI has centralized around Python, I see more of my code moving that way, like how I'm using LlamaIndex as my primary interface now, which supports ollama and many more model loaders / APIs