4 ms·

It should be fairly obvious that one can find alternative models and use them in the above command too. Look, I’m not arguing that a prebuilt binary that handl

It should be fairly obvious that one can find alternative models and use them in the above command too. Look, I’m not arguing that a prebuilt binary that handles model downloading has no value over a source build and manually pulling down gguf files. I just want to dispel some of the mystery. Local LLM execution doesn’t require some mysterious voodoo that can only be done by installing and running a server runtime. It’s just something you can do by running code that loads a model file into memory and feeds tokens to it. More programmers should be looking at llama.cpp language bindings than at Ollama’s implementation of the openAI api.

verdverm 3y ago
I'd rather focus on building on top of of LLMs than going lower level Ollama makes that super easy. I tried llama.cpp first and hit build issues. Ollama worked out of the box
- jameshart 3y ago
  Sure. Just be aware that there’s a lot of expressive difference between building on top of an HTTP API vs on top of a direct interface to the token sampler and model state.
  - verdverm 3y ago
    I'm aware, I don't need that amount of sophistication yet. Python seems to be the way to go deeper though. Is there a good reason I should be aware of to pick llama.cpp over python?
    - jameshart 3y ago
      Python’s as good a choice as any for the application layer. You’re either going to be using PyTorch or llama-cpp-python to get the CUDA stuff working - both rely on native compiled C/C++ code to access GPUs and manage memory at the scale needed for LLMs. I’m not actually up to speed on the current state of the game there but my understanding is that llama.cpp’s less generic approach has allowed it to focus on specifically optimizing performance of llama-style LLMs.
      - verdverm 3y ago
        I've seen more of the model fiddling, like logits restrictions and layer dropping, implemented in python, which is why I ask Most of AI has centralized around Python, I see more of my code moving that way, like how I'm using LlamaIndex as my primary interface now, which supports ollama and many more model loaders / APIs
roenxi 3y ago
There are 5 commands in that README two comments up, 4 can reasonably fail (I'll give cd high marks for reliability). `make` especially is a minefield and usually involves a half-hour of searching the internet and figuring out which dependencies are a problem today. And that is all assuming someone is comfortable with compiled languages. I'd hazard most devs these days are from JS land and don't know how to debug make. Finding the correct model weights is also a challenge in my experience, there are a lot of alternatives and it is often difficult to figure out what the differences are and whether they matter. The README is clear that I'm probably about to lose an hour debugging if I follow it. It might be one of those rare cases where it works first time but that is the exception not the rule.
- jameshart 3y ago
  Your mileage may vary. It runs first time for me on an Apple Silicon Mac.

verdverm 3y ago

I'd rather focus on building on top of of LLMs than going lower level Ollama makes that super easy. I tried llama.cpp first and hit build issues. Ollama worked out of the box

jameshart 3y ago
Sure. Just be aware that there’s a lot of expressive difference between building on top of an HTTP API vs on top of a direct interface to the token sampler and model state.
- verdverm 3y ago
  I'm aware, I don't need that amount of sophistication yet. Python seems to be the way to go deeper though. Is there a good reason I should be aware of to pick llama.cpp over python?
  - jameshart 3y ago
    Python’s as good a choice as any for the application layer. You’re either going to be using PyTorch or llama-cpp-python to get the CUDA stuff working - both rely on native compiled C/C++ code to access GPUs and manage memory at the scale needed for LLMs. I’m not actually up to speed on the current state of the game there but my understanding is that llama.cpp’s less generic approach has allowed it to focus on specifically optimizing performance of llama-style LLMs.
    - verdverm 3y ago
      I've seen more of the model fiddling, like logits restrictions and layer dropping, implemented in python, which is why I ask Most of AI has centralized around Python, I see more of my code moving that way, like how I'm using LlamaIndex as my primary interface now, which supports ollama and many more model loaders / APIs

roenxi 3y ago

There are 5 commands in that README two comments up, 4 can reasonably fail (I'll give cd high marks for reliability). `make` especially is a minefield and usually involves a half-hour of searching the internet and figuring out which dependencies are a problem today. And that is all assuming someone is comfortable with compiled languages. I'd hazard most devs these days are from JS land and don't know how to debug make. Finding the correct model weights is also a challenge in my experience, there are a lot of alternatives and it is often difficult to figure out what the differences are and whether they matter. The README is clear that I'm probably about to lose an hour debugging if I follow it. It might be one of those rare cases where it works first time but that is the exception not the rule.

jameshart 3y ago
Your mileage may vary. It runs first time for me on an Apple Silicon Mac.