3 ms·
This shows the value ollama provides I only need to know the model name and then run a single command
by verdverm 3y ago
This shows the value ollama provides
I only need to know the model name and then run a single command
- eclectic29 3y agoAnd what will you do after trying it? Sure, you saved a few mins in trying out a model or models. What next?
- verdverm 3y agoI focus on building the application rather than figuring out someone else preferred method for how I should work? I use Docker Compose locally, Kubernetes in the cloud I run in hot-reload locally, I build for production I often nuke my database locally, but I run it HA in production It is very rare to use the same technology locally (or the same way) as in production
- ramblerman 3y agoRelax. Not everything in this world was built exactly for you. You almost seem to have a problem with this.
- imtringued 3y agoThere is no "next", there is a whole world of people running LLMs locally on their computer and they are far more likely to switch between models on a whim every few days.
- jameshart 3y agoIt should be fairly obvious that one can find alternative models and use them in the above command too. Look, I’m not arguing that a prebuilt binary that handles model downloading has no value over a source build and manually pulling down gguf files. I just want to dispel some of the mystery. Local LLM execution doesn’t require some mysterious voodoo that can only be done by installing and running a server runtime. It’s just something you can do by running code that loads a model file into memory and feeds tokens to it. More programmers should be looking at llama.cpp language bindings than at Ollama’s implementation of the openAI api.
- verdverm 3y agoI'd rather focus on building on top of of LLMs than going lower level Ollama makes that super easy. I tried llama.cpp first and hit build issues. Ollama worked out of the box
- jameshart 3y agoSure. Just be aware that there’s a lot of expressive difference between building on top of an HTTP API vs on top of a direct interface to the token sampler and model state.
- verdverm 3y agoI'm aware, I don't need that amount of sophistication yet. Python seems to be the way to go deeper though. Is there a good reason I should be aware of to pick llama.cpp over python?
- jameshart 3y agoPython’s as good a choice as any for the application layer. You’re either going to be using PyTorch or llama-cpp-python to get the CUDA stuff working - both rely on native compiled C/C++ code to access GPUs and manage memory at the scale needed for LLMs. I’m not actually up to speed on the current state of the game there but my understanding is that llama.cpp’s less generic approach has allowed it to focus on specifically optimizing performance of llama-style LLMs.
- verdverm 3y agoI've seen more of the model fiddling, like logits restrictions and layer dropping, implemented in python, which is why I ask Most of AI has centralized around Python, I see more of my code moving that way, like how I'm using LlamaIndex as my primary interface now, which supports ollama and many more model loaders / APIs
- roenxi 3y ago
- hnfong 3y agoThe first 3 steps GP provided are literally just the steps for installation. The "value" you mentioned is just a packaged installer (or, in the case of Linux, apparently a `curl | sh` -- and I'd much prefer the git clone version). On multiple occasions I've been modifying llama.cpp code directly and recompiling for my own purposes. If you're using ollama on the command line, I'd say having the option to easily do that is much more useful than saving a couple commands upon installation.
- verdverm 3y agoWhen I get to the point of modification, I will go with Python. This is where the AI ecosystem is largely at I stopped using C++ when Go came out, no interest in ever having to write it again.
- read_if_gay_ 3y ago>Hacker News