5 ms·
I appreciate the sentiment. It definitely seems possible, but it doesn't ever seem as easy as copy + paste if you ever want performance or to go outside the sam
by swsieber 3y ago
I appreciate the sentiment. It definitely seems possible, but it doesn't ever seem as easy as copy + paste if you ever want performance or to go outside the same generic tutorial.
> Just like this:
Just is doing a lot of heavy lifting there.
Is this the first model you came across?
We're there any dependencies you had to install?
Did you have to check video card compatibility?
Where did you get the command arguments. It looks awfully complicated for a "just", as if there's a lot of options that aren't straightforward to use.
Etc
- rightbyte 3y agoYe ... with "just like this" I actually mean "proficient with compiling C projects on an Unixy system" which is like years of dev, "power user" or admin experience (not being sarcastic here). For the audience here I would say that "just" is about right though. I have no clue about anything LLM related. I just made it run after reading some comment on HN pointing in its direction. My point is that these locally run LLMs seems way "underreported". I even tried to make a "Show HN" post about it but it got zero interest. But maybe I am missing something?
- ilaksh 3y agoThe Makefile for llama.cpp is really good and doesn't require hacking to make it work.
- mitthrowaway2 3y agoEven on Windows?
- marcosdumay 3y agoIt's probably a hell to run it on Windows. But again, for the audience here, that's not expected to be a large roadblock.
- derac 3y agoIt's actually really easy to run this stuff on Windows. I've run oogabooga, koboldcpp, tabbyapi, and more with no issues.
- benlivengood 3y agoIt builds on Windows with a couple tweaks to the includes with mingw64, I think some of the Windows-related includes need to be explicitly included. This was a few months ago and things may be better now.
- ruune 3y agoWith almost every AI story I read on here (granted, I don't read that many, I find them quite boring), the top comment seems to be something like "you can do the same on your machine, see ollama". Case in point, your comment was the first one listed for me. So the interest for running LLMs on here seems quite high. Don't know why your post didn't catch anyone.
- deleted 3y ago[deleted]
- ilaksh 3y agoThe Makefile for llama.cpp is really good, it will work the way he said. You can also use ollama which makes it into a one line install and then a simple command to run any popular model.
- japanman185 3y ago[dead]
- yieldcrv 3y agocheck out LM Studio no command line involved, has a huggingface browser built in to load in the model of the day, has a chat-like interface for chatGPT like use, and can create a local server to run your local model for your programs to interact with it using an API that is identical to OpenAI's
- stavros 3y agoThis is the easiest way, by far. Then it's Ollama, then llama.cpp. But LM studio is just "install this app, select an LLM you want, done".