6 ms·
Llamafile is the new best way to run a LLM on your own computer
- manojlds 3y agoPrevious: https://news.ycombinator.com/item?id=38464057 https://news.ycombinator.com/item?id=38464057
- dang 3y agoThanks! Macroexpanded: Llamafile lets you distribute and run LLMs with a single file - https://news.ycombinator.com/item?id=38464057 https://news.ycombinator.com/item?id=38464057 - Nov 2023 (273 comments)
- behnamoh 3y agodupe
- bilsbie 3y agoIs there a good machine to buy in the $500-1000 range that is able to run some of this stuff?
- _neil 3y agoMaybe an M2 Mac mini?
- robterrell 3y agoDon't skimp on RAM.
- superkuh 3y agoPretty much any modern machine will do you fine for ~5 tokens/s on a 7B (small end) model like Mistral-7B or llama2-chat-7B (or any of their respective fine-tunes). The computer you already have can probably do this.
- StillBored 3y agoIt can run on CPU cores if you have enough RAM for the model. It feels like you have time warped back to the early 1990's and are talking to someone on a BBS (AKA the words appear slowly), but it is entirely functional if you have 32+GB of RAM.
- nielsole 3y agoFor reference: I run 7B 5bit quantized models on a ryzen 7 5700G with 64Gb at 8 tokens/second CPU only. It's not close to what you can get with a high end graphics card but for every day use it is alright and has headroom for bigger models. Upgrading CPU, RAM and Mainboard in a 10+year old PC cost me just 400€
- bilsbie 3y agoDo you specify the 5 bit at runtime or it’s a certain download?
- tarruda 3y agoI haven't tested, but I suspect you might be able to get good CPU inference on this mini pc: https://www.aliexpress.com/item/1005005825981362.html https://www.aliexpress.com/item/1005005825981362.html. It costs ~$800 in the maxed configuration with 64GB RAM clocked at 5600Mhz
- mratsim 3y agoHopefully, AMD wakes up and allows ROCm / HIP on their 7040 series, that would be a killer feature. And AFAIK the Ryzen pro do have an "AI coprocessor" but unsure of the API to use them.
- swuecho 3y agomac mini 8G runs at 10tokens/sec
- lxe 3y agoKeep in mind that gguf/llama.cpp, although highly performant and portable, is not the best performing way to launch certain models if you have a GPU. (even though llama.cpp does support GPU acceleration) ExLLAmA v2 + elx2 quantization, and maybe tensorrt-llm might be the contender for the top performer
- ComputerGuru 3y agoHow much more work is it to get those up and running?
- idonotknowwhy 3y agoAlmost none of you already have python. Download exl2, exui from github and run a few terminal commands. This let's me run the 120b param models, which won't fit in vram if I use llamacpp
- Rastonbury 3y agoWait, so using large models isn't limited by VRAM anymore?
- idonotknowwhy 3y agoIt is. I have 48GB of VRAM. But exl2 is more efficient, and can be quantized to partial bits. So you can run things like 4.75bpw, etc. I can run 120b models at 3bpw. The larger models like this are less affected (increased perplexity) by the quantization.
- ComputerGuru 3y agoDid you have to quantize it yourself to 4.75bpw and 3bpw or are they readily available for download?
- idonotknowwhy 3y ago
- superkuh 3y agoIt's not the best way. It's a really cool and technically interesting way. But embedding the model with the executable is terrible for anything beyond a demo. The best way would be just running llama.cpp (what llamafile uses) and loading external models. I get that compiling is too difficult for some people and llamafiles are great for them. Maybe even the best for them. But it's far from best for people that actually want to explore LLM and play.
- breckenedge 3y agoPer the article: > You can also download a much smaller llamafile binary from their releases, which can then execute any model that has been compiled to GGUF format
- dragonwriter 3y agoOr you can download LM Studio or oobabooga and execute any model compiled to GGUF format, and also models in a wide variety of formats that aren't GGUF.
- rodrigobellusci 3y agoFrom what I've read llamafile can also load other models if you pass a flag with a path to them. There's also LM Studio for anyone who'd like to play with a ChatGPT-like GUI.
- brandall10 3y agoI'd argue LM Studio is the best way by far. - Download and launch the app - Use their interface to huggingface models, download the one you want - Go to the chat window and select load - Chat away w/ a ChatGPT like interface that includes markdown processing and easy copying of code blocks, etc Outside of the time to download a model, it's about 30 seconds of work to get up and running
- deleted 3y ago[deleted]
- leonidbelyaev 3y agoDoesn't work on my NixOS workstation.
- SkyMarshal 3y agoWhy not, what’s the error, and how did you set it up? Any chance you tried it in a nix shell or nix vm or other isolated environment?
- psanford 3y agoIt fails because it hardcodes the path to certain binaries instead of looking in PATH: `line 60: /bin/mkdir: No such file or directory`
- SkyMarshal 3y agoOh lol, I've never heard of /bin/mkdir even on regular FHS linuxes. Is that actually a thing?
- FragenAntworten 3y agoSee this comment from the other discussion: https://news.ycombinator.com/item?id=38469749 https://news.ycombinator.com/item?id=38469749
- rasengan 3y agoThis is probably true for certain scenarios, but it's absolutely not true for all scenarios.
- Akashic101 3y agoI am currently planning my own small LLM trained on documents we use internally for work. Does anyone have any tips and tricks on how to make this work the best? Could a project like Llamafile help me with this, even if it is just for testing?
- bonniemuffin 3y agoHow did you choose between training a model from scratch vs using retrieval augmented generation with an existing off-the-shelf model? From what I've observed, RAG + off-the-shelf model seems to be the more common approach for use cases like "create LLM that answers questions about my company's internal documentation", particularly because the iteration/improvement cycle is much shorter-- it's much easier to iterate on RAG/prompts vs. training a whole new model to improve it. (If the answer is "I just wanted to try training a whole new llm", I won't fault you for that! :) )
- euroderf 3y agoI would like to be able to train an LLM on absolutely everything I have starred at Github.
- mud_dauber 3y agoSeconded. My copy launched without problem - now I want to learn how to ingest my library of blog posts, PDFs & images.
- youniverse 3y agoAnyone know if this (or any LLM) is worth running locally on a Macbook M2 Air? I'll give it a shot over the weekend but if anyone knows I'm curious!
- buffington 3y agoWithout knowing what you consider to be of worth, that's difficult to answer. The good news is that the article describes in detail how to determine for yourself if it's worth it. The author uses an M2 as well, so you can at least know it'll likely work.
- MPSimmons 3y agoIt depends on what you want them to do and what you want to get out of them. A lot of people use less machine than a M2 Air for different things.
- brandall10 3y agoIf you have 8GB you can play around with heavily quantized 7B models, or up to moderately quantized 13GB models w/ 16GB. Something like a Mistral-7B Dolphin finetune actually is surprisingly useful, like GPT-3.5 in some respects. I imagine it would render ~10 tok/sec on an M2 for at least short bursts until throttling sets in. As I mentioned in another post, try this out with LM Studio. Super simple GUI that even a non-tech person could probably figure out for finding/downloading/loading models w/ a ChatGPT like interface for chatting. https://lmstudio.ai/ https://lmstudio.ai/
- iJohnDoe 3y agoIs there any reason to run this if you already have LM Studio installed? Which LLM to download for general questions/info that has good accuracy? Separately, I couldn’t figure out how to use the NVIDIA GPU with LM Studio. Only wants to use the Intel card. I use NVIDIA with Diffusion with no problems if that makes any difference.
- dataveg 3y agoI have Ollama running llama 7B on an 32GB Mac M1 Pro and its fast enough to be useful.
- dragonwriter 3y agoBest in what sense? Seems by description a whole lot less convenient (and more limited) if you want to use more than one model (and especially more than just models available as GGUFs) than oobabooga/text-generation-webui or other similar tools that do things like bundle multiple backends for different LLM architectures, support downloading models from huggingface, present a common web UI for LLM configuration, managing prompts, and actually doing inference (in chat, notebook, and other styles), and also supports presenting an OpenAI-compatible API endpoint backed with a local LLM to support other frontends.
- deleted 3y ago[deleted]
- rismay 3y agoWill this approach work with image generation models?
- jawilson 3y agoIt would be nice if input could be taken from a command line argument or better yet, stdin so that it is fully scriptable. ollama has a way to do this and lets you play with a bunch of models without being very smart (just do ollama pull <model-name> and it downloads a model and makes it available to ollama run/ollama serve). (Sounds like I also need to play with LM Studio).
- maxloo1976 3y agoI've tried running Llamafile on my Lenovo Legion Pro 5 laptop with 8GB VRAM, but it has a dashboard that shows the GPU and CPU utilisation in real time, and almost all the processing is done on the CPU. Is there a way to shift the processing to the GPU like using gpt-fast? https://pytorch.org/blog/accelerating-generative-ai-2/ https://pytorch.org/blog/accelerating-generative-ai-2/