11 ms·
How I run LLMs locally
- koinedad 2y agoHelpful summary, short but useful
- drillsteps5 2y agoI did not find it useful at all. It's just a set of links to various places that can be helpful in finding useful information. Like LocalLLama subreddit is extremely helpful. You have to figure out what you want, what tools you want to use (by reading posts and asking questions), and then find some guides on setting up these tools. As you proceed you will run into issues and can easily find most of your answers in the same subreddit (or subreddits designated to the particular tools you're trying to set up) because hundreds of people had the same issues before. Not to say that the article is useless, it's just semi-organized set of links.
- emmelaich 2y agoIt should list Simon Willison's https://llm.datasette.io/en/stable/ https://llm.datasette.io/en/stable/
- jokethrowaway 2y agoI have a similar pc and I use text-generation-webui and mostly exllama quantized models. I also deploy text-generation-webui for clients on k8s with gpu for similar reasons. Last I checked, llamafile / ollama are not as optimised for gpu use. For image generation I moved from automatic webui to comfyui a few months ago - they're different beasts, for some workflow automatic is easier to use but for most tasks you can create a better workflow with enough comfy extensions. Facefusion warrants a mention for faceswapping
- Salgat 2y agoThere is a lot I want to do with LLMs locally, but it seems like we're still not quite there hardware-wise (well, within reasonable cost). For example, Llama's smaller models take upwards of 20 seconds to generate a brief response on a 4090; at that point I'd rather just use an API to a service that can generate it in a couple seconds.
- pickettd 2y agoMy gut feeling is that there may be optimization you can do for faster performance (but I could be wrong since I don't know your setup or requirements). In general on a 4090 running between Q6-Q8 quants my tokens/sec have been similar to what I see on cloud providers (for open/local models). The fastest local configuration I've tested is Exllama/TabbyAPI with speculative-decoding (and quantized cache to be able to fit more context)
- do_not_redeem 2y agoYou should absolutely be getting faster responses with a 4090. But that points to another advantage of cloud services—you don't have to debug your own driver issues.
- deleted 2y ago[deleted]
- zh3 2y agoDepends on the model, if it doesn't fit into VRAM performance will suffer. Response here is immediate (at ~15 tokens/sec) on a pair of ebay RTX 3090s in an ancient i3770 box. If your model does fit into VRAM, if its getting ejected there will be a startup pause. Try setting OLLAMA_KEEP_ALIVE to 1 (see https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-keep-a-model-loaded-in-memory-or-make-it-unload-immediately https://github.com/ollama/ollama/blob/main/docs/faq.md#how-d...).
- e12e 2y ago>> within reasonable cost > pair of ebay RTX 3090s So... 1700 USD?
- drillsteps5 2y agoIf you're looking for most bang for the buck 2x3060(12Gb) might be the best bet. GPUs will be around $400-$600.
- zh3 2y ago
- sturza 2y ago4090 has 24gb vram, not 16.
- vidyesh 2y agoThe mobile RTX 4090 has 16GB https://www.notebookcheck.net/NVIDIA-GeForce-RTX-4090-Laptop-GPU-Benchmarks-and-Specs.675091.0.html https://www.notebookcheck.net/NVIDIA-GeForce-RTX-4090-Laptop...
- deadbabe 2y agoMy understanding is that local LLMs are mostly just toys that output basic responses, and simply can’t compete with full LLMs trained with $60 million+ worth of compute time, and that no matter how good hardware gets, larger companies will always have even better hardware and resources to output even better results, so basically this is pointless for anything competitive or serious. Is this accurate?
- quest88 2y agoThat's a strange question. It sounds like you're saying the author isn't serious in their use of LLMs. Can you describe what you mean by competitive and serious? You're commenting on a post about how the author runs LLMs locally because they find them useful. Do you think they would run them and write an entire post on how they use it if they didn't find it useful? The author is seriously using them.
- prophesi 2y agoIt's the missing piece of the article. How is their experience running LLM's locally compared to GPT-4o / Sonnet 3.5 / etc, outside of data privacy?
- th0ma5 2y agoIn my experience, if you can make a reasonable guess as to the total contained information about your topic within the given size of model, then you can be very specific with your context and ask and it is generally as good as naive use of any of the larger models. Honestly, Llama3.2 is as accurate as anything and runs reasonably quick on a CPU. Larger models to me mostly increase the surface area for potential error and get exhausting to read even when you ask them to get on with it.
- momo_O 2y ago> if you can make a reasonable guess as to the total contained information about your topic within the given size of model Can you expand a bit on how you might do this?
- halyconWays 2y agoSuper basic intro but perhaps useful. Doesn't mention quant sizes, which is important when you're GPU poor. Lots of other client-side things you can do too, like KoboldAI, TavernAI, Jan, LangFuse for observability, CogVLM2 for a vision model. One of the best places to get the latest info on what people are doing with local models is /lmg/ on 4chan's /g/
- deleted 2y ago[deleted]
- pvo50555 2y agoThere was a post a few weeks back (or a reply to a post) showing an app entirely made using an LLM. It was like a 3D globe made with 3js, and I believe the poster had created it locally on his M4 MacBook with 96 GB RAM? I can't recall which model it was or what else the app did, but maybe someone knows what I'm talking about?
- jckahn 2y agoThat was with Qwen 32B, which is still the best coder model for the size. You could run that just fine with a 36 GB RAM Mac.
- pvo50555 2y agoIs it actually able to generate all the files and directory structure, etc? Or did the author just take all the responses he got from multi-shot prompting and eventually assemble it into the final product? I believe I used LlamaCoder some time ago to build the scaffolding of an app, but I'm not sure about the state of that project now.
- jckahn 2y agoI imagine _a lot_ of iteration was involved regardless of the model.
- dumbfounder 2y agoUpdation. That’s a new word for me. I like it.
- sccomps 2y agoIt's quite common in India but I guess not widely accepted internationally. If there can be deletion, then why not updation?
- exe34 2y agothat's double plus good.
- dumbfounder 2y agoWhat’s next, addition?
- sccomps 2y agoThere is already a word, Insertion!
- rspoerri 2y agoI run a pretty similar setup on an m2-max - 96gb. Just for AI image generation i would rather recommend krita with the https://github.com/Acly/krita-ai-diffusion https://github.com/Acly/krita-ai-diffusion plugin.
- donquichotte 2y agoAre you using an external provider for image generation or running something locally?
- Der_Einzige 2y agoStill nothing better than oobabooga (https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui) in terms of maximalism/"Pro"/"Prosumer" LLM UI/UX ALA Blender, Photoshop, Final Cut Pro, etc. Embarrassing and any VCs reading this can contact me to talk about how to fix that. lm-studio is today the closest competition (but not close enough) and Adobe or Microsoft could do it if they fired their current folks which prevent this from happening. If you're not using Oobabooga, you're likely not playing with the settings on models, and if you're not playing with your models settings, you're hardly even scratching the surface on its total capabilities.
- theropost 2y agothis.
- skarlso 2y agoHi, I'm a noob in this plane, but I'm learning slowly. Could you please tell me how this is different from running something like [Msty](https://msty.app/ https://msty.app/)? That also supports running local models that you already installed using ollama and things like, adding pdf and preserving chat history. So what am I looking at here with ooba? Thanks!
- Der_Einzige 2y agoOobabooga has a focus on and support for dozens more settings and parameters than any other UI does. You only get more by using base HF codebase via code. Mysty is like obsidian in that it’s not a true prosumer or pro level tool. This is not photoshop for text. It’s Lightroom for text.
- skarlso 2y agoThanks for the clarification! I see. I guess, I have to decide where and to what extend I would like to level my skills. And that where do I start. Not going to lie, getting Oobabooga to work on my laptop was no small feet. Conda didn't work, python modules didn't work. cmd_macos failed with an error... But eventually I got there by hand. :D So the "Getting Started" was quite steep.
- dividefuel 2y agoWhat GPU offers a good balance between cost and performance for running LLMs locally? I'd like to do more experimenting, and am due for a GPU upgrade from my 1080 anyway, but would like to spend less than $1600...
- redacted 2y agoNvidia for compatibility, and as much VRAM as you can afford. Shouldn't be hard to find a 3090 / Ti in your price range. I have had decent success with a base 3080 but the 10GB really limits the models you can run
- yk 2y agoSecond the other comment, as much vram as possible. A 3060 has 12 GB at a reasonable price point. (And is not too limiting.)
- grobbyy 2y agoThere's a huge step to I'm capability with 16gb and 24gb, for not to much more. The 4060 has a 16gb version, for example. On the the cheap end, the Intel Arc does too. Next major step up is 48GB and then hundreds of GB. But a lot of ML models target 16-24gb since that's in the grad student price range.
- navbaker 2y agoAt the 48GB level, L40S are great cards and very cost effective. If you aren’t aiming for constant uptime on several >70B models at once, they’re for sure the way to go!
- nickthegreek 2y agoDual 3090s are way cheaper than the l40s though. You can even buy a few backups.
- navbaker 2y ago
- deleted 2y ago[deleted]
- throwaway314155 2y agoOpen WebUI sure does pull in a lot of dependencies... Do I really need all of langchain, pytorch, and plenty others for what is advertised as a _frontend_? Does anyone know of a lighter/minimalist version?
- noman-land 2y agohttps://llamafile.ai/ https://llamafile.ai/
- throwaway314155 2y agoI love what llamafile is doing, but I'm primarily interested in a frontend for ollama, as I prefer their method of model/weights distribution. Unless I'm wrong, llamafile serves as both the frontend and backend.
- DrBenCarson 2y agoLM Studio?
- d3VwsX 2y agoIf I understand the distinction correctly, I run llamafile as a backend. I start it with the filename of a model on the command-line (might need a -M flag or something) and it will start up a chat-prompt for interaction in the terminal but also opens a port that speaks some protocol that I can connect to using a frontend (in my case usually gptel in emacs).
- figmert 2y agoI ran it temporarily in Docker the other day. It was a 8gb image. I'm unsure why a webui is 8gb.
- lolinder 2y agoSome of the features (RAG retrieval) now use embeddings that are calculated in Open WebUI rather than in Ollama or another backend. It does seem like it'd be nice for them to refactor to make things like that optional for those who want a simpler interface, but then again, there are plenty of other lighter-weight options.
- gulan28 2y agoYou can try out https://wiz.chat https://wiz.chat (my project) if you want to Run llama on your web browser. Still needs a GPU and the latest version of chrome but it's fast enough for my usage.
- amazingamazing 2y agoI never have seen the point of running locally. Not cost effective, worse model, etc.
- ogogmad 2y agoPrivacy?
- splintercell 2y agoEven if it was not cost-effective, or you're just running worse models, it's learning an important skill. Take for instance, self-hosting your website may have all these considerations, but you're getting information from the LLMs. It would be helpful to know that the LLM is in your control.
- Oras 2y agoSelf hosting website as local server running in your room? what’s the point? Same with LLMs, you can use providers who don’t log requests and SOC2 compliant. Small models that run locally is a waste of time as they don’t have adequate value compared to larger models.
- troyvit 2y ago> Self hosting website as local server running in your room? what’s the point? It depends on how important the web site is and what the purpose is. Personal blog (Ghost, WordPress)? File sharing (Nextcloud)? Document collaboration (Etherpad)? Media (Jellyfin)? Why _not_ run it from your room with a reverse proxy? You're paying for internet anyway. > Same with LLMs, you can use providers who don’t log requests and SOC2 compliant. Sure. Until they change their minds and decide they want to, or until they go belly-up because they didn't monetize you. > Small models that run locally is a waste of time as they don’t have adequate value compared to larger models. A small model won't get you GPT-4o value, but for coding, simple image generation, story prompts, "how long should I boil an egg" questions, etc. it'll do just fine, and it's yours. As a bonus, while a lot of energy went into creating the models you'd use, you're saving a lot of energy using them compared to asking the giant models simple questions.
- thangalin 2y agorun.sh: #!/usr/bin/env bash set -eu set -o errexit set -o nounset set -o pipefail readonly SCRIPT_SRC="$(dirname "${BASH_SOURCE[${#BASH_SOURCE[@]} - 1]}")" readonly SCRIPT_DIR="$(cd "${SCRIPT_SRC}" >/dev/null 2>&1 && pwd)" readonly SCRIPT_NAME=$(basename "$0") # Avoid issues when wine is installed. sudo su -c 'echo 0 > /proc/sys/fs/binfmt_misc/status' # Graceful exit to perform any clean up, if needed. trap terminate INT # Exits the script with a given error level. function terminate() { level=10 if [ $# -ge 1 ] && [ -n "$1" ]; then level="$1"; fi exit $level } # Concatenates multiple files. join() { local -r prefix="$1" local -r content="$2" local -r suffix="$3" printf "%s%s%s" "$(cat ${prefix})" "$(cat ${content})" "$(cat ${suffix})" } # Swapping this symbolic link allows swapping the LLM without script changes. readonly LINK_MODEL="${SCRIPT_DIR}/llm.gguf" # Dereference the model's symbolic link to its path relative to the script. readonly PATH_MODEL="$(realpath --relative-to="${SCRIPT_DIR}" "${LINK_MODEL}")" # Extract the file name for the model. readonly FILE_MODEL=$(basename "${PATH_MODEL}") # Look up the prompt format based on the model being used. readonly PROMPT_FORMAT=$(grep -m1 ${FILE_MODEL} map.txt | sed 's/.*: //') # Guard against missing prompt templates. if [ -z "${PROMPT_FORMAT}" ]; then echo "Add prompt template for '${FILE_MODEL}'." terminate 11 fi readonly FILE_MODEL_NAME=$(basename $FILE_MODEL) if [ -z "${1:-}" ]; then # Write the output to a name corresponding to the model being used. PATH_OUTPUT="output/${FILE_MODEL_NAME%.*}.txt" else PATH_OUTPUT="$1" fi # The system file defines the parameters of the interaction. readonly PATH_PROMPT_SYSTEM="system.txt" # The user file prompts the model as to what we want to generate. readonly PATH_PROMPT_USER="user.txt" readonly PATH_PREFIX_SYSTEM="templates/${PROMPT_FORMAT}/prefix-system.txt" readonly PATH_PREFIX_USER="templates/${PROMPT_FORMAT}/prefix-user.txt" readonly PATH_PREFIX_ASSIST="templates/${PROMPT_FORMAT}/prefix-assistant.txt" readonly PATH_SUFFIX_SYSTEM="templates/${PROMPT_FORMAT}/suffix-system.txt" readonly PATH_SUFFIX_USER="templates/${PROMPT_FORMAT}/suffix-user.txt" readonly PATH_SUFFIX_ASSIST="templates/${PROMPT_FORMAT}/suffix-assistant.txt" echo "Running: ${PATH_MODEL}" echo "Reading: ${PATH_PREFIX_SYSTEM}" echo "Reading: ${PATH_PREFIX_USER}" echo "Reading: ${PATH_PREFIX_ASSIST}" echo "Writing: ${PATH_OUTPUT}" # Capture the entirety of the instructions to obtain the input length. readonly INSTRUCT=$( join ${PATH_PREFIX_SYSTEM} ${PATH_PROMPT_SYSTEM} ${PATH_PREFIX_SYSTEM} join ${PATH_SUFFIX_USER} ${PATH_PROMPT_USER} ${PATH_SUFFIX_USER} join ${PATH_SUFFIX_ASSIST} "/dev/null" ${PATH_SUFFIX_ASSIST} ) ( echo ${INSTRUCT} ) | ./llamafile \ -m "${LINK_MODEL}" \ -e \ -f /dev/stdin \ -n 1000 \ -c ${#INSTRUCT} \ --repeat-penalty 1.0 \ --temp 0.3 \ --silent-prompt > ${PATH_OUTPUT} #--log-disable \ echo "Outputs: ${PATH_OUTPUT}" terminate 0 map.txt: c4ai-command-r-plus-q4.gguf: cmdr dare-34b-200k-q6.gguf: orca-vicuna gemma-2-27b-q4.gguf: gemma gemma-2-7b-q5.gguf: gemma gemma-2-Ifable-9B.Q5_K_M.gguf: gemma llama-3-64k-q4.gguf: llama3 llama-3-64k-q4.gguf: llama3 llama-3-1048k-q4.gguf: llama3 llama-3-1048k-q8.gguf: llama3 llama-3-8b-q4.gguf: llama3 llama-3-8b-q8.gguf: llama3 llama-3-8b-1048k-q6.gguf: llama3 llama-3-70b-q4.gguf: llama3 llama-3-70b-64k-q4.gguf: llama3 llama-3-smaug-70b-q4.gguf: llama3 llama-3-giraffe-128k-q4.gguf: llama3 lzlv-q4.gguf: alpaca mistral-nemo-12b-q4.gguf: mistral openorca-q4.gguf: chatml openorca-q8.gguf: chatml quill-72b-q4.gguf: none qwen2-72b-q4.gguf: none tess-yi-q4.gguf: vicuna tess-yi-q8.gguf: vicuna tess-yarn-q4.gguf: vicuna tess-yarn-q8.gguf: vicuna wizard-q4.gguf: vicuna-short wizard-q8.gguf: vicuna-short Templates (all the template directories contain the same set of file names, but differ in content): templates/ ├── alpaca ├── chatml ├── cmdr ├── gemma ├── llama3 ├── mistral ├── none ├── orca-vicuna ├── vicuna └── vicuna-short ├── prefix-assistant.txt ├── prefix-system.txt ├── prefix-user.txt ├── suffix-assistant.txt ├── suffix-system.txt └── suffix-user.txt If there's interest, I'll make a repo.
- ashleyn 2y agoanyone got a guide on setting up and running the business-class stuff (70B models over multiple A100, etc)? i'd be willing to spend the money but only if i could get a good guide on how to set everything up, what hardware goes with what motherboard/ram/cpu, etc.
- foundry27 2y agoAye, there’s the kicker. The correct configuration of hardware resources to run and multiplex large models is just as much of a trade secret as model weights themselves when it comes to non-hobbyist usage, and I wouldn’t be surprised if optimal setups are in many ways deliberately obfuscated or hidden to keep a competitive advantage Edit: outside the HPC community specifically, I mean
- deleted 2y ago[deleted]
- codybontecou 2y agoThe economic barrier to entry probably has a lot to do with it. I'd happily dig into this problem and share my findings but it's simply too expensive for a hobbyist that isn't specialized in it.
- dkkergoog 2y ago[dead]
- talldayo 2y agoI don't think you're going to find much of a guide out there because there isn't really a need for one. You just need a Linux client with the Nvidia drivers installed and some form of CUDA runtime present. You could make that happen in a mini PC, a jailbroken Nintendo Switch, a gaming laptop or a 3600W 4U rackmount. The "happy path" is complicated because you truly have so many functional options. You don't want an A100 unless you've already got datacenter provisioning at your house and an empty 1U rack. I genuinely cannot stress this enough - these are datacenter cards for a reason. The best bang-for-your buck will be consumer-grade cards like the 3060 and 3090, as well as the bigger devkits like the Jetson Orin.
- chown 2y agoIf anyone is looking for a one click solution without having to have a Docker running, try Msty - something that I have been working on for almost a year. Has RAG and Web Search built in among others and can connect to your Obsidian vaults as well. https://msty.app https://msty.app
- rgovostes 2y agoYour top menu has an "As" button, listing how you compare to alternative products. Is that the label you wanted?
- amrrs 2y agoHow is it different from LMstudio or GPT4ALl?
- ein0p 2y agoInteresting that you show Mistral Large in your screenshots. That model is only licensed for "research" use. While it's not terribly clear how "research" is defined, selling a product with it featured is clearly not "research".
- navbaker 2y agoThey’re not selling Mistral, they’re selling a product that let’s you use Mistral
- deleted 2y ago[deleted]
- upghost 2y ago> Before I begin I would like to credit the thousands or millions of unknown artists, coders and writers upon whose work the Large Language Models(LLMs) are trained, often without due credit or compensation I like this. If we insist on pushing forward with GenAI we should probably at least make some digital or physical monument like "The Tomb of the Unknown Creator". Cause they sure as sh*t ain't gettin paid. RIP.
- model-15-DAV 2y agoThis seems as useless as land-acknowledgements; "Hey look we took your stuff and are not paying you for it, and we are still profiting from it!!"
- 8note 2y agoboth this and land acknowledgements are making sure people are keeping those people in mind. that makes it harder to keep justifying screwing over those people.
- mvdtnz 2y agoI've seen lots of land acknowledgements from people who own land. Never once seen one put their money where their mouth is.
- deleted 2y ago[deleted]
- qntmfred 2y agoI was once a top 1% stack overflow user. I didn't start contributing to make money, just to share information and learn from each other. that can and should also be good enough for the AI era.
- o11c 2y agoThere's a difference between "I'm willing to exert myself to help people" and "I'm willing to exert myself so that somebody else can slap their name on my effort and make money, then deny me the right to do the same to them". Also, at a social level - the worst kind of user has always been a help vampire, and LLMs are really good at increasing the number of help vampires.
- farceSpherule 2y agoI pay for ChatGPT Teams. Much easier and better than this.
- erickguan 2y agoHow much memory can models take? I would assume the dGPU set up won't perform better at a certain point.
- coding123 2y agoWe will at some point have a JS API to run preliminary LLM to make local decisions, then the server will be final arbiter. So for example a comment rage moderator can help an end user change their proposed post while they write it, to help them not turn the comment into rage bate. This will be done best locally on the users browser. Then when they are ready to post, one final check by the server would be done. This would be like today's React front ends doing all the state and UI computation, reducing servers from having to render HTML, for example.
- prettyblocks 2y ago> I have a laptop running Linux with core i9 (32threads) CPU, 4090 GPU (16GB VRAM) and 96 GB of RAM. Is there somewhere I can find a computer like this pre-built?
- hasperdi 2y agoYes you can buy something like this prebuilt. Lenovo workstations for example. Quite expensive new, but there are many second hands on eBay
- chb 2y agoI’m surprised to see no mention of AnythingLLM (https://github.com/Mintplex-Labs/anything-llm https://github.com/Mintplex-Labs/anything-llm). I use it with an Anthropic API key, but am giving thought to extending it with local LLM. It’s a great app: good file management for RAG, agents with web search, cross platform desktop client, but can also easily be run as a server using docker compose. Nb: if you’re still paying $20/mo for a feature-poor chat experience that’s locked to a single provider, you should consider using any of the many wonderful chat clients that take a variety of API keys instead. You might find that your LLM utilization doesn’t quite fit a flat rate model, and that the feature set of the third-party client is comparable (or surpasses) that of the LLM provider’s. edit: included repo link; note on API keys as alternative to subscription
- doodlesdev 2y agoHow does it compare to OpenWebUI? It seems pretty solid, no ideia how I never heard of it before!
- stogot 2y agoHaven’t heard of this. What does it provide that the anthropoic UI does not?
- alfredgg 2y agoTotally agree! Other useful alternative: Lobe Chat (https://github.com/lobehub/lobe-chat https://github.com/lobehub/lobe-chat)
- masteruvpuppetz 2y agoDavid Bombal interviews a mysterious man where he shows how he uses AI/LLMs for his automated LinkedIn posts and other tasks. https://www.youtube.com/watch?v=vF-MQmVxnCs https://www.youtube.com/watch?v=vF-MQmVxnCs
- mikestaub 2y agoI just use MLC with WebGPU: https://codepen.io/mikestaub/pen/WNqpNGg https://codepen.io/mikestaub/pen/WNqpNGg
- rcdwealth 2y ago[dead]
- novok 2y agoAs a piece of writing feedback, I would convert your citation links into normal links. Clicking on the citation doesn't jump to the link or the citation entry, and you are basically using hyperlinks anyway.