11 ms·
Ask HN: What is the best LLM for consumer grade hardware?
I have a 5060ti with 16GB VRAM. I’m looking for a model that can hold basic conversations, no physics or advanced math required. Ideally something that can run reasonably fast, near real time.
- tiahura 1y agoWhat about for a 5090?
- sgt 1y agoComes with 32GB VRAM right? Speaking of, would a Ryzen 9 12 core be nice for a 5090 setup? Or should one really go dual 5090?
- kgeist 1y agoI run Qwen3-32B with Unsloth Dynamic Quants 2.0, quantized to 4+ bits, and with the key-value cache reduced to 8-bit. It's my favorite configuration so far. This configuration has the best quality/speed ratio at this moment imho. It's pretty magical - it often feels like I'm talking to GPT-4o or o1, until it makes a silly mistake once in a while. It supports reasoning out of the box, which improves results considerably. With the settings above, I get 60 tokens per second on an RTX 5090, because it fits entirely in GPU memory. It feels faster than GPT-4o. A 32k context with 2 parallel generations* consumes 28 GB of VRAM (with llama.cpp), so you still have 4 GB left for something else. * I use 2 parallel generations because there's a few of us sharing the same GPU. If you use only 1 parallel generation, you can increase the context to 64k
- btreecat 1y agoI only have 8gb of vram to work with currently, but I'm running OpenWebUI as a frontend to ollamma and I have a very easy time loading up multiple models and letting them duke it out either at the same time or in a round robin. You can even keep track of the quality of the answers over time to help guide your choice. https://openwebui.com/ https://openwebui.com/
- rthnbgrredf 1y agoBe aware of the the recent license change of "Open"WebUI. It is no longer open source.
- lolinder 1y agoThanks, somehow I missed that. https://docs.openwebui.com/license/ https://docs.openwebui.com/license/
- nicholasjarnold 1y agoAMD 6700XT owner here (12Gb VRAM) - Can confirm. Once I figured out my local ROCm setup Ollama was able to run with GPU acceleration no problem. Connecting an OpenWebUI docker instance to my local Ollama server is as easy as a docker run command where you specify the OLLAMA_BASE_URL env var value. This isn't a production setup, but it works nicely for local usages like what the immediate parent is describing.
- benterix 1y agoI'm afraid that 1) you are not going to get a definite answer, 2) an objective answer is very hard to give, 3) you really need to try a few most recent models on your own and give them the tasks that seem most useful/meaningful to you. There is drastic difference in output quality depending on the task type.
- kouteiheika 1y agoIf you want to run LLMs locally then the localllama community is your friend: https://old.reddit.com/r/LocalLLaMA/ https://old.reddit.com/r/LocalLLaMA/ In general there's no "best" LLM model, all of them will have some strengths and weaknesses. There are a bunch of good picks; for example: > DeepSeek-R1-0528-Qwen3-8B - https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B Released today; probably the best reasoning model in 8B size. > Qwen3 - https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2... Recently released. Hybrid thinking/non-thinking models with really great performance and plethora of sizes for every hardware. The Qwen3-30B-A3B can even run on CPU with acceptable speeds. Even the tiny 0.6B one is somewhat coherent, which is crazy.
- deleted 1y ago[deleted]
- ignoramous 1y ago> DeepSeek-R1-0528-Qwen3-8B https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B ... Released today; probably the best reasoning model in 8B size. ... we distilled the chain-of-thought from DeepSeek-R1-0528 to post-train Qwen3-8B Base, obtaining DeepSeek-R1-0528-Qwen3-8B ... on AIME 2024, surpassing Qwen3-8B by +10.0% & matching the performance of Qwen3-235B-thinking. Wild how effective distillation is turning out to be. No wonder, most shops have begun to "hide" CoT now: https://news.ycombinator.com/item?id=41525201 https://news.ycombinator.com/item?id=41525201
- arresin 1y ago> Beyond its improved reasoning capabilities, this version also offers a reduced hallucination rate, enhanced support for function calling, and better experience for vibe coding. Thank you for thinking of the vibe coders.
- luke-stanley 1y ago> Released today; probably the best reasoning model in 8B size. Actually DeepSeek-R1-0528-Qwen3-8B was uploaded Thursday (yesterday) at 11 AM UTC / 7 PM CST. I had to check if a new version came out since! I am waiting for the other sizes! ;D
- srazzaque 1y agoAgree with what others have said: you need to try a few out. But I'd put Qwen3-14B on your list of things to try out.
- emson 1y agoGood question. I've had some success with Qwen2.5-Coder 14B, I did use the quantised version: huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct-GGUF:latest It worked well on my MacBook Pro M1 32Gb. It does get a bit hot on a laptop though.
- ProllyInfamous 1y agoVEGA64 (8GB) is pretty much obsolete for this AI stuff, right (compared to e.g. M2Pro (16GB))? I'll give Qwen2.5 a try on the Apple Silicon, thanks.
- deleted 1y ago[deleted]
- kekePower 1y agoI have an RTX 3070 with 8GB VRAM and for me Qwen3:30B-A3B is fast enough. It's not lightning fast, but more than adequate if you have a _little_ patience. I've found that Qwen3 is generally really good at following instructions and you can also very easily turn on or off the reasoning by adding "/no_think" in the prompt to turn it off. The reason Qwen3:30B works so well is because it's a MoE. I have tested the 14B model and it's noticeably slower because it's a dense model.
- tedivm 1y agoHow are you getting Qwen3:30B-A3B running with 8GB? On my system it takes 20GB of VRAM to launch it.
- fennecfoxy 1y agoProbably offload to regular ram I'd wager. Or really, really, reaaaaaaally quantized to absolute fuck. Qwen3:30B-A3B Q1 with a 1k Q4 context uses 5.84GB of vram.
- kekePower 1y agoIt offloads to system memory, but since there are "only" 3 Billion active parameters, it works surprisingly well. I've been able to run models that are up to 29GB in size, albeit very, very slow on my system with 32GB RAM.
- dcminter 1y agoI think you'll find that on that card most models that are approaching the 16G memory size will be more than fast enough and sufficient for chat. You're in the happy position of needing steeper requirements rather than faster hardware! :D Ollama is the easiest way to get started trying things out IMO: https://ollama.com/ https://ollama.com/
- giorgioz 1y agoI found LM Studios so much easier than ollama given it has a UI: https://lmstudio.ai/ https://lmstudio.ai/ Did you know about LM Studio? Why is ollama still recommended given it's just a CLI with worse UX?
- ekianjo 1y agolM studio is closed source
- prophesi 1y agoAny FOSS solutions that let you browse models and guesstimates for you on whether you have enough VRAM to fully load the model? That's the only selling point to LM Studio for me. Ollama's default context length is frustratingly short in the era of 100k+ context windows. My solution so far has been to boot up LM Studio to check if a model will work well on my machine, manually download the model myself through huggingface, run llama.cpp, and hook it up to open-webui. Which is less than ideal, and LM Studio's proprietary code has access to my machine specs.
- nickthegreek 1y agohttps://huggingface.co/docs/accelerate/v0.32.0/en/usage_guides/model_size_estimator https://huggingface.co/docs/accelerate/v0.32.0/en/usage_guid...
- prophesi 1y ago
- spacecadet 1y agoLook for something in the 500m-3b parameters range. 3 might push it... SmolVLM is pretty useful. https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct
- rhdunn 1y agoIt is feasible to run 7B, 8B models with q6_0 in 8GB VRAM, or q5_k_m/q4_k_m if you have to or want to free up some VRAM for other things. With q4_k_m you can run 10B and even 12B models.
- vladslav 1y agoI use Gemma3:12b on a Mac M3 Pro, basically like Grammarly.
- deleted 1y ago[deleted]
- giorgioz 1y agoTry out some models with LM Studio: https://lmstudio.ai/ https://lmstudio.ai/ It has a UI so it's very easy to download the model and have a UI similar to the chatGPT app to query that model.
- redman25 1y agoGemma-3-12b-qat https://huggingface.co/google/gemma-3-12b-it-qat-q4_0-gguf https://huggingface.co/google/gemma-3-12b-it-qat-q4_0-gguf Qwen_Qwen3-14B-IQ4_XS.gguf https://huggingface.co/bartowski/Qwen_Qwen3-14B-GGUF https://huggingface.co/bartowski/Qwen_Qwen3-14B-GGUF Gemma3 is a good conversationalist but tends to hallucinate. Qwen3 is very smart but also very stubborn (not very steerable).
- m4r71n 1y agoWhat is everyone using their local LLMs for primarily? Unless you have a beefy machine, you'll never approach the level of quality of proprietary models like Gemini or Claude, but I'm guessing these smaller models still have their use cases, just not sure what those are.
- ativzzz 1y agoI think that the future of local LLMs is delegation. You give it a prompt and it very quickly identifies what should be used to solve the prompt. Can it be solved locally with locally running MCPs? Or maybe it's a system API - like reading your calendar or checking your email. Otherwise it identifies the best cloud model and sends the prompt there. Basically Siri if it was good
- eddythompson80 1y agoI completely disagree. I don't see the current status quo fundamentally changing. That idea makes so much sense on paper, but until you start implementing it that you realized why no one does it (including Siri). "Some tasks are complex and better suited for complex giant model, but small models are perfectly capable of running simple limited task" makes a ton of sense, but the component best equipped at evaluating that decision is the smarter component of your system. At which point, you might as well have had it run the task. It's like assigning the intern to triage your work items. When actually implementing the application with that approach, every time you encounter an "AI-miss" you would (understandably) blame the small model, and eventually give up and delegate yet-another-scnario to the cloud model. Eventually you feel you're artificially handcuffing yourself compared to literally every body else trying to ship something utilizing a 1b model. You have the worst of all options, crappy model with lots of hiccups yet it's still (by far) the most resource intensive part of your application making the whole thing super heavy and you are delegating more and more to the cloud model. The local LLM scenario is going to be entirely driven by privacy concerns (around which there is no option. It's not like an E2EE LLM API could exist) or cost concerns if you believe you can run it cheaper.
- 1y ago
- jaggs 1y agohf.co/bartowski/deepseek-ai_DeepSeek-R1-0528-Qwen3-8B-GGUF:Q6_K is a decent performing model, if you're not looking for blinding speed. It definitely ticks all the boxes in terms of model quality. Try a smaller quant if you need more speed.
- cowpig 1y agoOllama[0] has a collection of models that are either already small or quantized/distilled, and come with hyperparameters that are pretty reasonable, and they make it easy to try them out. I recommend you install it and just try a bunch because they all have different "personalities", different strengths and weaknesses. My personal go-tos are: Qwen3 family from Alibaba seem to be the best reasoning models that fit on local hardware right now. Reasoning models on local hardware are annoying in contexts where you just want an immediate response, but vastly outperform non-reasoning models on things where you want the model to be less naive/foolish. Gemma3 from google is really good at intuition-oriented stuff, but with an obnoxious HR Boy Scout personality where you basically have to add "please don't add any disclaimers" to the system prompt for it to function. Like, just tell me how long you think this sprain will take to heal, I already know you are not a medical professional, jfc. Devstral from Mistral performs the best on my command line utility where I describe the command I want and it executes that for me (e.g. give me a 1-liner to list the dotfiles in this folder and all subfolders that were created in the last month). Nemo from Mistral, I have heard (but not tested) is really good for routing-type jobs, where you need something with to make a simple multiple-choice decision competently with low latency, and is easy to fine-tune if you want to get that sophisticated. [0] https://ollama.com/search https://ollama.com/search
- unethical_ban 1y agoPhi-4 is scared to talk about anything controversial, as if they're being watched. I asked it a question about militias. It thought for a few pages about the answer and whether to tell me, then came back with "I cannot comply". Nidum is the name of uncensored Gemma, it does a good job most of the time.
- emmelaich 1y agoGenerally speaking, how can you tell how much vram a model will take? It seems to be a valuable bit of data which is missing from downloadable models (gguf) files.
- omneity 1y agoVery rougly you can consider the Bs of a model as GBs of memory then it depends on the quantization level. Say for an 8B model: - FP16: 2x 8GB = 16GB - Q8: 1x 8GB - Q4: 0.5x 8GB = 4GB It doesn't 100% neatly map like this but this gives you a rough measure. In top of this you need some more memory depending on the context length and some other stuff. Rationale for the calculation above: A model is basically a billions of variables with a floating number value. So the size of a model roughly maps to number of variables (weights) x word-precision of each variable (4, 8, 16bits..) You don't have to quantize all layers to the same precision this is why sometimes you see fractional quantizations like 1.58bits.
- rhdunn 1y agoThe 1.58bit quantization is using 3 values -- -1, 0, 1. The bits number comes from log_2(3) = 1.58.... For that level you can pack 4 weights in a byte using 2 bits per byte. However, there is one bit configuration in each that is unused. More complex packing arrangements are done by grouping weights together (e.g. a group of 3) and assigning a bit configuration to each combination of values into a lookup table. This allows greater compression closer to the 1.68 bits value.
- fennecfoxy 1y agoDepends on quantization etc. But there are good calculators that will calculate for your KV cache etc as well: https://apxml.com/tools/vram-calculator https://apxml.com/tools/vram-calculator.
- binary132 1y agoI’ve had awesome results with Qwen3-30B-A3B compared to other local LMs I’ve tried. Still not crazy good but a lot better and very fast. I have 24GB of VRAM though
- mrbonner 1y agoDid someone have a chance to try local llama for the new AMD AI Max+ with 128gb of unified RAM?
- nosecreek 1y agoRelated question: what is everyone using to run a local LLM? I'm using Jan.ai and it's been okay. I also see OpenWebUI mentioned quite often.
- op00to 1y agoLMStudio, and sometimes AnythingLLM.
- Havoc 1y agoLM studio if you just want an app. openwebui is just a front end - you'd need to have either llama.cpp or vllm behind it to serve the model
- fennecfoxy 1y agoKoboldCPP + SillyTavern, has worked the best for me.
- Havoc 1y agoIt's a bit like asking what flavour of icecream is the best. Try a few and see. For 16gb and speed you could try Qwen3-30B-A3B with some offload to system ram or use a dense model Probably a 14B quant
- fennecfoxy 1y agoBasic conversations are essentially RP I suppose. You can look at KoboldCPP or SillyTavern reddit. I was trying Patricide unslop mell and some of the Qwen ones recently. Up to a point more params is better than worrying about quantization. But eventually you'll hit a compute wall with high params. KV cache quantization is awesome (I use q4 for a 32k context with a 1080ti!) and context shifting is also awesome for long conversations/stories/games. I was using ooba but found recently that KoboldCPP not only runs faster for the same model/settings but also Kobold's context shifting works much more consistently than Ooba's "streaming_llm" option, which almost always re-evaluates the prompt when hooked up to something like ST.
- sabareesh 1y agoThis is what i have https://sabareesh.com/posts/llm-rig/ https://sabareesh.com/posts/llm-rig/ All You Need is 4x 4090 GPUs to Train Your Own Model
- yapyap 1y ago4 4090s is easily 8000$, nothing to scoff at IMO
- ge96 1y agoImagine some SLI 16 x 1080
- dr_kiszonka 1y agoCould you explain what is your use case for training 1B models? Learning or perhaps fine tuning?
- sabareesh 1y agoLearning, prototype and then scale it in to cloud. Also can be used as inference engine to train another model if you are using model as a judge for RL.
- dr_kiszonka 1y agoVery neat. Thanks. Since you know more about it than I, how do the lessons learned in training small models translate to scaling them up? If that is too much to explain, would you be able to recommend something to read about it?
- arh68 1y agoWow, a 5060Ti. 16gb + I'm guessing >=32gb ram. And here I am spinning Ye Olde RX 570 4gb + 32gb. I'd like to know how many tokens you can get out of the larger models especially (using Ollama + Open WebUI on Docker Desktop, or LM Studio whatever). I'm probably not upgrading GPU this year, but I'd appreciate an anecdotal benchmark. - gemma3:12b - phi4:latest (14b) - qwen2.5:14b [I get ~3 t/s on all these small models, acceptably slow] - qwen2.5:32b [this is about my machine's limit; verrry slow, ~1 t/s] - qwen2.5:72b [beyond my machine's limit, but maybe not yours]
- reissbaker 1y agoAt 16GB a Q4 quant of Mistral Small 3.1, or Qwen3-14B at FP8, will probably serve you best. You'd be cutting it a little close on context length due to the VRAM usage... If you want longer context, a Q4 quant of Qwen3-14B will be a bit dumber than FP8 but will leave you more breathing room. Mistral Small can take images as input, and Qwen3 will be a bit better at math/coding; YMMV otherwise. Going below Q4 isn't worth it IMO. If you want significantly more context, probably drop down to a Q4 quant of Qwen3-8B rather than continuing to lobotomize the 14B. Some folks have been recommending Qwen3-30B-A3, but I think 16GB of VRAM is probably not quite enough for that: at Q4 you'd be looking at 15GB for the weights alone. Qwen3-14B should be pretty similar in practice though despite being lower in param count, since it's a dense model rather than a sparse one: dense models are generally smarter-per-param than sparse models, but somewhat slower. Your 5060 should be plenty fast enough for the 14B as long as you keep everything on-GPU and stay away from CPU offloading. Since you're on a Blackwell-generation Nvidia chip, using LLMs quantized to NVFP4 specifically will provide some speed improvements at some quality cost compared to FP8 (and will be faster than Q4 GGUF, although ~equally dumb). Ollama doesn't support NVFP4 yet, so you'd need to use vLLM (which isn't too hard, and will give better token throughput anyway). Finding pre-quantized models at NVFP4 will be more difficult since there's less-broad support, but you can use llmcompressor [1] to statically compress any FP16 LLM to NVFP4 locally — you'll probably need to use accelerate to offload params to CPU during the one-time compression process, which llmcompressor has documentation for. I wouldn't reach for this particular power tool until you've decided on an LLM already, and just want faster perf, since it's a bit more involved than just using ollama and the initial quantization process will be slow due to CPU offload during compression (albeit it's only a one-time cost). But if you land on a Q4 model, it's not a bad choice once you have a favorite. 1: https://github.com/vllm-project/llm-compressor https://github.com/vllm-project/llm-compressor
- FuriouslyAdrift 1y agoI've had good luck with GPT4All (Nomic) and either reason v1 (Qwen 2.5 - Coder 7B) or Llama 3 8B Instruct.
- antirez 1y agoThe largest Gemma 3 and Qwen 3 you can run. Offload to RAM as many layers you can.
- DiabloD3 1y agoI'd suggest buying a better GPU, only because all the models you want need a 24GB card. Nvidia... more or less robbed you. That said, Unsloth's version of Qwen3 30B, running via llama.cpp (don't waste your time with any other inference engine), with the following arguments (documented in Unsloth's docs, but sometimes hard to find): `--threads (number of threads your CPU has) --ctx-size 16384 --n-gpu-layers 99 -ot ".ffn_.*_exps.=CPU" --seed 3407 --prio 3 --temp 0.6 --min-p 0.0 --top-p 0.95 --top-k 20` along with the other arguments you need. Qwen3 30B: https://huggingface.co/unsloth/Qwen3-30B-A3B-128K-GGUF https://huggingface.co/unsloth/Qwen3-30B-A3B-128K-GGUF (since you have 16GB, grab Q3_K_XL, since it fits in vram and leaves about 3-4GB left for the other apps on your desktop and other allocations llama.cpp needs to make). Also, why 30B and not the full fat 235B? You don't have 120-240GB of VRAM. The 14B and less ones are also not what you want: more parameters are better, parameter precision is vastly less important (which is why Unsloth has their specially crafted <=2bit versions that are 85%+ as good, yet are ridiculously tiny in comparison to their originals). Full Qwen3 writeup here: https://unsloth.ai/blog/qwen3 https://unsloth.ai/blog/qwen3
- bigyabai 1y ago> only because all the models you want need a 24GB card ??? Just run a q4 quant of the same model and it will fit no problem.
- DiabloD3 1y agoQ4_K_M is the "default" for a lot of models on HF, and they generally require ~20GB of VRAM to run. It will not fit entirely on a 16GB card. You want to be about 3-4GB VRAM on top of what the model requires. A back of the envelope estimate of specifically unsloth/Qwen3-30B-A3B-128K-GGUF is 18.6GB for Q4_K_M.
- lxe 1y agoThere's a new one every day it seems. Follow https://x.com/reach_vb https://x.com/reach_vb from huggingface.
- PhilippGille 1y agoHaven't seen Mozilla's LocalScore [1] mentioned in the comments yet. It's exactly made for the purpose of finding out how well different models run on different hardware. [1] https://www.localscore.ai/ https://www.localscore.ai/
- CrossdresserSWE 1y ago[flagged]
- janalsncm 1y agoPeople ask this question a lot and annoyingly the answer is: there are many definitions of “best”. Speed, capabilities (e.g. do you want it to be able to handle images or just text?), quality, etc. It’s like asking what the best pair of shoes is. Go on Ollama and look at the most popular models. You can decide for yourself what you value. And start small, these things are GBs in size so you don’t want to wait an hour for a download only to find out a model runs at 1 token / second.
- speedgoose 1y agoMy personal preference this month is the biggest Gemma3 you can fit on your hardware.
- arnaudsm 1y agoQwen3 family (and the R1 qwen3-8b distill) is #1 in programming and reasoning. However it's heavily censored on political topics because of its Chinese origin. For world knowledge, I'd recommend Gemma3. This post will be outdated in a month. Check https://livebench.ai https://livebench.ai and https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ for up to date benchmarks
- the_sleaze_ 1y ago> This post will be outdated in a month The pace of change is mind boggling. Not only for the models but even the tools to put them to use. Routers, tools, MCP, streaming libraries, SDKs... Do you have any advice for someone who is interested, developing alone and not surrounded by coworkers or meetups who wants to be able to do discovery and stay up to date?
- nickdothutton 1y agoI find Ollama + TypingMind (or similar interface) to work well for me. As for which models, I think this is changing from one month to the next (perhaps not quite that fast). We are in that kind of period. You'll need to make sure the model layers fit in VRAM.
- nsxwolf 1y agoI'm running llama3.2 out of the box on my 2013 Mac Pro, the low end quad core Xeon one, with 64GB of RAM. It's slow-ish but still useful, getting 5-10 tokens per second.
- nurettin 1y agopretty much all Q_4 models on huggingface fit in consumer grade cards.
- discardedrefuse 1y agoCaptain Eris Violet 12B fits those requirements.
- drillsteps5 1y agoI concur LocalLLama subreddit recommendation. Not in terms of choosing "the best model" but to answer questions, find guides, latest news and gossip, names of the tools, various models and how they stack against each other, etc. There's no one "best" model, you just try a few and play with parameters and see which one fits your needs the best. Since you're on HN, I'd recommend skipping Ollama and LMStudio. They might restrict access to the latest models and you typically only choose from the ones they tested with. And besides what kind of fun is this when you don't get to peek under the hood? llamacpp can do a lot itself, and you can do most recently released models (when changes are needed they adjust literally within a few days). You can get models from huggingface obviously. I prefer GGUF format, saves me some memory (you can use lower quantization, I find most 6-bit somewhat satisfactory). I find that the the size of the model's GGUF file with roughly tell me if it'll fit in my VRAM. For example 24Gb GGUF model will NOT fit in 16Gb, whereas 12Gb likely will. However, the more context you add the more RAM will be needed. Keep in mind that models are trained with certain context window. If it has 8Kb context (like most older models do) and you load it with 32Kb context it won't be much help. You can run llamacpp on Linux, Windows, or MacOS, you can get the binaries or compile on your local. It can split the model between VRAM and RAM (if the model doesn't fit in your 16Gb). It even has simple React front-end (llamacpp-server). The same module provides REST service which has similar (but simpler) protocol to OpenAI and all the other "big" guys. Since it implements OpenAI REST API, it also works with a lot of front-end tools if you want more functionality (ie oobabooga aka textgeneration webui). Koboldcpp is another backend you can try if you find llamacpp to be too raw (I believe it's the still llamacpp under the hood).
- gavmor 1y agoWhy skip ollama? I can pull any GGUF straight from HuggingFace eg: `ollama run hf.co/unsloth/DeepSeek-R1-0528-GGUF:Q8_0`
- lolinder 1y ago> Since you're on HN, I'd recommend skipping Ollama and LMStudio. I disagree. With Ollama I can set up my desktop as an LLM server, interact with it over WiFi from any other device, and let Ollama switch seamlessly between models as I want to swap. Unless something has changed recently, with llama.cpp's CLI you still have to shut it down and restart it with a different command line flag in order to switch models even when run in server mode. That kind of overhead gets in the way of experimentation and can also limit applications: there are some little apps I've built that rely on being able to quickly swap between a 1B and an 8B or 30B model by just changing the model parameter in the web request.
- deleted 1y ago[deleted]
- GuB-42 1y agoI like the Mistral models. Not the smartest but I find them to have good conversation while being small, fast and efficient. And the part I like the most is there is almost no censorship, at least not for the models I tried. For me, having an uncensored model is one of the most compelling reasons for running a LLM locally. Jailbreaks are a PITA and abliteration and other uncensoring fine-tunings tends to make models that have been made dumb by censorship even dumber.
- spott 1y agoDoes anyone know of any local models that are capable of the same tool use (specifically web searches) during reasoning that the foundation models are? I realize they aren’t going to be as good… but the whole search during reasoning is pretty great to have.
- y2244 1y agoPick up a used 3090 with more ram. It holds it's value so you won't lose much if anything when you resell it. But otherwise, as said, install Ollama and/or Llama.cpp and run the model using the --verbose flag. This will print out the token per second result after each promt is returned. Then find the best model that gives you a token per second speed you are happy with. And as also said, 'abliterated' models are less censored versions of normal ones.
- bionhoward 1y agoThe best place to look is HuggingFace Qwen is pretty good and in a variety of sizes. I’d suggest this one Qwen/Qwen3-14B-GGUF Q4_K_M for you given your vram and to run it using llama-server or lm studio (might be alternatives to lm studio but generally these are nice uis for llama server), it’ll use around 7-8GB for weights, leaving room for incidentals Llama 3.3 could work for you Devstral is too big but could run a quantized model Gemma is good, tends to refuse a lot. Medgemma is a nice thing to have in case “Uncensored” Dolphin models from Eric Hartford and “abliterated” models are what you want if you don’t want them refusing requests, it’s mostly not necessary for routine use, but sometimes you ask em to write a joke and they won’t do it, or if you wanted to do some business which involves defense contracting or security research, that kind of thing, could be handy. Generally it’s bf16 dtype so you multiply the number of billions of parameters by two to get the model size unquantized Then to get a model that fits on your rig, generally you want a quantized model, typically I go for “Q4_K_M” which means 4bits per param, so you divide the number of billions of params by two to calculate the vram for the weights. Not sure the overhead for activations but might be a good idea to leave wiggle room and experiment with sizes well below 16GB Llama server is a good way to run AI and has a gui on the index route and -hf to download models LM Studio is a good gui and installs llama server for you and can help with managing models Make sure you run some server that loads the model once. You definitely don’t want to load many gigabytes of weights into vram every question if you want fast realtime answers
- zkmorphic 1y agothis is yummy stuff
- mollaeilaw 1y ago[dead]
- kushjoy 1y agoQuestion might sound very basic - but this is a question for the hardware folks - Are there any AI enabled embedded software tools that makes the lives of a embedded developer easier? Also, how many embedded developers would be there in large MNCs like automobile, medical devices or consumer electronics? I am trying to judge and TAM for such a startup.