4 ms·
I got a quantized Llama 70B model working, using most of my 64GB of RAM but it's usable: ollama run hf.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF:Q3_K_M
by simonw 2y ago
I got a quantized Llama 70B model working, using most of my 64GB of RAM but it's usable:
ollama run hf.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF:Q3_K_M
That's a 34GB download. I'm accessing it via https://github.com/open-webui/open-webui https://github.com/open-webui/open-webui which I ran like this:
uvx --python 3.11 open-webui serve
I have TailScale on my laptop and phone so I can run experiments directly from my phone while leaving my laptop plugged in at home.
- jt_b 2y agoThanks for all your writing on these topics Simon! It has turned me from a bit of a naysayer to an optimist around this tooling, especially being able running stuff locally with access to tools. Have an M4 Pro mac mini arriving this week to have a similar self hosted setup over tailscale.
- brewtide 2y agoMaybe related, but I've also got a tailscale instance running for the same use case (on an older it box but...) I've also installed open-webui attached to ollama. That way the interface I can deal with when on my phone is just a simple nice to use webpage. May want to look into this? Thus far it's worked very slick.
- jt_b 2y agoNice, thanks for the suggestion. I got it set up just before leaving town for a few days, so have been doing a little tinkering with it. I was hoping to have a setup with LM Studio, where my laptop could use the API Server from the mini over the TS network. Unfortunately doesn't seem to be the case, so I'll set up a configuration like you mentioned to just have a global client from any device on the netowrk. It's very cool to be able to have access to such a high horsepower machine from anywhere though. Next step is figuring out the networking interface to be able access the host GPU/ollama API from pods running in a Colima VM/k3s cluster setup.
- matt_daemon 2y ago> Here's what I got - the joke itself is rubbish but the "thinking" section is fascinating Very funny. How does the 70B model do in comparison?
- pmarreck 2y agoTime to put this 128GB M4 Mac to work... bash> ollama run hf.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF:Q5_K_M (49GB!) (It wouldn't accept the Q_8 because it was sharded.) Will report back lol What app do you use on your phone to access your home LLM servers over tailscale?
- liamwire 2y agoNot to speak for him, but I’d suspect Simon would be using a terminal emulator. He often mentions preferring to use his own command line tool, named LLM, as it allows one to log everything to an SQLite database for later exploration.
- pmarreck 2y agoHah, I probably have a similar tool just called "ask" which logs to a JSON log instead of a sqlite db
- pmarreck 2y agobash> ollama run hf.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF:Q5_K_M pulling manifest ... success >>> Tell me a "what's the difference between" joke about techbros and another funny comparison of your choosing <think> Okay, so I need to come up with a "What's the difference between" joke comparing techbros and another group. Hmm. Let me think. First, what defines a techbro? They're usually associated with startups, tech gadgets, maybe working in Silicon Valley, into things like crypto, NFTs, and they often have a certain lingo or way of dressing—like hoodies, jeans, maybe beanies. They might be seen as overly optimistic about their ideas changing the world but sometimes disconnected from practical realities. Now, I need another group to compare them with. Maybe something else in tech but different enough to create humor. Let's think: developers? No, too similar. Maybe hipsters? Or perhaps venture capitalists? Wait, what about "developers who actually ship code"? That could work because it plays on the idea that while techbros talk a lot about their ideas, developers are more focused on delivering actual products. So the joke structure is usually two lines: one for each group, highlighting a key difference. It needs to be punchy and capture the essence of both groups in a humorous way. Let me draft it: "What's the difference between a techbro and a developer who actually ships code?" "Techbros are still explaining their MVP to you three years later, while developers just sent you a pull request titled 'it works now'." Wait, does that make sense? Let me break it down. A Minimum Viable Product (MVP) is something techbros often talk about but might take forever to actually build or ship. On the other hand, developers who are focused on shipping code would just send a pull request indicating they've fixed something and it's done now. I think that captures the difference between all talk and actual action. It's relatable for people in the tech industry who know the dynamics between idea people and implementers. Is there another angle? Maybe comparing them to something else, like entrepreneurs or investors, but I think the developer angle is solid because it directly contrasts talking about ideas versus executing them. So the final answer would be that joke. It highlights the techbro's tendency to over-explain without delivery, while developers focus on getting things done. </think> **What's the difference between a techbro and a developer who actually ships code?** *Techbros are still explaining their MVP to you three years later, while developers just sent you a pull request titled "it works now."* This joke humorously contrasts the tendency of techbros to focus on ideas and discussions with the action-oriented approach of developers who deliver tangible results.
- jondwillis 2y agoI set up a cloudflare tunnel recently so I could also use my local LLMs remotely! It’s nice.
- regularfry 2y agoI've previously had a 70B Q2 wedged into a 4090. Not much space for context but it runs...
- unstatusthequo 2y agoYes, I did the same but with Docker serving open-webui. It's definitely not fast. Took 2 minutes to answer "what is the highest peak in California," but it eventually go there. OpenAI o1 took 6 seconds. Pasting the logic for anyone interested: <details type="reasoning" done="true" duration="102"> <summary>Thought for 102 seconds</summary> > Okay, so I need to figure out what the tallest peak in California is. Hmm, I remember that the highest mountain in the contiguous United States is Mount Whitney, but wait, isn't that in California? Let me think. Yes, I'm pretty sure it's in California. But maybe there are other tall peaks too. > > Wait, no, Mount Whitney is definitely the tallest. It's over 14,000 feet. But sometimes people might confuse it with other mountains like Shasta or something else. Oh, wait, isn't Mount Shasta also a volcano? I think that's in California too but it's not as tall as Whitney. > > So, putting it together, Mount Whitney is the tallest peak in California at 14,505 feet. It's located in the Sierra Nevada range. People hike there a lot because it's such a high point. But I should double-check to make sure I'm not mixing up anything else. Yeah, that seems right. > I wonder if there are any taller peaks elsewhere. Oh, Denali in Alaska is higher, but that's not in California. So within the state, Whitney is the highest. Got it. > So the answer is Mount Whitney at 14,505 feet above sea level. It's a popular spot for hikers and climbers. Makes sense because it's such an iconic peak in the Sierra Nevada mountains. > I think I've got all that right. No other peaks in California are taller than Whitney. So yeah, Mount Whitney is the tallest. > Just to make sure, sometimes there are different names or maybe alternate names for peaks, but Whitney is pretty straightforward. Yeah, it's definitely the one. > Okay, so I'm confident now. The answer is Mount Whitney. > *Final Answer* > The tallest peak in California is Mount Whitney, with an elevation of 14,505 feet above sea level. > > \boxed{Mount Whitney} </details> To determine the tallest peak in California, we consider notable mountains such as Mount Whitney and Mount Shasta. Mount Whitney is recognized as the highest peak in the contiguous United States, standing at 14,505 feet. It is located in the Sierra Nevada mountain range and is a popular destination for hikers. Mount Shasta, although also a significant peak, does not surpass Whitney's elevation. Therefore, the tallest peak in California is confirmed to be Mount Whitney. *Final Answer* The tallest peak in California is \boxed{Mount Whitney}.