6 ms·
Google calls Gemma 3 the most powerful AI model you can run on one GPU
- cwoolfe 2y agoApparently it can also pray. Seriously, I asked it for biblical advice about a tough situation today and it said it was praying for me. XD
- rurp 2y agoThis reminds me a recent chat I had with Claude, trying to identify what looked like an unusual fossil. The responses included things along the lines of "What a neat find!" or "That's fascinating! I'd love it if you could share more details". The statements were normal nice things to hear from a friend, but I found them pretty off-putting coming from a computer that of course couldn't care less and isn't a living thing I have a relationship with. This sort of thing worries me quite a bit. The modern internet has already sparked an awful lot of pseudo or para-social relationships through social media, OnlyFans and the like, with serious mental health and social cohesion costs. I think we're heading into a world where a lot of the remaining normal healthy social behavior gets subsumed by LLMs pretending to be your friend or romantic interest.
- thijson 2y agohttps://www.youtube.com/watch?v=LghsLs3DYUs https://www.youtube.com/watch?v=LghsLs3DYUs Snip Snip was pleasant to talk to until the very end.
- hengheng 2y agoTrying to find the silver lining in this makes me God's advocate I guess? I was able to reflect a lot on my upbringing by reading reddit threads. Advice columns, relationships, parenting advice, just dealing with people. It was great to finally have a normalized, standardized world view to bounce my own concepts off. It was like an advice column in an old magazine, but infinitely big. In my early 20s I must have spent entire days on there. I guess LLMs are the modern, ultra personalized version of that. Internet average, westernized culture, infinite supply, instantly. Just add water and enjoy a normal view of the world, no matter your surroundings or how you grew up. This is going to help out so many kids. And they're not evil yet. Host your own LLMs before you tell them your secrets, people.
- lamuswawir 2y agoNew term, "God's advocate"!
- neither_color 2y agoIt was great to finally have a normalized, standardized world view to bounce my own concepts off. It was like an advice column in an old magazine, but infinitely big. In my early 20s I must have spent entire days on there. I guess LLMs are the modern, ultra personalized version of that. Internet average, westernized culture, infinite supply, instantly. That's a really interesting way to put it and actually made me look back at my own heavily internet-influenced upbringing. Setting healthy personal boundaries? Mindfulness for emotional management? Elevated respect for all types of people and ways of life beyond what my parents were exposed to? Yes. These were not automatically taught to me by my inherited culture or family. I would not have heard about them in a transformative way without the internet. Maybe passively, as something "those weird rich people" do, but not enough to become embedded in my mental operating system. Not to disparage the old culture. I still borrow a lot from it, but yeah I like westernized internet average culture.
- ryandrake 2y agoAlthough I agree with you and GP, there are cynics who will say: "Ha! You think that the totality of Reddit posts is some kind of normalized, standardized, Internet average world view? HA!" There are people deep in ideological bubbles that think Reddit is too liberal! or Reddit is too young! or Reddit is too atheist! or other complaints that amount to "The average Internet-Person doesn't match what I think the average should be!" and they would not be interested in using that ISO Standard World View for anything. I have a feeling if there is a market for this kind of LLM sounding board, the software writers will need to come up with many different models that differ ideologically, have different priors, and even know different facts and truths, in order to actually be acceptable to a broad swath of users. At the limit, you'd have a different model, tailored to each individual.
- kridsdale1 2y agoI’m in the same boat. And judging by the people I’ve met at Google and FB (before it was Meta) a lot of us are refugees from conservative minded illiberal cultures within North America, Asia, and Europe. Memes are our currency. A lot of the internal cultures of these two companies are steeped in formative memes of those born in the mid-80s who only had the internet to find their people in the early 2000s.
- nemothekid 2y ago>I think we're heading into a world where a lot of the remaining normal healthy social behavior gets subsumed by LLMs pretending to be your friend or romantic interest. This is already happening. https://x.com/zymillyyy/status/1902181493553733941 https://x.com/zymillyyy/status/1902181493553733941
- FridgeSeal 2y agoWow, Twitch parasocial relationships have _nothing_ on this.
- mixmastamyk 2y agoI installed a new “operating system” that sounds a lot like Scarlett Johansson. (Her 2013)
- cwoolfe 2y agoI had to upload that screenshot to ChatGPT and ask it to translate. I must be getting old! ... "Clock my tea" haha. never heard that one before.
- cube00 2y agoAfter giving me continuous wrong answers ChatGPT decided it would try allow me to indulge it in a "learning opportunity" instead. I completely understand your frustration, and I genuinely appreciate you pushing me to be more accurate. You clearly know your way around Rust, and I should have been more precise from the start. If you’ve already figured out the best approach, I’d love to hear it! Otherwise, I’m happy to keep digging until we find the exact method that works for your case. Either way, I appreciate the learning opportunity.
- deleted 2y ago[deleted]
- herbst 2y agoCoding with ChatGPT always makes me angry in the end. Claude at least stays professional. ChatGPT always ends up sounding kinda mean.
- scottyah 2y agoI loved when Gemini called out what I thought was a very niche problem as classic. I think there are very few people attempting this stack, to the point where the vendor's documentation is incorrect and hasn't been updated in two years. "Ah, the "SSL connection already established" error! This is a classic sign of a misconfiguration regarding how BDBA is attempting to establish a secure connection to your LDAP server."
- Boogie_Man 2y agoI spent a good half hour "talking" to 4 mini about why Picard never had a family and the nature of the crew as his family despite the professional distance required. It really praised me when I said the holodeck scene where Data is playing King Henry walking amongst his men and I felt pretty smart and then realized I'd not actually garnered the admiration of anyone or anything.
- prawn 2y agoI think there's a similar trap when you're using it for feedback on an idea or to brainstorm features and it gives you effusive praise. That's not a paying customer or even a real person. Like those people you quickly learn aren't too useful for seeking out over feedback because they rave about everything just to be nice.
- selcuka 2y ago> The statements were normal nice things to hear from a friend How do we know our friends are not "trained" to give such responses and not much different from an LLM in this aspect? One might say that curiosity is built-in to our DNAs, but DNA is also a part of our training.
- fwip 2y agoPeople can want things, so there's the possibility that your friends are being honest. LLMS cannot desire, and in fact are so devoid of interiority that they are not even lying when they say that they do.
- xtiansimon 2y ago> “…I found them pretty off-putting coming from a computer that of course couldn't care less and isn't a living thing I have a relationship with.” I’ve prolly complained about it here, but Spectrum cable’s pay by phone line in NYC has an automated assistant with a few emotive quirks. I’m shocked how angry that robot vote makes me feel. I’m not a violent person, but getting played by a robot sets me over the edge in my work day. Reminds me of a BoingBoing story from years ago about greeter robots being attacked in Japan. Japan has a tradition of verbally greeting customers as they enter the building and large department stores will have dedicated human greeters stationed at the entrance. IIFC this was a large store who replaced human greeters with these robots. Rando customers were attacking these robots. I now know how they feel.
- k8sToGo 2y agoI asked Claude for advice and it said something about it being heartbreaking.
- johnisgood 2y agoLLMs always do. "I'm very sorry to hear that, seems like you are going through a lot" yadda yadda. ChatGPT has been giving me too many emojis lately, however. I might tell it to avoid the use of emojis.
- harvey9 2y agoDouglas Adams' Electric Monk finally available to the public!
- mdp2021 2y agoWhich reveals strong reasons to suspect strong "parroting" qualities. "Parroting" qualities that should have been fought in implementation since day 0.
- Obscurity4340 2y agoIts fun to pick it apart sometimes and get it to correct itself but you often would never know unless you had direct or deep cut knowledge to interrogate it from
- SubiculumCode 2y agoI'm not sure how much people parrot on the daily.
- mdp2021 2y agoAnd we have tackled the issue long ago with education. What are you trying to imply? Imitating fools or foolery is not a goal, replicating the unintelligent is not intelligence - it is strictly undesirable.
- snypher 2y agoI always laughed when ChatGPT would reply with the same emoji I typed, regardless of context. Not sure if parroting exactly, but I assumed it would understand meaning (if not context) of emoji?
- deleted 2y ago[deleted]
- KoolKat23 2y agoAnd henceforth, I'll equate that statement as a pithy one GPU level effort from whoever offers it. "Make sure you write atleast a Jetson Orin Nano Super level message in the condolence card"
- snypher 2y agoReply sounds like it was written by someone who has never had to ask AWS to up their vCPU limit.
- FreebasingLLMs 2y ago[dead]
- CamperBob2 2y agoTechnically, the 1.58-bit Unsloth quant of DeepSeek R1 runs on a single GPU+128GB of system RAM. It performs amazingly well, but you'd better not be in a hurry.
- bestouff 2y agoYou probably meant 128GB
- CamperBob2 2y agoEdited. TBF, I did say it was slow...
- refulgentis 2y agoHave you tried the 0.000000001 bit quant? IIUC it's a single param so youd get much faster speeds.
- jfim 2y agoWhat's the recommended way to run LLMs these days? Ollama seems to work with DeepSeek R1 with enough memory using an older CPU but it's around 1 token/second on my desktop.
- CamperBob2 2y agoI've looked into it and the only sane answer right now is still "If it flies, floats, or infers, rent it." You need crazy high memory bandwidth for good inference speed, and that means GPUs which are subject to steep monopoly pricing. That doesn't look to be changing anytime soon. Second place is said to be the latest Macs with lots of unified memory, but it's a distant second place. The recently announced hardware from nvidia is either underpowered, overpriced, or both, so there's not much point waiting for it.
- jfim 2y agoMakes sense, thanks for sharing! I'll take it your recommendation to not buy also includes the upcoming project digits box from Nvidia?
- ForTheKidz 2y ago...until this coming tuesday? ...let's talk value. EDIT: I do feel like a fool, thank you.
- refulgentis 2y agoGood news: it's free. Infinite value.
- rafaelmn 2y agoPlenty of free things with no value
- 2OEH8eoCRo0 2y agoThere's nothin more expensive than free
- impure 2y agoIt’s a 27B model, I highly doubt that.
- xnx 2y agoWhich part do you doubt? That it is the most powerful? That it runs on a single GPU?
- warkdarrior 2y agoWhat's a better model to run on one GPU?
- int_19h 2y agoQwQ-32
- snovv_crash 2y agoIf I ask qwq32 anything that is even slightly complicated it will ramble until it exceeds the context window, then forget my question. Q4k which is all that fits (with context) on a 3090. Gemma3 27B gives me a rapid 1shot response, and actually works really well for the type of rubber duck brainstorming partner I often need.
- hy4000days 2y agoI’ve had this same exact experience.
- RandyRanderson 2y agoSo says Gemma 3.
- deleted 2y ago[deleted]
- ChrisArchitect 2y agoGoogle post from last week: https://blog.google/technology/developers/gemma-3/ https://blog.google/technology/developers/gemma-3/
- odysseus 2y agoDoes it run on the severed floor?
- butterlettuce 2y agoI love how this show (2022) is just heavily emanating into pop culture. Stoked for the season 2 finale today. It'll be like watching the Super Bowl.
- siva7 2y agowhat show?
- deleted 2y ago[deleted]
- alas44 2y agoSeverance
- jprd 2y agoSeverance
- fortyseven 2y agoSeverance
- tailspin2019 2y agoSeverance
- vsgherzi 2y agoactually the finale is on friday not today!
- rjeli 2y agothey come out thursday nights (in the US) - 9pm EST
- zeroq 2y agoI call it the biggest bs since I had my supper.
- grej 2y agoIt lasted until Mistral released 3.1 Small a week later. Such is the pace of AI...
- nirav72 2y agoyeah, I can't even keep up these days. So now mainly focus on what I can run locally via Ollama.
- simonw 2y agoGemma 3 is on Ollama now https://ollama.com/library/gemma3 https://ollama.com/library/gemma3 but surprisingly they don't have Mistral 3.1 yet. I've managed to run Mistral 3.1 on my laptop using MLX, notes here https://simonwillison.net/2025/Mar/17/mistral-small-31/ https://simonwillison.net/2025/Mar/17/mistral-small-31/
- thot_experiment 2y agoAre you running vision tasks? Works just fine in ollama as an LLM but the vision component is unimplemented.
- pitzips 2y agoI'm using gemma3:27b with the latest ollama (0.6.2) and vision is working.
- thot_experiment 2y agoYes but we're talking about mistral 3.1 here, which doesn't have the vision component implemented, looks like vllm supports it though.
- simonw 2y agoMistral 3.1 vision works with MLX and mlx-vlm - https://github.com/Blaizzy/mlx-vlm/pull/256 https://github.com/Blaizzy/mlx-vlm/pull/256 https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-Instruct-2503-4bit https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-I...
- pram 2y agoGemma 3 is a lot better at writing for sure, compared to 2, but the big improvement is I can actually use a 32k+ context window and not have it start flipping out with random garbage.
- williamDafoe 2y agoDoes anyone use GoogleAI? For an AI Company with an AI Ceo using AI language translation, I think their actual GPT products are all terrible and have a terrible rep. And who wants their private conversation shipped back to google for spying?
- LeoPanthera 2y agoI use Gemini (Advanced) over ChatGPT. Google's privacy issues are concerning, but no moreso than giving my conversations to OpenAI. In my experience, Gemini simply works better.
- KoolKat23 2y agoGemini 2.0 Flash et al are excellent. Think you need to try them again.
- deepsquirrelnet 2y agoI tried a lot of models on openrouter recently, and I have to say that I found Gemini 2.0 flash to be surprisingly useful. I’d never used one of Google’s proprietary models before that, but it really hits a sweet spot in the quality vs latency space right now.
- herbst 2y agoThe thing they ship on android is just horrible. It appears closer to markov chain than actual AI
- timmg 2y agoI'm wondering how small of a model can be "generally intelligent" (as in LLM intelligent, not AGI). Like there must be a size too small to hold "all the information" in. And I also wonder at what point we'll see specialized small models. Like if I want help coding, it's probably ok if the model doesn't know who directed "Jaws". I suspect that is the future: many small, specialized models. But maybe training compute will just get to the point where we can run a full-featured model on our desktop (or phone)?
- idonotknowwhy 2y ago> Like there must be a size too small to hold "all the information" in. We're already there. If you running a Mistral-Large-2411 and Mistral-Small-2409 locally, you'll find the larger model is able to recall more specific details about works of fiction. And Deepseek-R1 is aware of a lot more. Then you ask one of the Qwen2.5 coding models, and they won't even be aware of it, because they're: > small, specialized models. > But maybe training compute will just get to the point where we can run a full-featured model on our desktop (or phone)? Training time compute won't allow the model to do anything out of distribution. You can test this yourself if you run one of the "R1 Distill" models. Eg. If you run the Qwen R1 distill and ask it about niche fiction, no matter how long you let it <think> for, it can't tell you something the original Qwen didn't know.
- timmg 2y ago> We're already there. If you running a Mistral-Large-2411 and Mistral-Small-2409 locally, you'll find the larger model is able to recall more specific details about works of fiction. Oh, for sure. I guess what I'm wondering is if we know the Small model (in this case) is too small -- or if we just haven't figured out how to train well enough? Like, have we hit the limit already -- or, in (say) a year, will the Small model be able to recall everything the Big model does (say, as of today)?
- hnuser123456 2y agoI suppose we could eventually get to a super-MoE architecture. Models are limited to 4-16GB in size, but you could have hundreds of models on various topics. Load from storage to RAM and unload as needed. Should be able to load up any 4-16GB model in a few seconds. Maybe as well as a 4GB "Resident LLM" that is always ready to figure out which expert to load.
- LeoPanthera 2y agoMaybe Llama 3.3 70B doesn't count as running on "one GPU", but it certainly runs just fine on one Mac, and in my tests it's far better at holding onto concepts over a longer conversation than Gemma 3 is, which starts getting confused after about 4000 tokens.
- m00x 2y agoI found Mistral Small 3.1, which released slightly after Gemma 3, much better. Much fewer refusals, more accurate, less babbling, generally better, but especially at coding.
- pretoriusdre 2y agoMy instinct is that it would be cheaper overall to buy API credits when needed, compared with buying a top-of-the-line GPU which sits idle for most of the day. That also opens up access to larger models.
- tootie 2y agoIt's a choice. Running local means personal safety and privacy. It could also mean easier compliance with any enterprise that doesn't want to share data.
- brookst 2y agoYeah. Classic capacity / utilization problem.
- rs186 2y agoAgrees with my own experience. I have a 4070 super which of course is nothing to brag about, but tps using quantized 27b model is miserable. I could go down to 12b or even smaller, but it would sacrifice in quality. Then I could upgrade my gears, but I realize that however much I spend, the experience is not going to be as smooth as off-the-shelf LLM products, and definitely not worth the cost. Of course it is nice to have an LLM running locally where nobody gets to know your data. But I don't see a point in me spending thousands of $ to do that.