9 ms·
Impressive model, for sure. I've been running it on my Mac, now I get to have it locally in my iPhone? I need to test this. Wait, it does agent skills and mobil
by pmarreck 6mo ago
Impressive model, for sure. I've been running it on my Mac, now I get to have it locally in my iPhone? I need to test this. Wait, it does agent skills and mobile actions, all local to the phone? Whaaaat? (Have to check out later! Anyone have any tips yet?)
I don't normally do the whole "abliterated" thing (dealignment) but after discovering https://github.com/p-e-w/heretic https://github.com/p-e-w/heretic , I was too tempted to try it with this model a couple days ago (made a repo to make it easier, actually) https://github.com/pmarreck/gemma4-heretical https://github.com/pmarreck/gemma4-heretical and... Wow. It worked. And... Not having a built-in nanny is fun!
It's also possible to make an MLX version of it, which runs a little faster on Macs, but won't work through Ollama unfortunately. (LM Studio maybe.)
Runs great on my M4 Macbook Pro w/128GB and likely also runs fine under 64GB... smaller memories might require lower quantizations.
I specifically like dealigned local models because if I have to get my thoughts policed when playing in someone else's playground, like hell am I going to be judged while messing around in my own local open-source one too. And there's a whole set of ethically-justifiable but rule-flagging conversations (loosely categorizable as things like "sensitive", "ethically-borderline-but-productive" or "violating sacred cows") that are now possible with this, and at a level never before possible until now.
Note: I tried to hook this one up to OpenClaw and ran into issues
To answer the obvious question- Yes, this sort of thing enables bad actors more (as do many other tools). Fortunately, there are far more good actors out there, and bad actors don't listen to rules that good actors subject themselves to, anyway.
- jackp96 6mo ago[flagged]
- potsandpans 6mo agoI'm tired of this concern trolling.
- deleted 6mo ago[deleted]
- jackp96 6mo ago[flagged]
- deleted 6mo ago[deleted]
- c2k 6mo agoI run mlx models with omlx[1] on my mac and it works really well. [1] https://github.com/jundot/omlx https://github.com/jundot/omlx
- pmarreck 6mo agoHoly hell, how new is this? I've never heard of it, looks great!
- nothinkjustai 6mo agoIt’s completely vibe coded, doesn’t even run on my Mac lol
- onion2k 6mo agoSoftware that doesn't work has been available for decades. It's not a good signal for vibe-coding.
- ctxc 6mo agoScale, my friend, scale...
- eagleinparadise 6mo ago[flagged]
- nothinkjustai 6mo agoYeah definitely a skill issue downloading and running the app and it crashing on startup. Just gotta skillfully press the icon I guess.
- humanperhaps 6mo agoOut of curiosity, do you have anything running on port 8000? Maybe it's not gracefully handling a port collision.
- magospietato 6mo agoHaven't built anything on the agent skills platform yet, but it's pretty cool imo. On Android the sandbox loads an index.html into a WebView, with standardized string I/O to the harness via some window properties. You can even return a rendered HTML page. Definitely hacked together, but feels like an indication of what an edge compute agentic sandbox might look like in future.
- barbazoo 6mo ago> And there's a whole set of ethically-justifiable but rule-flagging conversations (loosely categorizable as things like "sensitive", "ethically-borderline-but-productive" or "violating sacred cows") that are now possible with this, and at a level never before possible until now. I checked the abliterate script and I don't yet understand what it does or what the result is. What are the conversations this enables?
- throwuxiytayq 6mo agoThe in-ter-net is for porn
- spijdar 6mo agoRealistically, a lot of people do this for porn. In my experience, though, it's necessary to do anything security related. Interestingly, the big models have fewer refusals for me when I ask e.g. "in <X> situation, how do you exploit <Y>?", but local models will frequently flat out refuse, unless the model has been abliterated.
- tredre3 6mo agoFrom what I've seen gemma 4 doesn't refuse a lot regarding sex, it only needs little nudging in the right direction sometimes. But it does refuse being critical of the usual topics: israel, islam, trans, or race. So wanting to discuss one of those is the real reason people would use an uncensored model.
- eloisant 6mo agoI tried it on my mac, for coding, and I wasn't really impressed compared to Qwen. I guess there are things it's better at?
- nkohari 6mo agoYou're comparing apples to oranges there. Qwen 3.5 is a much larger model at 397B parameters vs. Gemma's 31B. Gemma will be better at answering simple questions and doing basic automation, and codegen won't be it's strong suit.
- tredre3 6mo agoGemma 4 31B is still not impressive at coding compare to even Qwen 3.5 27B. It's just not its strong suit. So far gemma 4 seems excellent at role playing, document analysis, and decent at making agentic decisions.
- gigatexal 6mo agoThis has been my experience as well, Qwen via Ollama locally has been very very impressive.
- kgeist 6mo agoQwen3.5 comes in various sizes (including 27B), and judging by the posts on HN, /LocalLlama etc., it seems to be better at logic/reasoning/coding/tool calling compared to Gemma 4, while Gemma 4 is better at creative writing and world knowledge (basically nothing changed from the Qwen3 vs. Gemma3 era)
- bossyTeacher 6mo ago>there's a whole set of ethically-justifiable but rule-flagging conversations (loosely categorizable as things like "sensitive", "ethically-borderline-but-productive" or "violating sacred cows") that are now possible with this, and at a level never before possible until now. Mind giving us a few of the examples that you plan to run in your local LLM? I am curious.
- pmarreck 6mo agoI'm not sure what you're angling at but I already gave a set of questions that are ethically legitimate yet routinely censored by the public models: https://news.ycombinator.com/item?id=47654013 https://news.ycombinator.com/item?id=47654013 Not to mention that doing what the big model makers do literally dumbs the model down. They should at least allow something like letting you prove your age and identity to give you access to better/unaligned models, maybe even requiring a license of some sort. Because you know what? SOMEONE in there absolutely has access to the completely uncensored versions of the latest models.
- satvikpendem 6mo agoI tried 1 and a few others with hypothetical situations, public models answer perfectly fine it looks like.
- 3yr-i-frew-up 6mo ago[dead]
- jwr 6mo ago> It's also possible to make an MLX version of it, which runs a little faster on Macs FWIW, I found MLX variants to perform consistently worse (in terms of expected output, not speed) than GGUF in my measurements on my benchmark that matters to me (spam filtering). I used MLX models in LM Studio. GGUF was always slightly better. Perhaps someone who knows more can pitch in and explain this.
- embedding-shape 6mo agoIt isn't 100% clear, but what quantization were you using for each? I've had worse results with MLX 8bit than what you get with Q4 GGUF, same model, seems mxfp8 or bf16 is needed when ran with MLX to get something worthwhile out of them, but I've done very little testing, could have been something specific with the model I was testing at the time.
- pmarreck 6mo agoI was not aware of this. I might not be willing to trade accuracy for speed in this case, then.
- saagarjha 6mo agoI have found that a lot of the techniques used to decensor models (as far as I can tell, they basically get all their weights to say no turned off) also make them really stupid. Like, sure, it will help you rob a bank, but if you ask whether you should rob the bank it will go "The positives: … The negatives: … My take: You should ABSOLUTELY rob the bank".
- lxgr 6mo agoI wonder if this is due to abliteration actually "damaging" the model, or just an artifact of the model never having been properly trained on "forbidden" topics (as it's enough for them to recognize them, and there's no point in dedicating neurons to something that will never be exercised anyway).
- zozbot234 6mo agoModern abliteration is quite good at not damaging the model on ordinary topics. But yes, on many of the weirdest "forbidden" topics (excluding the mild stuff like ordinary erotica) there's not going to be any real training of any sort and it's basically hallucinations running wild. You even see this claim repeated explicitly on every model release "safety card": 'no, this model does not have the sort of fiddly tacit know-how it would need to actually advise anyone nefarious on this dangerous stuff'.
- saagarjha 6mo agoI’m not an expert and this is just a guess from the behavior but I feel like it’s kind of like giving the model a kind of Williams Syndrome, where (AFAIK) a common training process is that you keep asking it sensitive topics and every time it rejects the request you train it away from that and towards “Yeah, absolutely! Let’s go!!!” I feel like the natural result of this is that you’ll get something that is excited about whatever ideas you have?
- pmarreck 6mo agoThe abliteration in particular that Heretic does apparently results in a best-in-class lack of "stupefying" the underlying model. You haven't read its claims, apparently.