9 ms·
Running Qwen3 on your macbook, using MLX, to vibe code for free
- avetiszakharyan 1y agoI'd thought to share this quick tutorial to get an actual autonomous agent running on your local and doing some simple tasks. Still in progress trying to figure ou right MLX settings or proper model version to do it, but the framework around this approach is solid, so i hought i'd share!
- nottorp 1y agoNow how do you feed it an existing codebase as part of your prompt? Does it even support that (prompt size etc).
- avetiszakharyan 1y agoYe you can just run it in a folder, and ask it to look around, it can execute bash commands, do anything that Claude Code can do. it will read all the codebase if it has to
- pylotlight 1y agoTypically I'd use tools for that as context will be finite, but I hear it does a decent job at tool calling too so should see solid perf there.
- avetiszakharyan 1y ago8b was very bad at tool calling. 30b does an "ok" job but requires the wrapper to do a lot of job, like for example the model will be very random, either have multiple tool_call ags, or have multiple tool calls in one tag inconsistant. but "handle-able"
- walthamstow 1y agoLooks good. I've been looking for a local-first AI-assisted IDE to work with Google's Gemma 3 27B I do think you should disclose that Localforge is your own project though.
- danw1979 1y agoPersonally, I assumed that a blog post on the domain localforge.dev was written by the developers of localforge, but I might be wrong.
- walthamstow 1y agoSure, if you already know what Localforge is before clicking.
- avetiszakharyan 1y agoWhere do i put that, in the blogpost or?
- walthamstow 1y agoIf you can still edit it, adding it to your first comment is fine I would say. "Disclosure: I am the author of Localforge" or similar.
- avetiszakharyan 1y agoNo thats the only thing I can't edit tbh :(
- tasuki 1y agoI didn't know, and still assumed the blog post on localforge.dev was written by the localforge.dev people. Who else?
- deleted 1y ago
- endlessvoid94 1y agoI've found the local models useful for non-coding tasks, however the 8B parameter models so far have proven lacking enough for coding tasks that I'm waiting another few months for whatever the Moore's law equivalent of LLM power is to catch up. Until then, I'm sticking with Sonnet 3.7.
- walthamstow 1y agoIf you have a 32GB Mac then you should be able to run up to 27B params, I have done so with Google's `gemma3:27b-it-qat`
- endlessvoid94 1y agoHm, I've got an M2 air w/ 24GB. Running the 27B model was crawling. Maybe I had something misconfigured.
- 100721 1y agoNo, that sounds right. 24GB isn’t enough to feasibly run 27B parameters. The rule of thumb is approximately 1GB of ram per billion parameters. Someone in another comment on this post mentioned using one of the micro models (Qwen 0.6B I think?) and having decent results. Maybe you can try that and then progressively move upwards? EDIT: “Queen” -> “Qwen”
- simonw 1y agoYou also need to leave space for other apps. If you run a 27B model on a 32GB machine you may find that you can't productively run other apps. I have 64GB and I can only just fit a bunch of Firefox and VS Code windows at the same time as running a 27B model.
- brandall10 1y agoThat rule of thumb is only related to 8 bit quants at low context. The default for ollama is 4 bit, which puts it roughly about 14GB. The vast majority of people run between 4-6 bit depending on system capability. The extra accuracy above 6 tends to not be worth it relative to the performance hit.
- ttoinou 1y agoGreat thank you. Side topic : anyone knows a way to have a centralized proxy to all LLMs services, online or local, that lets our services connect to it and we manage access to LLMs only once there ? And also records calls to LLM. Would make the whole UX of switching LLMs weekly easier, we would only reconfigure the proxy. I know only LiteLLM that can do that but its record of all LLMs calls is a bit clunky to use properly
- deleted 1y ago[deleted]
- mnholt 1y agoI’ve been looking for this for my team but haven’t found it. Providers like OpenAI and Anthropic offer admin token to manage team accounts and you look hook into Ollama or another self managed service for local AI. Seems like a great way to roll out AI to a medium sized team where a very small team can coordinate access to the best available tools so the entire team doesn’t need to keep pace at the current break-neck speed.
- ramesh31 1y agohttps://openrouter.ai/ https://openrouter.ai/
- Havoc 1y agoLitellm is definitely your best bet. For recording - you can probably vibe code a proxy in front of it that mitms it and dumps the request into whatever format you need
- rcarmo 1y agoLitellm can log stuff pretty well on its own.
- tidbeck 1y agoCould you maybe make use of Simon Willsons [LLM lib/app](https://github.com/simonw/llm https://github.com/simonw/llm)? It has great LLM support (just pass in the model to use) and records everything by default.
- chuckadams 1y agoAnyone know of a setup, perhaps with MCP, where I can get my local LLM to work in tandem on tasks, compress context, or otherwise act in concert with the cloud agent I'm using with Augment/Cursor/whatever? It seems silly that my shiny new M3 box just renders the UI while the cloud LLM alone refactors my codebase, I feel they could negotiate the tasks between themselves somehow.
- _joel 1y agoThere's a few Ollama-MCP bridge servers already (from a quick search, also interested myself): ollama-mcp-bridge: A TypeScript implementation that "connects local LLMs (via Ollama) to Model Context Protocol (MCP) servers. This bridge allows open-source models to use the same tools and capabilities as Claude, enabling powerful local AI assistants" simple-mcp-ollama-bridge: A more lightweight bridge connecting "Model Context Protocol (MCP) servers to OpenAI-compatible LLMs like Ollama" rawveg/ollama-mcp: "An MCP server for Ollama that enables seamless integration between Ollama's local LLM models and MCP-compatible applications like Claude Desktop" How you route would be an interesting challenge, presumably could just tell it to use the mcp for certain tasks, thereby offloading locally.
- rcarmo 1y agoI've been toying with Visual Studio Code's MCP and agent support and gotten it to offload things like reference searches and targeted web crawling (look up module X on git repo Y via this URL pattern that the MCP server goes, fetches and parses). I started by giving it a reference Python MCP server and asking it to modify the code to do that. Now I have 3-4 tools that give me reproducible results.
- 101011 1y agoThis is the closest I've found that's akin to Claude Code: https://aider.chat/ https://aider.chat/
- crazymoka 1y agoWhy do you need mlx? Like your blog post by you never explain why things need to be used. Why isn't using localforge enough as it ties into models?
- turnsout 1y agoI believe mlx will allow you to run the models marginally faster (per a recent blog post by @simonw)
- simonw 1y agoYeah, you don't necessarily need it but it's optimized for Apple Silicon and in my experience feels like it gives slightly better performance than GGUFs. I really need to formally measure that so I'm not just running on vibes!
- indigodaddy 1y agoI for one, am willing to just trust you bro ;)
- turnsout 1y agoYeah I’ll go with Simon’s vibes over most people’s measurements!
- freeone3000 1y agomlx is an alternative model format to GGUF. It executes natively on apple silicon using Apple's AI accelerator, rather than through GGUF as a compute shader(!). It's faster and uses fewer resources on Apple devices.
- avetiszakharyan 1y agoI was just trying to make sure is maximally performant, and did it with MLX because i am running on mac hardware and wanted to be able to run 30b in reasonable time so it can actually autonomously code something. Otherwise there are many ways to do it!
- omneity 1y agoI'm using Qwen3-30B-A3B locally and it's very impressive. Feels like the GPT-4 killer we were waiting for for two years. I'm getting 70 tok/s on an M3 Max, which is pushing it into the "very usable" quadrant. What was even more impressive is the 0.6B model which made the sub 1B actually useful for non-trivial tasks. Overall very impressed. I am evaluating how it can integrate with my current setup and will probably report somewhere about that.
- mtw 1y agohow much RAM do you have? I want to compare with my local setup (M4 Pro)
- omneity 1y ago128GB but it's not using much. I'm running Q4 and it's taking 17.94 GB VRAM with 4k context window, 20GB with 32k tokens.
- A4ET8a8uTh0_v2 1y agoI am not a mac person, but I am debating buying one for the unified ram now that the prices seem to be inching down. Is it painful to set up? The general responses I seem to get range from "It is takes zero effort" to "It was a major hassle to set everything up."
- maille 1y agoI have a Windows PC with a GTX 5070 (12GB) any chance to run it?
- simonw 1y agoI expect that will run Qwen 3 8B quite happily, and I've found that to be a surprisingly capable model for its size.
- UK-Al05 1y agoThe 30B one requires 20 GB of memory for me. But some of the lower parameters one should be ok
- avetiszakharyan 1y agoFor me it was peaking at 35GB even when using
- api 1y agoI'm really impressed and also very interested to see models I can run on my MacBook Pro start to generate results close to large hosted "frontier" models, and do so with what I assume are far fewer parameters. I wonder how far this can go?
- simonw 1y agoIt's been a solid trend for the last two years: I've not upgraded my laptop in the time and the quality of results I'm getting from local models on that same machine has continued to rise. My hunch is that there's still some remaining optimization fruit to be harvested but I expect we may be nearing a plateau. I may have to upgrade from 64GB of RAM this year.
- api 1y agoSeeing diffusion language models mature and get better will be interesting. They can be much, much faster on less hardware.
- at0mic22 1y agoIs there a way to achieve the same with ollama?
- simonw 1y agoYes, Ollama has Qwen 3 and it works great on a Mac. It may be slightly slower than MLX since Ollama hasn't integrated that (Apple Silicon optimized) library yet, but Ollama models still use the Mac's GPU. https://ollama.com/library/qwen3 https://ollama.com/library/qwen3
- avetiszakharyan 1y agoYes, i did that but its not apple silicon optimized so it was taking forever for 30b models. So its ok, but its not fantastic
- spmurrayzzz 1y agoYou can just use llama.cpp instead (which is what ollama is using under the hood via bindings). Just need to make sure youre using commit `d3bd719` or newer. I normally use this with nvidia/cuda, but tested on my mbp and havent had any speed issues thus far. Alternatively, LMStudio has MLX support you can use as well.
- freeone3000 1y agoThere needs to be more mention about the requirement of setting the model-name correctly. For this tutorial to be executed top-to-bottom, the model name must be "mlx-community/Qwen3-30B-A3B-8bit". Other model names will result in a 404 -- rightly so, as this is used to determine which model is executed in mlx_lm.serve!
- rickydroll 1y agoI understand why people use the Mac for their local LLM work. I can't bring myself to spend any money on Apple products. I need to find an alternative platform that runs under Linux, and preferably, since I would run this remotely from my work laptop. I would also want to find some way to modulate the power consumption to turn it off automatically when I'm idle.
- telotortium 1y agoEntirely due to the unified RAM between CPU and GPU in Apple Silicon. Laptops otherwise almost never have a GPU with sufficient RAM for LLMs.
- rickydroll 1y agoShould have been clearer. I was thinking of a dedicated in-house LLM server I could use from different laptops.
- lreeves 1y agoThe new AMD chips in the Framework laptops would be a good candidate and I think you can get 96GB RAM in them. Also if the LLM software is idle (like llama.cpp or ollama) there is negligible extra power consumption.
- organsnyder 1y agoI preordered a Framework Desktop with 128GB RAM for exactly this reason. Apparently under Linux it's possible to assign >100GB to the GPU.
- badsectoracula 1y agoIf you don't mind going through the eldritchian horror that is building ROCm from source[0], Qwen_Qwen3-30B-A3B-Q6_K (6bit quantization of the LLM mentioned in the article which in practice shouldn't be much different) works decently fast on a RX 7900 XTX using koboldcpp and llama.cpp. And by "decently fast" i mean "it writes faster i can read". If you're on Debian AFAIK AMD is paying someone to experience the pain in your place, so that is an option if you're building something from scratch, but my openSUSE Tumbleweed installation predates the existence of llama.cpp by a few years and i'm not subjecting myself to the horror that is Python projects (mis)managed by AI developers[1] :-P. EDIT: my mistake, ROCm isn't needed (or actually, supported) by koboldcpp, it uses Vulkan. ROCm is available via a fork. Still, with Vulkan it is fast too. [0] ...and more than once as after some OS upgrade it might break, like mine [1] ok, i did it once, because recently i wanted to try out some tool someone wrote that relied on some AI stuff and i was too stubborn to give up - i had to install Python from source on a Debian docker container because some dependency 2-3 layers deep didn't compile with a newer minor version release of Python. It convinced me to thank yet again to thank Georgi Gerganov for making AI-related tooling that enables people to stick with C++
- xnx 1y agoIt's very cool that useful models can be run on single personal computers at all. For coding, your time is very valuable, and I'd never want to use anything less than the best. I'm happy to pay pennies to use a frontier model with a huge context model and great speed.
- chipsrafferty 1y agoThis is mostly for one of 4 reasons: 1. Sovereignty over data, your outputs can't be stolen or trained on 2. Just for fun / learning / experiment on 3. Avoid detection that you're using AI 4. No Internet connection, in the woods at your cabin or something
- marcalc 1y agoThis is my key points too. I love the power of having search engine on my laptop.
- biker142541 1y agoAgreed. It’s definitely been fun playing locally, learning, fine tuning, etc, but these models just don’t quite cut it for serious development tasks (yet, and assuming none of the above considerations apply). I haven’t found better than Gemini 2.5 for my work so far.
- joejoo 1y agoWhat’s the difference between using MLX and MPS?
- Tokumei-no-hito 1y agoi think MPS is the term for the APIs apple exposes to control their GPUs and MLX is a machine learning framework optimized for using MPS.
- kamranjon 1y agoJust wanted to give a shout out to MLX and MLX-LM - I’ve been using it to fine-tune Gemma 3 models locally and it’s a surprisingly well put together library and set of tools from the Apple devs.
- croemer 1y agoSite seems to have been struck with the HN hug of death
- avetiszakharyan 1y agoI just wana say i got it to make a snake game! :D for free
- nico 1y agoVery cool to see this and glad to discover localforge. Question about localforge, can I combine two agents to do something like: pass an image to a multimodal agent to provide html/css for it, and another to code the rest? In the post I saw there’s gemma3 (multimodal) and qwen3 (not multimodal). Could they be used as above? How does localforge know when to route a prompt to which agent? Thank you
- avetiszakharyan 1y agoyou can combine agents in 2 ways, you can constantly swap agents during one conversation, or you can have the in separate conversations and collaborate. I was even thinking 2 agents can work on 2 separate git clones, and then do PR's to each other. I also like using code to do the image parsing and css, adn then gemini to do the coding. I tried using gemma and qwen for "real stuff" but its more of a, simple stuff only, if i really need output, id' rather spend money for now. Hopefully to change soon. as for rotuing, localforge does NOT know. you choose the agent, and it will loop inside that agent forever. Like, the way it works is that unless agent decides to talk to a user, it will forever be doing function calls and "talking to functions", as one agent. The only routing happens this way. there is main model and there is Expert model. main model knows to ask expert model (see system promp), when its stuck. so for any rouing to happen 1) system prompt needs to mention it 2) a routing to another model should be a function call. that way model knows how to ask another model for something
- nico 1y agoGreat insights, thank you for the extended and detailed answer, I'll have to try it out
- artdigital 1y agoCool, but running qwen3 and doing a ls tool call is not “vibe coding”, this reads more like a lazy ad for localforge I doubt it can perform well with actual autonomous tasks like reading multiple files, navigating dirs and figuring out where to make edits. That’s at least what I would understand under “vibe coding”
- 85392_school 1y agoYou should try it. It's trained for tool calling and thinks before taking action.
- avetiszakharyan 1y agoDefinitely try it, it can navigate files search for stuff, run bash commands, and while 30b is a bit cranky it gets the job done (much worse then i would get when i plug in gpt-4.1, but its still not bad, Kudos o qwen. As for localforge, it really is a vibe coding tool, just like claude or codex, but with the possibility to plug more than just one provider. What's wrong with that?
- tough 1y agoThey're just pointing out how most -educational- content is actually marketing in disguise, which is fine, but also fine to acknowledge i guess, even if a bit snarkily
- avetiszakharyan 1y agoWell its an oss project, free, I kind of didnt see it that way i guess, that something thats given for free is bad-tone to market in any possible way. Iguess from my standpoint its more of a, I just want to show this thing to people, because I am proud of it as a personal project, and it brings me joy to just, put it out there And since if you "just put it out there" it will sink to the bottom of the HN pit, why not get a bit more creative.
- tough 1y ago
- paul7986 1y agoForgive me I am just digesting the term "vibe coding," which doesn't seem like coding at all? It's just typing into your AI's text prompt and describing it to do xyz and then keep making edits til the AI has a working prototype for what you seek. Is that a correct assumption?
- prophesi 1y agoKarpathy's original tweet defining "vibe coding": https://x.com/karpathy/status/1886192184808149383 https://x.com/karpathy/status/1886192184808149383
- paul7986 1y agoSo it's not coding ... it's talking to a LLM via voice or chat and have it code for you. Then ask it to change/edit things and then review the code some or just run an error check so the LLM fixes the error and your done. And so people who are vibe coding are getting paid multiple six figure salaries .... that's not sustainable anyone at any age and in any country can vibe code. Looks like we are embracing the demise of our skill-sets, careers and livelihoods quickly!
- colesantiago 1y agoDo not fear, there will be new jobs available from AI.
- jimbokun 1y agoAnd AI will do those too.
- abc_lisper 1y agolol
- paul7986 1y agoIndeed it will DOGE all those jobs too
- desireco42 1y agoYou can just use Ollama and have a bunch of models, some are good for planning, some are for executing tasks... this sounds more complex then it should be or maybe I am lazy and want everything neatly sorted. I have models on external drive because Apple and through Ollama server they interact really well with Cline or Roo code or even Bolt, but I found Bolt really not working well.
- desireco42 1y agoTo add, you can use so called, abliterated models that are stripped of censorship for example. Much better experience sometimes.
- seanhunter 1y agoYou can already do this, with qwen or (which I use) deepscaler using aider and ollama. This is just an advert for localforge.
- jononor 1y agoRunning models locally is starting to get interesting now. Especially the 30B-A3B version seems like a promising direction, though it is still out of reach on 16 GB VRAM (quite accessible). Hoping for new Nvidia RTX cards with 24/32 GB VRAM. Seems that we might get to GPT4-ish levels within a few years? Which is useful for a bunch of tasks.
- avetiszakharyan 1y agoI think we are just tiny bit away of being able to really "code" with ai, locally. Because even if it would be on gemini2.5 level, since its free, you can make it self prompt a bit more and eventually solve any problem. if i could ran 200b or if 30b wouldve been as good - it wouldve been enough
- rcarmo 1y agoCoincidentally, I just managed to get Qwen3 to go into a loop by using a fairly simple prompt: "create a python decorator that uses a trie to do mqtt topic routing” phi4-reasoning works, but I think the code is buggy phi4-mini-reasoning freaks out qwen3:30b starts looping and forgets about the decorator mistral-small gets straight to the point and the code seems sane https://mastodon.social/@rcarmo/114433075043021470 https://mastodon.social/@rcarmo/114433075043021470 I regularly use Copilot models, and they can manage this without too many issues (Claude 3.7 and Gemini output usable code with tests), but local models seem to not have the ability to do it quite yet.
- avetiszakharyan 1y agoIs there an additional system prompt before that? Or i can repro with just this?
- rcarmo 1y agoJust that. I purposefully used exactly the same thing I did with Claude and Gemini to see how the models dealt with ambiguity.
- datpuz 1y agoI think your prompt is bad. Still impressive that Claude 3.7 handled your bad prompt, but qwen3 had no problem with this prompt: Create a Python decorator that registers functions as handlers for MQTT topic patterns (including + and # wildcards). Internally, use a trie to store the topic patterns and match incoming topic strings to the correct handlers. Provide an example showing how to register multiple handlers and dispatch a message to the correct one based on an incoming topic.
- rcarmo 1y agoI purposefully used exactly the same thing I did with Claude and Gemini to see how the models dealt with ambiguity. It shouldn't have degraded the chain of thought to the point where it starts looping.
- jedisct1 1y agoQwen3 is great, but not for writing code. Even after the recent fixes, and with the recommended parameters, it gets often trapped in a loop. Qwen2.5-32B, Cogito-32B and GLM-32B remain the best options for local coding agents, even though the recently released MiMo model is also quite good for its size.
- bitbasher 1y agoI've been vibe coding for over a decade. All you need is a decent pair of headphones and a pot of coffee. It's not free, but it's pretty cheap.
- datpuz 1y agoThat's just called coding
- redcobra762 1y ago*with good vibes
- madduci 1y agoAnd the "for free" in the title excludes the electricity costs
- thih9 1y agoIn case someone didn’t see that yet, vibe coding has a recent and more specific meaning. https://en.m.wikipedia.org/wiki/Vibe_coding https://en.m.wikipedia.org/wiki/Vibe_coding > a programming paradigm dependent on artificial intelligence (AI), where a person describes a problem in a few sentences as a prompt to a large language model (LLM) tuned for coding. > A key part of the definition of vibe coding is that the user accepts code without full understanding.
- pritambarhate 1y agoJust tried Qwen 3 32B 4bit and 8bit GGUF via Ollama today on RunPod A40 GPUs. For general knowledge questions the results were very underwhelming. Here are answers to 3 questions I asked to the 8 bit GGUF model (skipping the thinking tokens to keep it short): Q: who won the cricket worldcup 2023? A: The *2023 ICC Cricket World Cup* was won by *Australia*. They defeated *New Zealand* in the final on *November 19, 2023*, at the *M. A. Chidambaram Stadium in Chennai, India*. ### Key Details: - *Final Score*: Australia scored *334/5* in their 50 overs, while New Zealand reached *199/8* in their chase. - *Man of the Match*: Pat Cummins (Australia) with 5 wickets for 34 runs. - *Tournament Host*: India. - *Notable Performance*: Australia's aggressive batting and disciplined bowling sealed the victory, marking their *sixth Cricket World Cup title* (tying them with West Indies for the most titles). This was the 13th edition of the ICC Cricket World Cup (50-over format). The T20 World Cup is a separate tournament, with India winning the 2024 edition. Let me know if you need further details! total duration: 34.911890735s load duration: 65.034584ms prompt eval count: 20 token(s) prompt eval duration: 305.460537ms prompt eval rate: 65.47 tokens/s eval count: 511 token(s) eval duration: 34.539660654s eval rate: 14.79 tokens/s Comment: Wrong - https://www.espncricinfo.com/series/icc-cricket-world-cup-2023-24-1367856/india-vs-australia-final-1384439/full-scorecard https://www.espncricinfo.com/series/icc-cricket-world-cup-20... Q: give last 10 movies to win the oscar best picture award Here is a clear and accurate list of the *last 10 Best Picture winners* at the Academy Awards, from *2014 to 2023*, including the ceremony year, the winning film, and the film's release year: --- 1. *2023 (95th Oscars)* - *Winner*: Everything Everywhere All at Once - *Release Year*: 2022 2. *2022 (94th Oscars)* - *Winner*: The Power of the Dog - *Release Year*: 2021 3. *2021 (93rd Oscars)* - *Winner*: Nomadland - *Release Year*: 2020 4. *2020 (92nd Oscars)* - *Winner*: Parasite - *Release Year*: 2019 5. *2019 (91st Oscars)* - *Winner*: Green Book - *Release Year*: 2018 6. *2018 (90th Oscars)* - *Winner*: The Shape of Water - *Release Year*: 2017 7. *2017 (89th Oscars)* - *Winner*: Moonlight - *Release Year*: 2016 8. *2016 (88th Oscars)* - *Winner*: Spotlight - *Release Year*: 2015 9. *2015 (87th Oscars)* - *Winner*: 12 Years a Slave - *Release Year*: 2013 10. *2014 (86th Oscars)* - *Winner*: Argo - *Release Year*: 2012 Comment: Answer number 2 ans 9 are wrong. (https://en.wikipedia.org/wiki/Academy_Award_for_Best_Picture https://en.wikipedia.org/wiki/Academy_Award_for_Best_Picture) I would have expected it to get things which are such big events right at least.
- 999900000999 1y agoVery impressive, it doesn't need to be as good as the pay for token models. For example I've probably spent at least $300 last month on vibe coding, a big part of this is I want to know what tools I'm going to end up competing with, and another is I got a working implementation of one of my side projects, and then I decided I wanted it to be rewritten in another programming language. Even if I chill out a bit here, a refurbished Nvidia laptop would pay for itself within a year. I am a bit disappointed Ollama can't handle the full flow yet, IE it could be a single command. ollama code qwen3
- _bin_ 1y agoI just tried it. It got stuck looping on a `cargo check` call and literally wouldn't do anything else. No additional context, just repeatedly spitting out the same tool call. The problem is the best models barely clear the bar for some stuff in terms of coherence and reliability; anything else just isn't particularly usable.
- 999900000999 1y agoThis happens when I'm using Claude Code too. Even the best models need humans to get unstuck. Fron what I've seen most of them are good at writing new code from scratch. Refactoring is very difficult.
- _bin_ 1y agoI tried it 3-4 times before giving up and it did this every single time. I checked the tool call output and it was running cargo check appropriately. I think maybe the 30b-scale models just aren't sufficient for typical development. You're generally correct though, that from-scratch gets better results. This is a huge constraint of them: I don't want a model that will write something its way. I've already gone through my design and settled on the style/principles/libraries I did for a reason; the bot working terribly with that is a major flaw and I don't see saying "let the bot do things its preferred way" as a good answer. Some systems, things like latency matters, and the bot's way just isn't good enough. The vast majority of man-hours are maintaining and extending code, not green-fielding new stuff. Vendors should be hyper-focused on this, on compliance with user directions, not with building something that makes a react todo-list app marginally faster or better than competitors.
- Tacite 1y agoTrying on Macbook Pro (M4) with 24 GB, the whole system freeze after the first question.
- avetiszakharyan 1y agoQuick Video i made on this topic: https://www.youtube.com/watch?v=-h_IZhOdAeU https://www.youtube.com/watch?v=-h_IZhOdAeU
- idcrook 1y agoI wish when someone publishes something like this included the amount of RAM required! It spits the value out when it runs. And include which specific Apple Silicon CPU. This Qwen3-30B-A3B-8bit - how much RAM under MLX? My 16GB Apple Silicon Macs want to play.
- hadlock 1y agoWith 16gb you can comfortably run a 12b model that's been quantized. Quantizing is (bad example) effectively lossy compression.