5 ms·
>Time to first token measured with an 8K-token prompt using a 14-billion parameter model with 4-bit quantization Oh dear 14B and 4-bit quant? There are going t
by butILoveLife 8mo ago
>Time to first token measured with an 8K-token prompt using a 14-billion parameter model with 4-bit quantization
Oh dear 14B and 4-bit quant? There are going to be a lot of embarrassed programmers who need to explain to their engineering managers why their Macbook can't reasonably run LLMs like they said it could. (This already happened at my fortune 20 company lol)
- knicholes 8mo agoI wonder if Apple has foresight into locally running LLMs becoming sufficiently useful.
- b112 8mo agoThey do! "You're holding it wrong*
- DiscourseFan 8mo agoIt won’t handle serious tasks but I have Gemma 3 installed on my M2 Mac and it is good for most of my needs—-esp data I don’t want a corporation getting its hands on.
- andai 8mo agoWhat kind of tasks are you using it for? I haven't really found any uses for small models.
- DiscourseFan 8mo agoNothing like coding, just like relatively basic stuff. Idk its hard to explain but I use AI so frequently for work that I have a sense for what it is capable of.
- andai 8mo agoWhich size Gemma are you using? I should clarify that by small I mean in the 3-8B range. I haven't tested the 14-30B ones, my experience is only about the smaller ones. In my experience, small models are not good for coding (except very basic tasks), they're not good for general knowledge. So the only purpose I could see for them would be, when they're given the information, i.e. summarization or RAG. But in my summarization experiments, they consistently misunderstood the information given to them. They constantly made basic errors and failed to understand the text. So having eliminated programming, general knowledge, summarization and (by extension, RAG, because if you can't understand the information, then you can't do RAG either, by definition) -- I have eliminated all the use cases that I had in mind! That would leave very basic tasks like classification or keywords, but I think there they would be in the awkward middle ground of being disappointing relative to big LLMs for many tasks, and cumbersome relative to small specialized models which can run fast and cheap and be fine tuned.
- fnordpiglet 8mo agoI run Qwen 3.5 30B MOE and it’s reasonable at most tasks I would use a local model for - including summarizing things. For instance I auto update all my toolchains automatically in the background when I log in and when finished I use my local model to summarize everything updated and any errors or issues on the next prompt rendering. It’s quite nice b/c everything stay updated, I know whats been updated, and I am immediately aware of issues. I also use it for a variety of “auto correct” tasks, “give me the command for,” summarize the man page and explain X, and a bunch of tasks that I would rather not copy and paste etc.
- bbshfishe 8mo agoYeah no it didn’t. If you have a fully speced out M3/4 MacBook with enough memory you’re running pretty decent models locally already. But no one is using local models anyway.
- weird-eye-issue 8mo ago> Yeah no it didn’t What is "it" and what didn't it do?
- jordhy 8mo agoWith OpenClaw and powerful local models like Kimi 2.5, these specs make a lot of sense.
- jbellis 8mo agoK2.5 isn't remotely a local model
- oofbey 8mo agoYou can totally run it locally. If you have 500GB of RAM.
- deleted 8mo ago[deleted]
- razster 8mo agoOh sure it is! I’ve helped set up an AI cluster rack with four K2.5s. With some custom tooling, we built our own local enterprise setup: Support ticketing system Custom chat support powered by our trained software-support model Resolved repository with detailed step-by-step instructions User-created reports and queries Natural language-driven report generation (my favorite — no more dragging filters into the builder; our (Secret) local model handles it for clients) In-application tools (C#/SQL/ASP.NET) to support users directly, since our software runs on-site and offline due to PPI A cool repair tool: import/export “support file packet patcher” that lets us push fixes live to all clients or target niche cases Qwen3 with LoRA fine-tuning is also incredible — we’re already seeing great results training our own models. There’s a growing group pushing K2.5s to run on consumer PCs (with 32GB RAM + at least 9GB VRAM) — and it’s looking very promising. If this works, we’ll be retooling everything: our apps and in-house programs. Exciting times ahead!
- VladVladikoff 8mo agoI don’t really get why people are smack talking this, are there other laptops available that can do better?
- khana 8mo ago[dead]
- piokoch 8mo agoNope, but other producers does not claim that their hardware "can run AI".
- butILoveLife 8mo agoMy 2023 Nvidia 3060 laptop I spent $700 on?
- john_alan 8mo agoyou can't run models that are bigger than 16GB, not comparable.
- nickthegreek 8mo agosure you can. system ram is will be your limiter here.
- john_alan 8mo agoit's too slow for usable inference though.
- nickthegreek 8mo agoSlow and can’t run are different things, but I get your point.
- diffeomorphism 8mo agoWrong question. If you sell a 6k€ machine "for AI", then you are judged on your own merits. Replies like "but, but other laptops" are very weak attempts at deflection.
- velcrovan 8mo agoThis wasn’t a statement about capability. It’s just a detail about what model they used to compare the speed of two chips for this purpose. You want a bigger model, run a bigger model.