10 ms·
The M2 is more advanced than it seemed
- saagarjha 3y agoDidn’t we see this get posted a couple days ago? I could’ve sworn I saw a post that was about bfloat16 here…
- GLaNIK 3y agohttps://news.ycombinator.com/item?id=38978129 https://news.ycombinator.com/item?id=38978129
- saagarjha 3y agoIs this just a rewrite of the older article? What’s the point?
- tomalbrc 3y ago[flagged]
- Dalewyn 3y ago"Newer CPU is better than older CPU; go buy it." That's the point. Whether you adhere to it is none of my concern. Personally, I won't. I'm far more interested in buying a 14700K to upgrade my 12700K daily driver desktop than I am replacing my M2 Macbook Air that spends most of its time as a portable jukebox.
- MrBuddyCasino 3y agotl;dr: upgrade from ARMv8.5-A to the ARMv8.6-A ISA, which supports bfloat16 for better AI performance.
- kxrm 3y agoIf I read this right, this has more to do with training new models than actually running existing models correct?
- foobar1962 3y agoYes, and the article writer is suggesting that more AI training will be done on the client computer, and gave spelling and grammar checking as an example.
- Cthulhu_ 3y agoI don't know much about AI and stuff, but I know Apple's been doing machine learning / image recognition and the like for a long time now, so that the Photos app on your phone can recognize the same people in photos or categorise them - all done on device instead of uploading your photos.
- xanderlewis 3y agoThe processes of training and running use similar low-level operations, but training obviously requires a lot more of them. So it’s relevant to both, but one is much more computation-heavy to begin with. The point is that at the moment only a few of us are interested in training our own models, but this will change gradually as Apple introduces more machine learning models trained on our own local data behind the scenes. At that point, Macs with less support for this will start to slow down (well, or they’ll just not support those features). And the M1 chip will be worse in this regard than the M2.
- DeathArrow 3y agoAt that point, probably both M1 and M2 will be outdated. And people will sell their outdated laptops and buy M5 or M6 laptops. Worrying today about how your hardware will behave in 5 years from now wouldn't help you much.
- ilaksh 3y agoAre PC laptops going to catch up? I wonder if there is some kind of conflict of interest with Nvidia holding back more capable PC chips or software integration.
- azeirah 3y agoSnapdragon is making big claims about the Snapdragon X Elite. It is an ARM chip instead of x86, meant for the laptop market. I haven't delved deep into it yet so I can't make any comparisons or give my own take. https://www.anandtech.com/show/21112/qualcomm-snapdragon-x-elite-performance-preview-a-first-look-at-whats-to-come https://www.anandtech.com/show/21112/qualcomm-snapdragon-x-e...
- incrudible 3y agoThe newer AMD chips have already caught up in terms of performance/efficiency. Software is more of a challenge.
- conradev 3y agoAt both ends of the spectrum? My intuition is that on the low end they’d take a hit in efficiency by not having low-power cores to schedule background tasks onto like Intel and ARM have been pushing
- incrudible 3y agoFair point. They do have efficiency cores (Zen4c) in some of the latest mobile and server chips. However, even when they do have dedicated efficiency cores, it's not clear that this would be a major advantage in the Windows software landscape.
- ahoka 3y agoWhat do you mean?
- mrweasel 3y agoAny modern CPU, made within the past 5 years or so (perhaps longer), is plenty fast, but the software either doesn't take advantage of that speed, or is does so in a very wrong way. There is no reason for Windows to be sluggish on any mid-range CPU or up, yet here we are. I should not, I haven't used Windows for a good while, but I see demos and people are amazed by the speed of which windows open and coming from the Mac or a light weight Linux desktop, I can't say that I'm impressed. Apple still holds the advantage that they just need to target a very small set of hardware configurations, all of which they have designed and control. It does still allow them to build the most pleasant desktop experience of all the current offering, even with less amazing hardware (mostly their GPU). But this is HN, what people probably mean is: LLM are still better with an Nvidia GPU and neither Apples nor AMDs APIs and libraries are as good as Nvidias CUDA.
- DeathArrow 3y ago>pretty well everyone saw it as evolutionary, significantly faster than its predecessor the M1 When it comes for single threaded performance, both Geekbench and Cinebench show that M2 to M3 was a bigger jump than from M1 to M2.
- deleted 3y ago[deleted]
- ki_ 3y agoI have an M2 for my work. MacOS still gives me random lag spikes sometimes. But it's probably because the OS is bad and not the hardware. This is never an issue on win/linux. idk. im not a fan. but.. im just saying.. if the hardware is so superior, why does it feel so inferior? Like.. my $500 linux laptop feels smoother and faster (GUI wise)
- deleted 3y ago[deleted]
- mschuster91 3y ago> I have an M2 for my work. MacOS still gives me random lag spikes sometimes. You're most likely running low on RAM and the machine is swapping. Check in Activity Monitor what's going on.
- crimsontech 3y agoI’ve had a few corporate MacBooks sent to me by clients for security testing and they always seem much slower than my own. It’s normally down to security tooling they install on them.
- charrondev 3y agoThis 100%. My personal MacBook Pro (m1 pro) feels (and benchmarks) far faster than my work M1 Max which has cylance and Jamf on it. Recently it has had a cpu core dedicated to `find / -iname log4j-core*` which has a very unreasonable impact on everything else on the machine.
- eurekin 3y agooh wow! I've cought that system wide find on a few occasions and thought it was related to the specific of my employer. AFAIRC, it was scanning for vulnerable log4j versions
- ctur 3y agoCareful, your homedir has a CloudStorage folder and if you are using, say, Dropbox or Google Drive then that find will be incredibly slow (in addition to security software possibly slowing it down).
- tomtom1337 3y agoIs it possible to "pretend" that one's float16 datatype is in fact bfloat16? Since they occupy the same about of memory?
- lloydatkinson 3y agoIt's possible to act, in software, as any type of arbitrary length numeric value. It's much faster to do it in hardware thought.
- tomtom1337 3y agoOh, I should have clarified - could one start with a bfloat16 on software-side, convert to float16 (so that e.g a 3.4E38 float16 becomes a 65504 float16), then do any "heavy math" in fast hardware float16 instructions, and then convert back at the end?
- warpspin 3y agoNo, you can't. The exponent in a float16 is too small. You'd rather convert back to a 32-bit float, do your operations, and then throw away the surplus precision and convert back to bfloat16.
- tryfinally 3y agoYou can store it and move it around, but arithmetic operations are prohibitively expensive without hardware acceleration. (Note that bfloat16 has a different range than float16, so you can't interpret one as the other)
- tomtom1337 3y agoOh, I should have clarified - could one start with a bfloat16 on software-side, convert to float16 (so that e.g a 3.4E38 float16 becomes a 65504 float16), then do any "heavy math" in fast hardware float16 instructions, and then convert back at the end?
- drakonka 3y agoI'm considering whether I should trade in my M1 MBP for either an M2 15" MBA or M3 MBP today, or wait until M3 MBAs become a thing. I'm also wary of the trade-in value of the M1 MBP, which I guess will go down the longer I wait... But I suppose the value of jumping to a more powerful chip may outweigh the trade-in value of jumping to an M2 today...
- throwaway290 3y agoIf you have so much as worn down paint on keycaps Apple will give you 0$ for it, they want almost exactly new machines.
- drakonka 3y agoOh good to know, thanks. The online quote they gave me was really decent and they didn't ask about keycaps, but I've never done a trade-in before so maybe they'd back out at the last minute after seeing my poor N key...
- turquoisevar 3y agoDon’t listen to them. I’ve traded in more MacBooks than I care to count, almost always via mail (the method with the most complaints because it’s asynchronous). Never had any issues and my keycaps have almost always been worn out. As long as your device meets the condition as answered in the questionnaire, you should get what is quoted. Those are the only things they care about, they don’t even want the charger for example. If you’re truly anxious about it, you can also walk into an Apple Store and you’ll get the decision on the spot.
- throwaway290 3y ago> Don’t listen to them. Thanks, I guess you are the only one with valid experience here. > As long as your device meets the condition as answered in the questionnaire > If you’re truly anxious about it, you can also walk into an Apple Store and you’ll get the decision on the spot. That's exactly what I did. The questionnaire was slightly different than for mail-in. The in-store questionnaire asked for "cracked keycaps" (online didn't) and the employee proceeded to answer yes saying that worn-out paint on keys is "cracked keycaps". Result: $0 value. You can trade in by mail only for store credit, right? Do you know what happens if some employee uses the same stupid logic when handling mail-in? I suppose they just keep the device and give no store credit?
- fauigerzigerk 3y agoThis makes me wonder whether on-device AI will ever have a real chance against cloud based AI except for extremely latency sensitive tasks. If the technology advances at such a pace that almost everyone is sitting on obsolete or far too weak hardware then the economics of on-device AI processing may not work out for app developers. If it's just a matter of running a little bit slower or a little bit faster then it won't be a problem. But if people miss out on important capabilities altogether unless they spend thousands on hardware at least every other year then AI will be almost exclusively a cloud thing.
- brookst 3y agoI expect to see dynamic, hybrid AI frameworks that let the same application run anywhere, with differing amounts of local compute depending on device capabilities. Its the only way huge AI models can be latency and cost efficient across massively heterogeneous execution environments.
- fauigerzigerk 3y agoI'm not sure if it will pan out that way. It sounds awfully complex. I rather suspect that people will do some lowest common denominator inferencing on-device and shift everything else to the cloud.
- brookst 3y agoMulti-layer caches are also complex, but the benefits are worth it. We already see NPUs spanning 3 orders of magnitude in TOPs in consumer devices. In the next few years that will probably be eight or ten orders of magnitude. I don’t think a LCD approach is going to work. I would say that everything gets pushed to the cloud, but the economics are brutal. All of those companies spending billions a year on GPUs have very strong incentives to push as much inference to the edge as possible.
- mark_l_watson 3y agoA few months ago I bought a Mac with a M2 Pro 32G, and it is great for LLMs. I never thought that I would be running models like mixtral-8-7B-q3 locally on my own computer. I think the M2 Pro has 200GB/sec internal bandwidth, about 25% faster than the M3. I think smaller models will keep getting better. For privacy and economic reasons, it makes sense to do as much AI on-device as possible.
- dpacmittal 3y agoI've had a similar experience with lm studio and draw things on my Macbook.
- pc86 3y agoCan you recommend any primers about running small models on-device, training/fine tuning on particular datasets (e.g. a specific code base), etc?
- erichocean 3y agohttps://github.com/ml-explore/mlx https://github.com/ml-explore/mlx https://github.com/ml-explore/mlx-examples https://github.com/ml-explore/mlx-examples This is where all of the action is at right now. You can also use LM Studio: https://lmstudio.ai/ https://lmstudio.ai/
- ParetoOptimal 3y agoAn open source lmstudio software released recently, but I'm having trouble finding a link.
- 2OEH8eoCRo0 3y agoWithout context I thought they meant the M2 Browning. First produced in 1933 and still widely in use today. Pretty advanced for it's day! https://en.wikipedia.org/wiki/M2_Browning https://en.wikipedia.org/wiki/M2_Browning
- pmx 3y agoSurely the context is that this is a tech discussion forum, not a gun one?
- 2OEH8eoCRo0 3y agoThe M2 Browning isn't tech?
- latchkey 3y ago`sysctl -A | grep BF16`