3 ms·

The above poster is mostly happy with a 7840HS which tells us they're a laptop user and likely looking at workloads that would work on a laptop with spare time

The above poster is mostly happy with a 7840HS which tells us they're a laptop user and likely looking at workloads that would work on a laptop with spare time hobby levels of effort. Modern laptop chip memory bandwidths are in the 100GB/s to 400GB/s range and empirically a run of the mill Mac is enough for a lot of people to get started even with language models, the main issue being quality of software support. Edit: To be more precise current x86 laptop chips are more like 80GB/s - 120GB/s, have to get a Mac for more than that.

sva_ 2y ago
I guess your post is rather charitable, and I'm a bit of a freetard (personally), but > mostly happy with a 7840HS which tells us they're a laptop user and likely looking at workloads that would work on a laptop with spare time hobby levels of effort. kind of triggered me a little. I'm an ML engineer and do not know any of my colleagues who run any serious training on their laptops, be it x86 or the latest Apple M-whatever. We all just use notebooks with GPUs hosted somewhere else. Perhaps the Apple M chips are a bit better at doing some inference, but at a certain level it is better to just use a dedicated GPU, and I think it will probably stay that way for a while (maybe forever). I have to say, the Apple M chips do impress me, and I wish there was some x86 equivalent, but alas there is none. And Linux support isn't anywhere close where I'd feel confident sacrificing my time sending out bug reports. I don't think any AMD NPU will replace NVIDIA GPUs anytime soon, and that wasn't the purpose of my post. I just wish AMD would actually put out a driver for me to be able to tinker with their hardware which so far they have not. And that sort of inability to even put out a driver to use their consumer-grade hardware does not inflict much confidence in their dedicated GPUs being a viable alternative to NVIDIA cards.
- hedgehog 2y ago
  Hah, ok, yeah except for small org + tiny models no ML team is training on laptops. On the other hand I think we're already seeing more app devs using language models while not doing any training at all, and those people will probably want to work on their laptops. Think code completion, OCR / data extraction, image processing, or even as a local stand-in for cloud APIs similar to how SQLite is used as a database stand-in. In that sense local inference is pretty important.

sva_ 2y ago

I guess your post is rather charitable, and I'm a bit of a freetard (personally), but > mostly happy with a 7840HS which tells us they're a laptop user and likely looking at workloads that would work on a laptop with spare time hobby levels of effort. kind of triggered me a little. I'm an ML engineer and do not know any of my colleagues who run any serious training on their laptops, be it x86 or the latest Apple M-whatever. We all just use notebooks with GPUs hosted somewhere else. Perhaps the Apple M chips are a bit better at doing some inference, but at a certain level it is better to just use a dedicated GPU, and I think it will probably stay that way for a while (maybe forever). I have to say, the Apple M chips do impress me, and I wish there was some x86 equivalent, but alas there is none. And Linux support isn't anywhere close where I'd feel confident sacrificing my time sending out bug reports. I don't think any AMD NPU will replace NVIDIA GPUs anytime soon, and that wasn't the purpose of my post. I just wish AMD would actually put out a driver for me to be able to tinker with their hardware which so far they have not. And that sort of inability to even put out a driver to use their consumer-grade hardware does not inflict much confidence in their dedicated GPUs being a viable alternative to NVIDIA cards.

hedgehog 2y ago
Hah, ok, yeah except for small org + tiny models no ML team is training on laptops. On the other hand I think we're already seeing more app devs using language models while not doing any training at all, and those people will probably want to work on their laptops. Think code completion, OCR / data extraction, image processing, or even as a local stand-in for cloud APIs similar to how SQLite is used as a database stand-in. In that sense local inference is pretty important.