4 ms·
Local / online hybrid LLM’s could replace googling and Apple is in a great position to provide them. The scary question is how they will monetize this.
by smodo 3y ago
Local / online hybrid LLM’s could replace googling and Apple is in a great position to provide them. The scary question is how they will monetize this.
- api 3y agoApple Silicon (M1/M2) is already great for local LLMs. More cost-effective in most cases than GPUs.
- brutus1213 3y agoI'm curious if you can back this up from a technical perspective. I work with mobile (but all Android these days) and it is a mess. Best training ecosystem is Pytorch (and maybe Jax). For inference, you need to endure all manner of pain to optimize/quantize your model (PTQ has perf loss so Training aware quant is key). There are lil operator optimizations (fusion helps perf) but then certain ops have issues on the accelerator you care about. This android experience is mostly related to ARM issues. Q seems to be investing heavily in tooling but now that they are laying people off, I think their toolchain is a bit risky (and may not work if not on Q hardware). Is the story for optimization much different for Apple silicon?
- upbeat_general 3y agoNot sure about the current situation but since Apple controls literally the entire stack, I can’t imagine it’s a big issue. If they really need a new op, they add it.