3 ms·
> Doing everything on-device would result in a horrible user experience. They might as well not participate in this generative AI rush at all if they hoped to k
by Terretta 2y ago
> Doing everything on-device would result in a horrible user experience. They might as well not participate in this generative AI rush at all if they hoped to keep it on-device.
On the contrary, I'm shocked over the last few months how "on device" on a Macbook Pro or Mac Studio competes plausibly with last year's early GPT-4, leveraging Llama 3 70b or Qwen2 72b.
There are surprisingly few things you "need" 128GB of so-called "unified RAM" for, but with M-series processors and the memory bandwidth, this is a use case that shines.
From this thread covering performance of llama.cpp on Apple Silicon M-series …
https://github.com/ggerganov/llama.cpp/discussions/4167 https://github.com/ggerganov/llama.cpp/discussions/4167
… "Buy as much memory as you can afford would be my bottom line!"
- philjohn 2y agoYes - but people don't want to pay $4k for a phone with 128GB of unified memory, do they? And whilst the LLM's running locally are cool, they're still pretty damn slow compared to Chat-GPT, or Meta's LLM.
- theshrike79 2y agoDepending on what you want to do though. If I want some help coding or ideas about playlists, Gemini and ChatGPT are fine. But when I'm writing a novel about an assassin with an AI assistant and the public model keeps admonishing me that killing people is bad and he should seek help for his tendencies, it's a LOT faster to just use an uncensored local LLM. Or when I want to create some people faces for my RPG campaign and the online generator keeps telling me my 200+ word prompt is VERBOTEN. And finally I figure out that "nude lipstick" is somehow bad. Again, it's faster to feed all this to a local model and just get it done overnight than fight against puritanised AIs.
- chuckadams 2y agoTo say nothing of battery life.