4 ms·
Even better, run locally. That should be the goal.
by wsgeorge 4y ago
Even better, run locally. That should be the goal.
- barbazoo 4y agoIs this realistic though anytime soon? How long until this can be self hosted and how long after that until it'll run on a laptop or phone? Granted I know nothing about the ML hardware that's in Apple hardware and even Google phones so maybe we're closer than I imagine. Do you know if that hardware is capable of running LLM or is that optimized for something else?
- wsgeorge 4y ago> Do you know if that hardware is capable of running LLM or is that optimized for something else? I do! Checkout llama.cpp [0] and related projects. The game changer with this was Gerganov's ggml format that makes inference on CPUs realistic. I'm currently running alpaca.cpp and llama.cpp, using 7B models 4-bit quantized (so they're the lowest powered, lightest weight models) on an M1 Pro with 16gig RAM. You could run same on an M1 Macbook Air with 8gig RAM. Gerganov shows the Whisper model running on an Android phone. [0]https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp