3 ms·
It's quietly broken in favor of local LLMs over the past month. I say this as someone who was a huge skeptic until maybe 2 weeks ago. It's not GPT-4 but it does
by refulgentis 3y ago
It's quietly broken in favor of local LLMs over the past month. I say this as someone who was a huge skeptic until maybe 2 weeks ago. It's not GPT-4 but it doesn't need to be.
StableLM 3B can handle RAG inputs. This is huge. 7B models can't run on consumer mobile hardware:
https://x.com/jpohhhh/status/1747451790969184682?s=20 https://x.com/jpohhhh/status/1747451790969184682?s=20
llama.cpp got examples for iOS / Android in December. It can run on all platforms via one library: https://x.com/jpohhhh/status/1748852554920849579?s=20 https://x.com/jpohhhh/status/1748852554920849579?s=20
retrieval/vector DB can run on all platforms via one library:
https://github.com/Telosnex/fonnx https://github.com/Telosnex/fonnx