3 ms·
Actually if you poke around at it a little more closely you'll find there's also a version of Llamafile that just ships llama.cpp, enabling you to supply a mode
by chaosharmonic 3y ago
Actually if you poke around at it a little more closely you'll find there's also a version of Llamafile that just ships llama.cpp, enabling you to supply a model of your choosing.
Also the point of a tool like this to make the ecosystem more accessible to everyone, not just software developers.
- cyanydeez 3y agogonna be awhile before sufficiently powerful hardware is GA for LLM
- chaosharmonic 3y agoI mean, that depends somewhat on your use case. My primary development machine -- a Framework with a recent board and 32GB of RAM -- is perfectly capable of running smaller models on CPU, and while it's certainly easy to find where the ceiling is (and this obviously isn't the hardware profile of an average user), it's still more than sufficient for basic use Even if it does also chew through my battery...