4 ms·

LLama.cpp Got Screwd

11 points by trilogic 3mo ago

trilogic 3mo ago
We are heading towards a closed and monitored system, better said a proprietary one soon. Get the last genuine llama.cpp build before to late b9925. The CLI now asking for access to load the model? Here the change that screw everything: b9927 @github-actions github-actions released this 17 hours ago b9927 c264f65 Details cli : move to HTTP-based implementation (#24948) cli: move to HTTP-based implementation wip working remote server ok cli support router mode Co-authored-by: Piotr Wilkin ilintar@gmail.com case: router with only one model Apply suggestions from code review Co-authored-by: Piotr Wilkin (ilintar) piotr.wilkin@syndatis.com remove outdated comment use destructor instead add ftype cli-view --> cli-ui pimpl no more json in header nits fixes also show model aliases Adios llama.cpp.
rimworld 3mo ago
only a matter of time, thx for heads up
magicalhippo 3mo ago
Here[1] is the offending pull request. The llama.cpp devs are trying to refactor the code, so the "engine" bits are shared between the current server and client applications. Georgi himself noted it should be possible to replace the transport layer from HTTP to say an internal message queue, and the main developer of the PR said the HTTP bit is already abstracted and intended to include a non-HTTP transport example, so seems clear to me it's not intended to be HTTP-only. [1]: https://github.com/ggml-org/llama.cpp/pull/24948 https://github.com/ggml-org/llama.cpp/pull/24948
- trilogic 3mo ago
  How is adding compulsory HTTP to CLI necessary for refactoring the code? You realize that now the CLI needs http permissions to load a model locally. Still confused why, the server already does that by all means. First they changed the remote downloading weights in server. Then they approved and merged claude code snippets. Now mandatory monitoring. Everyone has a price
  - magicalhippo 3mo ago
    If you check out the preceeding PR[1], linked from the merged PR, the use-case is if you got a llama-server running but is restricted to console, or you just want to debug, so you want to connect to it using llama-cli. So it's easy to see how HTTP became the primary transport, because it facilitated the motivating use-case. [1]: https://github.com/ggml-org/llama.cpp/pull/21674 https://github.com/ggml-org/llama.cpp/pull/21674

11 points by trilogic 3mo ago

We are heading towards a closed and monitored system, better said a proprietary one soon. Get the last genuine llama.cpp build before to late b9925. The CLI now asking for access to load the model? Here the change that screw everything: b9927 @github-actions github-actions released this 17 hours ago b9927 c264f65 Details cli : move to HTTP-based implementation (#24948) cli: move to HTTP-based implementation wip working remote server ok cli support router mode Co-authored-by: Piotr Wilkin ilintar@gmail.com case: router with only one model Apply suggestions from code review Co-authored-by: Piotr Wilkin (ilintar) piotr.wilkin@syndatis.com remove outdated comment use destructor instead add ftype cli-view --> cli-ui pimpl no more json in header nits fixes also show model aliases Adios llama.cpp.

rimworld 3mo ago

only a matter of time, thx for heads up

magicalhippo 3mo ago

Here[1] is the offending pull request. The llama.cpp devs are trying to refactor the code, so the "engine" bits are shared between the current server and client applications. Georgi himself noted it should be possible to replace the transport layer from HTTP to say an internal message queue, and the main developer of the PR said the HTTP bit is already abstracted and intended to include a non-HTTP transport example, so seems clear to me it's not intended to be HTTP-only. [1]: https://github.com/ggml-org/llama.cpp/pull/24948 https://github.com/ggml-org/llama.cpp/pull/24948

trilogic 3mo ago
How is adding compulsory HTTP to CLI necessary for refactoring the code? You realize that now the CLI needs http permissions to load a model locally. Still confused why, the server already does that by all means. First they changed the remote downloading weights in server. Then they approved and merged claude code snippets. Now mandatory monitoring. Everyone has a price
- magicalhippo 3mo ago
  If you check out the preceeding PR[1], linked from the merged PR, the use-case is if you got a llama-server running but is restricted to console, or you just want to debug, so you want to connect to it using llama-cli. So it's easy to see how HTTP became the primary transport, because it facilitated the motivating use-case. [1]: https://github.com/ggml-org/llama.cpp/pull/21674 https://github.com/ggml-org/llama.cpp/pull/21674