4 ms·
I feel like https://github.com/ggerganov/llama.cpp/issues/171 https://github.com/ggerganov/llama.cpp/issues/171 is a better approach here? With how fast llama.
by cozzyd 4y ago
I feel like https://github.com/ggerganov/llama.cpp/issues/171 https://github.com/ggerganov/llama.cpp/issues/171 is a better approach here?
With how fast llama.cpp is changing, this seems like a lot of churn for no reason.
- ronsor 4y agoOh hey, that's my issue. It seems like most of the work would simply be moving the inference stuff (feeding tokens to the model and sampling) outside of main(). Most of the other functionality such as model weights loading are already handled in their own functions.
- tantony 4y agoI am working on https://github.com/ggerganov/llama.cpp/issues/77 https://github.com/ggerganov/llama.cpp/issues/77 to "librarify" the code a bit. I eventually did want to Rewrite It In Rust™, but OP just beat me to it.