3 ms·
Oh hey, that's my issue. It seems like most of the work would simply be moving the inference stuff (feeding tokens to the model and sampling) outside of main()
by ronsor 4y ago
Oh hey, that's my issue.
It seems like most of the work would simply be moving the inference stuff (feeding tokens to the model and sampling) outside of main(). Most of the other functionality such as model weights loading are already handled in their own functions.
- tantony 4y agoI am working on https://github.com/ggerganov/llama.cpp/issues/77 https://github.com/ggerganov/llama.cpp/issues/77 to "librarify" the code a bit. I eventually did want to Rewrite It In Rust™, but OP just beat me to it.