3 ms·
Great job porting the C++ code! Seems like the reasoning was to provide the code as a library to embed in a HTTP Server, cannot wait to see that happen and try
by ilovefood 4y ago
Great job porting the C++ code! Seems like the reasoning was to provide the code as a library to embed in a HTTP Server, cannot wait to see that happen and try it out.
Looking at how the inference runs, this shouldn't be a big problem, right? https://github.com/setzer22/llama-rs/blob/main/llama-rs/src/main.rs#L42 https://github.com/setzer22/llama-rs/blob/main/llama-rs/src/...
- saidinesh5 4y agoIt shouldn't be too much effort to extend/rewrite that function to use websockets (i wouldn't use just http for something like this). All the important functions used by that function (llama_eval, sample_top_p_top_k etc.. ) seem to be public anyway.