3 ms·
Does it offer a local api so i can embed this in my python build?
by ragnarok123 3y ago
Does it offer a local api so i can embed this in my python build?
- AMICABoard 3y agonot yet
- sa-code 3y agoThe ability to hook it into frameworks like streamlit would be huge. Sorta like llama-cpp-python
- AMICABoard 3y agoAgreed and yes would be awesome... But currently this does not infer large meta 7b model efficiently - like 1 token or lower per seconds. But the small toy story (not so useful) models are fast. If the above mentioned API / python binding is ready, I'll make a streamlit interface demo. The streamlit demo should be simple. But I have to figure out python binding.
- AMICABoard 3y agoIf you add an issue you can do so on https://github.com/trholding/llama2.c https://github.com/trholding/llama2.c I have a planned a web api. Are you looking for python binding? I'm interested to know. The issues would keep me organized.