4 ms·
Could use LocalAI to get around this: “The openAI models perform better (but require you to have an openAI API key, so cannot be used for free);” https://www.r
by anton5mith2 3y ago
Could use LocalAI to get around this: “The openAI models perform better (but require you to have an openAI API key, so cannot be used for free);”
https://www.reddit.com/r/selfhosted/comments/12w4p2f/localai_openai_compatible_api_to_run_llm_models/ https://www.reddit.com/r/selfhosted/comments/12w4p2f/localai...
- syntaxing 3y agoI was so excited until I saw that it's CPU based only. Would you happen to know of any alternative for GPU support, particularly GPTQ models? Edit: I think textgen itself can support this nowadays
- anton5mith2 3y agoFor now it is CPU only yes, uses AVX instructions. But it's pretty fast anyway, try it out. I have it running on my mbp M1 and it's pretty decent. I think GPU support will come eventually. I wrote an app that uses the openai API and it was nice and simple to just point it at my own local service instead.