By the time this is production ready it will no longer fit in that cache. There is a reason it is tiny. Notice how the majority of the features are planned. It

By the time this is production ready it will no longer fit in that cache. There is a reason it is tiny. Notice how the majority of the features are planned. It wouldnt take much for an experienced engineer to simply deploy llama.cpp or one of the other inference backends directly themselves. llama.cpp already includes an openAI compatible API: https://github.com/ggerganov/llama.cpp/blob/master/examples/server/api_like_OAI.py https://github.com/ggerganov/llama.cpp/blob/master/examples/...