Ooba is an outlier, and has tons of overhead over llama.cpp and llama-cpp-python for some reason. Most llama.cpp openai servers are pretty close to vanilla lla

Ooba is an outlier, and has tons of overhead over llama.cpp and llama-cpp-python for some reason. Most llama.cpp openai servers are pretty close to vanilla llama.cpp, albeit without the batching support.