3 ms·
Koboldcpp has an OpenAI (and kobold api) endpoint now, and supports grammar syntax like you said: https://github.com/LostRuins/koboldcpp https://github.com/Los
by brucethemoose2 3y ago
Koboldcpp has an OpenAI (and kobold api) endpoint now, and supports grammar syntax like you said:
https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp
The biggest catch is it doesn't support llama.cpp's continuous batching yet. Maybe soon?