4 ms·
I think drop-in replacements for OpenAI's API are an interesting idea but should be avoided in practice. I believe writing your own wrapper around llama.cpp usi
by behnamoh 3y ago
I think drop-in replacements for OpenAI's API are an interesting idea but should be avoided in practice. I believe writing your own wrapper around llama.cpp using its special flags and using good types for your functions (something that the IDE can use to give suggestions) is the better approach because it reminds the user that they can't simply expect to change the model name and expect everything to work the same. There are all manners of model-specific "stuff" that the programmer needs to know about. From the prompt template to the max context window, whether the model is loaded in llama.cpp to use grammars or not, and what sort of log probs output one can expect from the endpoint....
To this end, I'm working on a simple library (that sadly doesn't have a shiny website like this post) and will make it public soon.
- sho_hn 3y agoThis sounds quite nice and like something I'd be interested in. C/C++?
- behnamoh 3y agoThanks! I'm using Python but with heavy use of types (miss my c++ era). I wanted to be able to call llama.cpp from Python, but I didn't want to use the llama-cpp-python wrapper because it automatically downloads and builds llama.cpp, which I didn't want to do. I like the simplicity of llama.cpp and its unix philosophy of doing one thing well, so I prefer to build llama.cpp myself and then call it from Python.
- sho_hn 3y agoRoger. I'll keep an eye out, do post it on HN :-)
- behnamoh 3y agoCopy that. Will post it on HN in the next few days!
- LoganDark 3y ago> I didn't want to use the llama-cpp-python wrapper because it automatically downloads and builds llama.cpp As in, after you've installed the package, it will pause your application at runtime to download and build llama.cpp? That's not how bindings are supposed to work! That sounds seriously cursed.
- deleted 3y ago[deleted]
- behnamoh 3y agoI meant that when you install llama-cpp-python, it downloads and builds llama.cpp _for_ you, which I don't like because then you're stuck with llama.cpp commits that are kinda old.
- yawnxyz 3y agoVery cool! Do you have a less shiny website or a github link :)
- behnamoh 3y agoThanks for showing interest! I am going to finish up some final touches and soon will setup a less shiny website and make the github repo public.
- arthurcolle 3y agoRefreshingly good take. As much as I like role/content, actually knowing how things get tokenized, processed is pretty invaluable to mastering LLMs as an applied ML practitioner
- brucethemoose2 3y agoI disagree, for 2 reasons: - OpenAI APIs expose the extra features (like llama.cpp's grammar) anyway, as extra parameters. - There's a huge productivity benefit to making your backend swappable, without having to redo all the API calls. Prompt formatting in particular is a huge sticker with OpenAI endpoints though. Something does need to be done about that.