2 ms·
Assistant API is too much of a beta still. I was about to release an app based on the new Assistant API but just a day before the release the response times in
by mrtksn 3y ago
Assistant API is too much of a beta still.
I was about to release an app based on the new Assistant API but just a day before the release the response times increased to 8s flat. When I have function calls, that meant up to a minute to get a response.
I had to dismantle everything Assistant API and implement it with Chat API. Which turned out to be great because in Assistant API the context management was very bad and after a few back and forth messages the cost ballooned to over 10K tokens per message.
When I looked closely at the Assistant API and Chat API, I noticed that Assistant API is just a wrapper over Chat API and acts as a web service that stores the previous messages(so slow response problem was probably due to the web server which keeps track of the context). So I went ahead and implemented my own Assistant API which has more control. For example, I set max token cost per message and if the context balloons over that, I make a request with the context and ask OpenAI to create a summary with all the facts so far, add that summary as a system prompt and my context gets compressed back into reasonable territory.
- infecto 3y agoI still don't even know what the Assistant API is supposed to afford me.
- mrtksn 3y agoIt's useful if you just need to hook up a chat assistant and don't want to bother with the busywork doing it. All you care is loading the messages from the thread(which are conveniently kept for you) and add new messages.
- lobsterthief 3y agoIs the training method similar? For example, a company chatbot would need to know it’s a chatbot for Company Y.
- infecto 3y agoWhat training? Beyond supplying context, I don't think assistants has any fine tuning involved.
- mrtksn 3y agoSo, the Assistant API in OpenAI is just a wrapper over the Chat API. They let you choose which model you would like to use, so as a result of you fine tune a model you should be able to use it. However I never tried fine tuning, I rely on RAG and the Assistant API does provide you some tools to make this a bit easier. What tools? They provide an "editor interface" where you can set function calls, upload some files and access the code interpreter. So if you are making a chatbot for Company Y, you can create an assistant which has information about Company Y in the system prompt and also can access up to date information about the company through function calls you define and the files you upload. If you use only Chat API, you will have to handle these stuff yourself. Actually, though I'm using Chat API I do use the Assistant Editor UI to manage the functions and the system prompts. What I do is, I retrieve the assistant info from the OpenAI Assistant API and then I use this on Chat API. This way I don't have to bother with creating my own UI or fiddle with text files or the code. As Assistant API is just a wrapper, most the data structures I receive from Assistant API directly work in Chat API.
- infecto 3y agoYeah that was kind of my idea, it does not serve much if any purpose and only limits the capability.
- SoulAuctioneer 3y agoIt does considerably more than (poorly) managing the context window. It also (poorly) enables persistent document storage, knowledge retrieval, function calling and code execution.