3 ms·
Oh is that how it works under the hood?
by barbazoo 4y ago
Oh is that how it works under the hood?
- qeternity 4y agoYes. The chat functionality is really just a clever trick. All LLMs are still just predicting the next token in a sequence (i.e. finishing the sentence).
- vdfs 4y agoIt's not a secret, in OpenAI api you have to keep sending previous question and responses on top of your new question, essentially you are asking new question but give it more context with the previous questions and answers
- d4rkp4ttern 4y agoThank you for clarifying this mystery. My mental model of how ChatGPT works is now clearer. I was somehow thinking that in chat mode it would either (a) need to “update state”, or (b) be fed increasingly longer history. I thought (b) would slow down later responses but I guess with their ginormous compute it’s not perceptible to users.
- Datagenerator 4y agoSomeone made a version with a buffer which only has a fixed size context history so it stays in constant speed mode. It will only focus on the latest questions.
- Veen 4y agoYes, it's why in longer conversations it might forget what was said earlier in the thread. The token limits still apply.