5 ms·

One thing I still don't understand is what _is_ the ChatGPT front end exactly? I've used other "conversational" implementations built with the API and they neve

by extr 3y ago

One thing I still don't understand is what _is_ the ChatGPT front end exactly? I've used other "conversational" implementations built with the API and they never work quite as well, it's obvious that you run out of context after a few conversation turns. Is ChatGPT doing some embedding lookup inside the conversation thread to make the context feel infinite? I've noticed anecdotally it definitely isn't infinite, but it's pretty good at remembering details from much earlier. Are they using other 1st party tricks to help it as well?

MaxLeiter 3y ago
It uses a sliding context windows. Older tokens are dropped as new ones stream in
- extr 3y ago
  I don't believe that's the whole story. Other conversational implementations use sliding context windows and it's very noticable as context drops off. Whereas ChatGPT seems to retain the "gist" of the conversation much longer.
  - lsaferite 3y ago
    I mean, I explicitly have the LLM summarize content that's about to fall out of the window as a form of pre-emptive token compression. I'd expect maybe they do something similar.
    - kuchenbecker 3y ago
      I feel like we're describing short vs long term memory.
      - tsunamifury 3y ago
        That’s exactly what it is. It’s just it turns out you need very good generalized or focused simple reasoning to do accurate compression or else the abstraction and movement to long term memory doesn’t include the most important content. Or worse distracting details. I’ve been working on short and long term memory windows at allofus.ai for about 6 months now and it’s way more complex than I had originally thought it would be. Even if you can magically extend the content window, the added data confuses and waters down the reasoning of the LLM. You must do layered abstraction and compression with goal based memory for it to continue to reason without distraction of irrelevant data. It’s an amazing realization, almost like a proof that memory is a kind of layered reasoning compression system. Intelligence of any kind can’t understand everything forever. It must cull the irrelevant details, process the remains and reason on a vector that arises from them.
        yowlingcat 3y ago
        Is it unfair to consider this some kind of correlate to the Nyquist theorem that makes me skeptical of even the theoretical possibility of AGI claims?
        tsunamifury 3y ago
        I consider GPT4 AGI, so I'm probably not the one to ask this too. It reasons, it understands sophisticated topics, it can be given a purpose and pursue it, it can communicate with humans, and it can perform a reasonable task considering its modalities. I don't really know what any sort of "big leap" beyond this people are expecting, incremental performance for sure. But what else?
        lsaferite 3y ago
        I guess for me it needs to have active self-reflection and the ability to act independently/without directions. I'm sure there are many other criteria if I think about it some more, but those two were missing from your list.
        tsunamifury 3y ago
        This is mostly just that gpt4 API/app have this disabled rather than it’s not capable. When you enable it, it is pretty shocking. And it’s pretty simple to enable. You just give it a meta instruct to decide when to message you and what to store to introspect on.
        lsaferite 3y ago
        As a frequent user of the OpenAI APIs, I don't really know what you are talking about here. Could you point me to some documentation?
  - qwertox 3y ago
    At least in 3.5 it's very noticeable when the context drops. They could use summarization, akin to what they are doing when detecting the topic of the chat, but applied to question-answer-pairs in order to "compress" the information. But that would require additional calls into a summarization LLM so I'm really not sure if it is worth it. Maybe they dump some tokens they have on a blacklist or text snippets like "I want to" or replace "could it be that" with "chance of".
shubb 3y ago
This is one of the things that make me uncomfortable about proprietary llm. They get task performance by doing a lot more than just feeding a prompt straight to an llm, and then we performance compare them to raw local options. The problem is, as this secret sauce changes, your use case performance is also going to vary in ways that are impossible for you to fix. What if it can do math this month and next month the hidden component that recognizes math problems and feeds them to a real calculator is removed? Now your use case is broken. Feels like building on sand.
- BoorishBears 3y ago
  I'm not sure you realize how proprietary LLMs are being built on. No one is doing secret math in the backend people are building on. The OpenAI API allows you to call functions now, but even that is just a formalized way of passing tokens into the "raw LLM". All the features in the comment you replied to only apply to the web interface, and here you're being given an open interface you can introspect.
  - edgyquant 3y ago
    It was a contrived example to make a point, one that seems to have flown over your head.
    - BoorishBears 3y ago
      No it was a bad (straight up wrong) example because you don't understand how people are building applications on proprietary LLMs. If you did you'd also know what evals are.
  - shubb 3y ago
    Thank you for pointing that out - I had assumed that things were not how they are. Although performance has varied over time https://arxiv.org/pdf/2307.09009.pdf https://arxiv.org/pdf/2307.09009.pdf I also notice that the API allows you to use a frozen version of the model which avoids the worries I mentioned.
SOLAR_FIELDS 3y ago
They definitely do some proprietary running summarization to rebuild the context with each chat. Probably a RAG like approach that has had a lot of attention and work
- extr 3y ago
  This is effectively my question. I assume there is some magic going on. But how many engineering hours worth of magic, approximately? There is a lot of speculation around GPT-4 being MoE and whatnot. But very little speculation about the magic of the ChatGPT front end specifically that makes it feel so fluid.
  - BoorishBears 3y ago
    That's mostly because there's very little value in deep speculation there. It's not particularly more fluid than anything you couldn't whip up yourself (and the repo linked proves that) but there's also not much value in trying to compete with ChatGPT's frontend. For most products ChatGPT's frontend is the minimal level of acceptable performance that you need to beat, not an maximal one really worth exploring.
    - extr 3y ago
      What front end is better than ChatGPT? Is the OP implementation doing running summarization or in-convo embedding lookup?
      - BoorishBears 3y ago
        It sounds like a cop-out but: it's one made for your use-case. If you're letting people do fun long-form roleplay adventures using summarization alongside some sort of named entity K-V store driven by the LLM would be a good strategy. If you're building a tool that's mostly for internal data, something that leans heavily into detailed answers with direct verbatim citations and having your frontend create new threads when there's a clear break in the topic of a request is a clever strategy since quality drops with context length and you want to save tokens for citations. People who are saying LLMs suck or are X or are Y are mostly just completely underutilizing them because LLMs make it super easy to solve problems superficially: when it comes to actually scaling those solutions to production you need more than random RAG vector database wrappers.
simonbutt 3y ago
Logic for azure chatgpt's "infinite context" summarisation is in https://github.com/microsoft/azurechatgpt/blob/main/src/features/chat/chat-data/chat-data-api.ts https://github.com/microsoft/azurechatgpt/blob/main/src/feat... *Edit Azure chatgpt, would be amazed/disappointed if chatgpt used langchain.
- furyofantares 3y ago
  That doesn't really look right to me, it looks like that's for responding regarding uploaded documents. I see nothing related to infinite context. Also this is the azure repo from OP, nothing to do with the actual ChatGPT front-end that was asked about. I highly doubt the official ChatGPT front-end uses langchain, for example.
- qwertox 3y ago
  I don't see anything related to an infinite context in there. There's only a reference to a server-side `summary` variable which suggests that there is a summary of previous posts which will get sent along with the question for context, as is to be expected. Nothing suggests an infinite context.
- Xenoamorphous 3y ago
  This is Azure's docs to create a conversation: https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/chatgpt?pivots=programming-language-chat-completions#creating-a-basic-conversation-loop https://learn.microsoft.com/en-us/azure/cognitive-services/o...

by extr 3y ago

MaxLeiter 3y ago

It uses a sliding context windows. Older tokens are dropped as new ones stream in

extr 3y ago
I don't believe that's the whole story. Other conversational implementations use sliding context windows and it's very noticable as context drops off. Whereas ChatGPT seems to retain the "gist" of the conversation much longer.
- lsaferite 3y ago
  I mean, I explicitly have the LLM summarize content that's about to fall out of the window as a form of pre-emptive token compression. I'd expect maybe they do something similar.
  - kuchenbecker 3y ago
    I feel like we're describing short vs long term memory.
    - tsunamifury 3y ago
      That’s exactly what it is. It’s just it turns out you need very good generalized or focused simple reasoning to do accurate compression or else the abstraction and movement to long term memory doesn’t include the most important content. Or worse distracting details. I’ve been working on short and long term memory windows at allofus.ai for about 6 months now and it’s way more complex than I had originally thought it would be. Even if you can magically extend the content window, the added data confuses and waters down the reasoning of the LLM. You must do layered abstraction and compression with goal based memory for it to continue to reason without distraction of irrelevant data. It’s an amazing realization, almost like a proof that memory is a kind of layered reasoning compression system. Intelligence of any kind can’t understand everything forever. It must cull the irrelevant details, process the remains and reason on a vector that arises from them.
      - yowlingcat 3y ago
        Is it unfair to consider this some kind of correlate to the Nyquist theorem that makes me skeptical of even the theoretical possibility of AGI claims?
        tsunamifury 3y ago
        I consider GPT4 AGI, so I'm probably not the one to ask this too. It reasons, it understands sophisticated topics, it can be given a purpose and pursue it, it can communicate with humans, and it can perform a reasonable task considering its modalities. I don't really know what any sort of "big leap" beyond this people are expecting, incremental performance for sure. But what else?
        lsaferite 3y ago
        I guess for me it needs to have active self-reflection and the ability to act independently/without directions. I'm sure there are many other criteria if I think about it some more, but those two were missing from your list.
        tsunamifury 3y ago
        This is mostly just that gpt4 API/app have this disabled rather than it’s not capable. When you enable it, it is pretty shocking. And it’s pretty simple to enable. You just give it a meta instruct to decide when to message you and what to store to introspect on.
        lsaferite 3y ago
        As a frequent user of the OpenAI APIs, I don't really know what you are talking about here. Could you point me to some documentation?
- qwertox 3y ago
  At least in 3.5 it's very noticeable when the context drops. They could use summarization, akin to what they are doing when detecting the topic of the chat, but applied to question-answer-pairs in order to "compress" the information. But that would require additional calls into a summarization LLM so I'm really not sure if it is worth it. Maybe they dump some tokens they have on a blacklist or text snippets like "I want to" or replace "could it be that" with "chance of".

shubb 3y ago

This is one of the things that make me uncomfortable about proprietary llm. They get task performance by doing a lot more than just feeding a prompt straight to an llm, and then we performance compare them to raw local options. The problem is, as this secret sauce changes, your use case performance is also going to vary in ways that are impossible for you to fix. What if it can do math this month and next month the hidden component that recognizes math problems and feeds them to a real calculator is removed? Now your use case is broken. Feels like building on sand.

BoorishBears 3y ago
I'm not sure you realize how proprietary LLMs are being built on. No one is doing secret math in the backend people are building on. The OpenAI API allows you to call functions now, but even that is just a formalized way of passing tokens into the "raw LLM". All the features in the comment you replied to only apply to the web interface, and here you're being given an open interface you can introspect.
- edgyquant 3y ago
  It was a contrived example to make a point, one that seems to have flown over your head.
  - BoorishBears 3y ago
    No it was a bad (straight up wrong) example because you don't understand how people are building applications on proprietary LLMs. If you did you'd also know what evals are.
- shubb 3y ago
  Thank you for pointing that out - I had assumed that things were not how they are. Although performance has varied over time https://arxiv.org/pdf/2307.09009.pdf https://arxiv.org/pdf/2307.09009.pdf I also notice that the API allows you to use a frozen version of the model which avoids the worries I mentioned.

SOLAR_FIELDS 3y ago

They definitely do some proprietary running summarization to rebuild the context with each chat. Probably a RAG like approach that has had a lot of attention and work

extr 3y ago
This is effectively my question. I assume there is some magic going on. But how many engineering hours worth of magic, approximately? There is a lot of speculation around GPT-4 being MoE and whatnot. But very little speculation about the magic of the ChatGPT front end specifically that makes it feel so fluid.
- BoorishBears 3y ago
  That's mostly because there's very little value in deep speculation there. It's not particularly more fluid than anything you couldn't whip up yourself (and the repo linked proves that) but there's also not much value in trying to compete with ChatGPT's frontend. For most products ChatGPT's frontend is the minimal level of acceptable performance that you need to beat, not an maximal one really worth exploring.
  - extr 3y ago
    What front end is better than ChatGPT? Is the OP implementation doing running summarization or in-convo embedding lookup?
    - BoorishBears 3y ago
      It sounds like a cop-out but: it's one made for your use-case. If you're letting people do fun long-form roleplay adventures using summarization alongside some sort of named entity K-V store driven by the LLM would be a good strategy. If you're building a tool that's mostly for internal data, something that leans heavily into detailed answers with direct verbatim citations and having your frontend create new threads when there's a clear break in the topic of a request is a clever strategy since quality drops with context length and you want to save tokens for citations. People who are saying LLMs suck or are X or are Y are mostly just completely underutilizing them because LLMs make it super easy to solve problems superficially: when it comes to actually scaling those solutions to production you need more than random RAG vector database wrappers.

simonbutt 3y ago

Logic for azure chatgpt's "infinite context" summarisation is in https://github.com/microsoft/azurechatgpt/blob/main/src/features/chat/chat-data/chat-data-api.ts https://github.com/microsoft/azurechatgpt/blob/main/src/feat... *Edit Azure chatgpt, would be amazed/disappointed if chatgpt used langchain.

furyofantares 3y ago
That doesn't really look right to me, it looks like that's for responding regarding uploaded documents. I see nothing related to infinite context. Also this is the azure repo from OP, nothing to do with the actual ChatGPT front-end that was asked about. I highly doubt the official ChatGPT front-end uses langchain, for example.
qwertox 3y ago
I don't see anything related to an infinite context in there. There's only a reference to a server-side `summary` variable which suggests that there is a summary of previous posts which will get sent along with the question for context, as is to be expected. Nothing suggests an infinite context.
Xenoamorphous 3y ago
This is Azure's docs to create a conversation: https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/chatgpt?pivots=programming-language-chat-completions#creating-a-basic-conversation-loop https://learn.microsoft.com/en-us/azure/cognitive-services/o...