4 ms·

I feel like there is an emerging consensus that [Chat]GPT 3.5/4 is not just 1 big model. A large part of the magic in the final product appears to be many inte

I feel like there is an emerging consensus that [Chat]GPT 3.5/4 is not just 1 big model. A large part of the magic in the final product appears to be many intermediate layers of classification that select the appropriate LLM/method to query. The cheaper models (e.g. Ada/Babbage) could be used for this purpose. Think about why offensive ChatGPT prompts are rejected so quickly compared to legitimate asks for code. Imagine the architectural advantage of a big switch statement over models trained in different domains or initial vectors. Cross-domain queries could be managed mostly across turns of conversation w/ summarization. Think about the Stanford Alpaca parse analysis diagram [0]. You could have an instruction-following model per initial keyword. All of "Write..." might fit into a much smaller model if isolated. This stuff could be partitioned in ways that turn out to be mildly intuitive to a layman. Retraining 7B parameters vs 175B is a big delta. The economics of this must have forced a more modular architecture at scale. Consider why ChatGPT is so cheap. Surely, they figured out a way to break down one big box into smaller ones. [0]: https://github.com/tatsu-lab/stanford_alpaca/blob/main/assets/parse_analysis.png

rcme 4y ago
> Imagine the architectural advantage of a big switch statement over models trained in different domains or initial vectors. Given that the emergent abilities come from the large parameter count and massive amount of training data, using smaller models seems like a distinct disadvantage.
- bob1029 4y ago
  > Given that the emergent abilities come from the large parameter count Where can I find evidence of this?
  - rcme 4y ago
    https://openreview.net/pdf?id=yzkSU5zdwD https://openreview.net/pdf?id=yzkSU5zdwD https://arxiv.org/pdf/2203.15556.pdf https://arxiv.org/pdf/2203.15556.pdf There were also some informal comparisons of GPT models with various parameter counts.
    - bob1029 4y ago
      Excellent info - I did find a bit in the conclusion from the arXiv article: > While the desire to train these mega-models has led to substantial engineering innovation, we hypothesize that the race to train larger and larger models is resulting in models that are substantially underperforming compared to what could be achieved with the same compute budget. This mirrors some of my experience. Training/tuning a 7B parameter model feels like goldilocks right now. We are thinking more about 1 specific domain with 3-4 highly-targeted tasks. Do we need 175B+ parameters for that? I can't imagine it would make our lives easier at the moment. Iteration times & cost are a really big factor right now. Being able to go 10x faster/cheaper makes it worth trying to encourage the smaller model(s) to fit the use case.
  - ux-app 4y ago
    https://ai.googleblog.com/2022/11/characterizing-emergent-phenomena-in.html https://ai.googleblog.com/2022/11/characterizing-emergent-ph...
  - airstrike 4y ago
    https://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf
- notahacker 4y ago
  That doesn't mean that if your parameter count is big enough to handle the input, the next step can't more efficiently hand off [parts of] the query to a specialised module. The emergent properties of LLMs learning maths and chess from text inputs alone are interesting, but specialised engines are much more efficient and more accurate.
- precompute 4y ago
  I agree. But I think do think they use a tiered approach: they have an uncensored model at the very bottom, to which the queries go to first. And then there's the "clean" model, and (for lack of a better explanation) the two outputs are then "XOR"d to get the clean public version.
  - bob1029 4y ago
    I agree the core models are uncensored (as this would introduce a lot of noise), but I don't think any queries are allowed to touch them before a cheaper model first performs screening. If someone is spamming toilet humor, there is no reason to exercise the full 175B+ parameter model each time. The only reason ChatGPT is affordable is because of caching, filtering bad prompts, etc. The underlying LLM would be far too expensive if everyone was hitting it 1:1.
    - precompute 4y ago
      Agree 100%. Have you noticed any lag with more "difficult" queries?
      - bob1029 4y ago
        Yes. I think there are multiple levels of LLM for each use case now. The stuff that absolutely must run on 100B+ models can be pre-classified by something like Ada/Babbage. Attempts at arithmetic or dimensional analysis can be binned and shipped to a monster model. Anything that is more routine information retrieval needs maybe goes to a 7B parameter model. Lots of models with lots of temperature levels/hyperparameters/etc is the only way to achieve the kinds of performance we are seeing. The secret sauce is looking increasingly like a huge classification layer. Nothing you see in ChatGPT is as it appears. Lengthy conversations are managed with recursive summarization. Every conversational turn is potentially handled by a different LLM.
nighthawk454 4y ago
Not exactly the same, but Google’s Pathways architecture is sorta along the same lines https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/ https://blog.google/technology/ai/introducing-pathways-next-...
deleted 4y ago
[deleted]
imranq 4y ago
I suspect they must be selecting different models based on the question asked. I've noticed this myself asking simple questions vs difficult questions, chatGPT is much slower on the difficult questions even though each token has the same computation cost if given to the same model

rcme 4y ago

> Imagine the architectural advantage of a big switch statement over models trained in different domains or initial vectors. Given that the emergent abilities come from the large parameter count and massive amount of training data, using smaller models seems like a distinct disadvantage.

bob1029 4y ago
> Given that the emergent abilities come from the large parameter count Where can I find evidence of this?
- rcme 4y ago
  https://openreview.net/pdf?id=yzkSU5zdwD https://openreview.net/pdf?id=yzkSU5zdwD https://arxiv.org/pdf/2203.15556.pdf https://arxiv.org/pdf/2203.15556.pdf There were also some informal comparisons of GPT models with various parameter counts.
  - bob1029 4y ago
    Excellent info - I did find a bit in the conclusion from the arXiv article: > While the desire to train these mega-models has led to substantial engineering innovation, we hypothesize that the race to train larger and larger models is resulting in models that are substantially underperforming compared to what could be achieved with the same compute budget. This mirrors some of my experience. Training/tuning a 7B parameter model feels like goldilocks right now. We are thinking more about 1 specific domain with 3-4 highly-targeted tasks. Do we need 175B+ parameters for that? I can't imagine it would make our lives easier at the moment. Iteration times & cost are a really big factor right now. Being able to go 10x faster/cheaper makes it worth trying to encourage the smaller model(s) to fit the use case.
- ux-app 4y ago
  https://ai.googleblog.com/2022/11/characterizing-emergent-phenomena-in.html https://ai.googleblog.com/2022/11/characterizing-emergent-ph...
- airstrike 4y ago
  https://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf
notahacker 4y ago
That doesn't mean that if your parameter count is big enough to handle the input, the next step can't more efficiently hand off [parts of] the query to a specialised module. The emergent properties of LLMs learning maths and chess from text inputs alone are interesting, but specialised engines are much more efficient and more accurate.
precompute 4y ago
I agree. But I think do think they use a tiered approach: they have an uncensored model at the very bottom, to which the queries go to first. And then there's the "clean" model, and (for lack of a better explanation) the two outputs are then "XOR"d to get the clean public version.
- bob1029 4y ago
  I agree the core models are uncensored (as this would introduce a lot of noise), but I don't think any queries are allowed to touch them before a cheaper model first performs screening. If someone is spamming toilet humor, there is no reason to exercise the full 175B+ parameter model each time. The only reason ChatGPT is affordable is because of caching, filtering bad prompts, etc. The underlying LLM would be far too expensive if everyone was hitting it 1:1.
  - precompute 4y ago
    Agree 100%. Have you noticed any lag with more "difficult" queries?
    - bob1029 4y ago
      Yes. I think there are multiple levels of LLM for each use case now. The stuff that absolutely must run on 100B+ models can be pre-classified by something like Ada/Babbage. Attempts at arithmetic or dimensional analysis can be binned and shipped to a monster model. Anything that is more routine information retrieval needs maybe goes to a 7B parameter model. Lots of models with lots of temperature levels/hyperparameters/etc is the only way to achieve the kinds of performance we are seeing. The secret sauce is looking increasingly like a huge classification layer. Nothing you see in ChatGPT is as it appears. Lengthy conversations are managed with recursive summarization. Every conversational turn is potentially handled by a different LLM.

nighthawk454 4y ago

Not exactly the same, but Google’s Pathways architecture is sorta along the same lines https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/ https://blog.google/technology/ai/introducing-pathways-next-...

deleted 4y ago

[deleted]

imranq 4y ago

I suspect they must be selecting different models based on the question asked. I've noticed this myself asking simple questions vs difficult questions, chatGPT is much slower on the difficult questions even though each token has the same computation cost if given to the same model