4 ms·
> Imagine the architectural advantage of a big switch statement over models trained in different domains or initial vectors. Given that the emergent abilities
by rcme 4y ago
> Imagine the architectural advantage of a big switch statement over models trained in different domains or initial vectors.
Given that the emergent abilities come from the large parameter count and massive amount of training data, using smaller models seems like a distinct disadvantage.
- bob1029 4y ago> Given that the emergent abilities come from the large parameter count Where can I find evidence of this?
- rcme 4y agohttps://openreview.net/pdf?id=yzkSU5zdwD https://openreview.net/pdf?id=yzkSU5zdwD https://arxiv.org/pdf/2203.15556.pdf https://arxiv.org/pdf/2203.15556.pdf There were also some informal comparisons of GPT models with various parameter counts.
- bob1029 4y agoExcellent info - I did find a bit in the conclusion from the arXiv article: > While the desire to train these mega-models has led to substantial engineering innovation, we hypothesize that the race to train larger and larger models is resulting in models that are substantially underperforming compared to what could be achieved with the same compute budget. This mirrors some of my experience. Training/tuning a 7B parameter model feels like goldilocks right now. We are thinking more about 1 specific domain with 3-4 highly-targeted tasks. Do we need 175B+ parameters for that? I can't imagine it would make our lives easier at the moment. Iteration times & cost are a really big factor right now. Being able to go 10x faster/cheaper makes it worth trying to encourage the smaller model(s) to fit the use case.
- ux-app 4y agohttps://ai.googleblog.com/2022/11/characterizing-emergent-phenomena-in.html https://ai.googleblog.com/2022/11/characterizing-emergent-ph...
- airstrike 4y agohttps://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf
- notahacker 4y agoThat doesn't mean that if your parameter count is big enough to handle the input, the next step can't more efficiently hand off [parts of] the query to a specialised module. The emergent properties of LLMs learning maths and chess from text inputs alone are interesting, but specialised engines are much more efficient and more accurate.
- precompute 4y agoI agree. But I think do think they use a tiered approach: they have an uncensored model at the very bottom, to which the queries go to first. And then there's the "clean" model, and (for lack of a better explanation) the two outputs are then "XOR"d to get the clean public version.
- bob1029 4y agoI agree the core models are uncensored (as this would introduce a lot of noise), but I don't think any queries are allowed to touch them before a cheaper model first performs screening. If someone is spamming toilet humor, there is no reason to exercise the full 175B+ parameter model each time. The only reason ChatGPT is affordable is because of caching, filtering bad prompts, etc. The underlying LLM would be far too expensive if everyone was hitting it 1:1.
- precompute 4y agoAgree 100%. Have you noticed any lag with more "difficult" queries?
- bob1029 4y agoYes. I think there are multiple levels of LLM for each use case now. The stuff that absolutely must run on 100B+ models can be pre-classified by something like Ada/Babbage. Attempts at arithmetic or dimensional analysis can be binned and shipped to a monster model. Anything that is more routine information retrieval needs maybe goes to a 7B parameter model. Lots of models with lots of temperature levels/hyperparameters/etc is the only way to achieve the kinds of performance we are seeing. The secret sauce is looking increasingly like a huge classification layer. Nothing you see in ChatGPT is as it appears. Lengthy conversations are managed with recursive summarization. Every conversational turn is potentially handled by a different LLM.