3 ms·
Simply scaling up the current LLM models will not fix the fundamental problems of hallucination and lack of planing abilities. If they cost $10M+ to train, will
by verdverm 4y ago
Simply scaling up the current LLM models will not fix the fundamental problems of hallucination and lack of planing abilities. If they cost $10M+ to train, will scaling that cost 100X (to $1B) really make sense. I find it doubtful that the ROI is there
Consider that GPT4 is 15-30X the cost of GPT3.5, is it really 15-30X better? Not in my experience. I would expect that we are approaching the point of diminishing returns with text based LLMs, at least for the current architectures and training regimes. I'm interested to see what the Large Multi-Modal Models produce, but I suspect the planning & lying will still be there.
Chaining these models will increase the cost even more, without certainty that it will produce usable results for anything beyond simple, context limited tasks.
- rbanffy 4y agoWhen it becomes that large and expensive, it makes sense for governments to do it and open up the result so everyone can benefit.
- verdverm 4y agoThat also makes sense when we account for how gov'ts are really good at wasting money for below average results... This is not something I would trust them to do competently or within a reasonable budget. If it was done through the NSF or DARPA, with an industry partner, maybe? But I can't imagine the current politicians and them arguing about if & how to put restraints on it. It will be interesting to see how the CCP goes about this. Maybe the EU can offer a third alternative?
- rbanffy 4y ago> That also makes sense when we account for how gov'ts are really good at wasting money for below average results... The goal of governments is not efficiency, but fairness and strategic action aiming at long-term results. I like to joke that one of the key goals of NASA is to prevent rocket scientists from moving to North Korea.
- flangola7 4y ago15-30x more expensive doesn't necessarily mean it actually requires 15-30x more computation. I'm certain GPT-4 is extremely profitable; they can charge whatever they want because nothing else like it exists. 3.5 turbo also isn't GPT-3/Davinci, which is still more expensive and often better for non chatbot functions. Additionally, 4 definitely provides at least 15x more value for me than 3.5. Turbo is a toy by comparison. The differences on complex tasks are subtle but profound. OpenAI just received $10B and had to pause new paid subscriptions because more people were signing up than their hardware could handle. Even if they think it is diminishing returns they can afford to try.