3 ms·

> Here's the thing, current computational power isn't enough to achieve the scale of model that would be required for "god like" AI. I don't see how that is th

> Here's the thing, current computational power isn't enough to achieve the scale of model that would be required for "god like" AI. I don't see how that is the case at all. The human brain only needs about 100 watts. A GPT-3 instance only needs a few GPUs and maybe 1000 watts. GPT-4 is presumably in the same ballpark, let's say 10000 watts to be generous (even though it's probably closer to 1000 too), but still small enough to fit inside a full height rack. Many datacenters are megawatt-level. Seeing how GPT-4 is already human-level at many tasks and watt-to-subjective-capability increases seem to grow linearly at worst, an AI could be ran in a single data center and perform 100x GPT-4 abilities. I'm not sure what 10x, much less 100x GPT-4 abilities would be, but that sure sounds like godlike or near-god to me.

verdverm 4y ago
Simply scaling up the current LLM models will not fix the fundamental problems of hallucination and lack of planing abilities. If they cost $10M+ to train, will scaling that cost 100X (to $1B) really make sense. I find it doubtful that the ROI is there Consider that GPT4 is 15-30X the cost of GPT3.5, is it really 15-30X better? Not in my experience. I would expect that we are approaching the point of diminishing returns with text based LLMs, at least for the current architectures and training regimes. I'm interested to see what the Large Multi-Modal Models produce, but I suspect the planning & lying will still be there. Chaining these models will increase the cost even more, without certainty that it will produce usable results for anything beyond simple, context limited tasks.
- rbanffy 4y ago
  When it becomes that large and expensive, it makes sense for governments to do it and open up the result so everyone can benefit.
  - verdverm 4y ago
    That also makes sense when we account for how gov'ts are really good at wasting money for below average results... This is not something I would trust them to do competently or within a reasonable budget. If it was done through the NSF or DARPA, with an industry partner, maybe? But I can't imagine the current politicians and them arguing about if & how to put restraints on it. It will be interesting to see how the CCP goes about this. Maybe the EU can offer a third alternative?
    - rbanffy 4y ago
      > That also makes sense when we account for how gov'ts are really good at wasting money for below average results... The goal of governments is not efficiency, but fairness and strategic action aiming at long-term results. I like to joke that one of the key goals of NASA is to prevent rocket scientists from moving to North Korea.
- flangola7 4y ago
  15-30x more expensive doesn't necessarily mean it actually requires 15-30x more computation. I'm certain GPT-4 is extremely profitable; they can charge whatever they want because nothing else like it exists. 3.5 turbo also isn't GPT-3/Davinci, which is still more expensive and often better for non chatbot functions. Additionally, 4 definitely provides at least 15x more value for me than 3.5. Turbo is a toy by comparison. The differences on complex tasks are subtle but profound. OpenAI just received $10B and had to pause new paid subscriptions because more people were signing up than their hardware could handle. Even if they think it is diminishing returns they can afford to try.

verdverm 4y ago

Simply scaling up the current LLM models will not fix the fundamental problems of hallucination and lack of planing abilities. If they cost $10M+ to train, will scaling that cost 100X (to $1B) really make sense. I find it doubtful that the ROI is there Consider that GPT4 is 15-30X the cost of GPT3.5, is it really 15-30X better? Not in my experience. I would expect that we are approaching the point of diminishing returns with text based LLMs, at least for the current architectures and training regimes. I'm interested to see what the Large Multi-Modal Models produce, but I suspect the planning & lying will still be there. Chaining these models will increase the cost even more, without certainty that it will produce usable results for anything beyond simple, context limited tasks.

rbanffy 4y ago
When it becomes that large and expensive, it makes sense for governments to do it and open up the result so everyone can benefit.
- verdverm 4y ago
  That also makes sense when we account for how gov'ts are really good at wasting money for below average results... This is not something I would trust them to do competently or within a reasonable budget. If it was done through the NSF or DARPA, with an industry partner, maybe? But I can't imagine the current politicians and them arguing about if & how to put restraints on it. It will be interesting to see how the CCP goes about this. Maybe the EU can offer a third alternative?
  - rbanffy 4y ago
    > That also makes sense when we account for how gov'ts are really good at wasting money for below average results... The goal of governments is not efficiency, but fairness and strategic action aiming at long-term results. I like to joke that one of the key goals of NASA is to prevent rocket scientists from moving to North Korea.
flangola7 4y ago
15-30x more expensive doesn't necessarily mean it actually requires 15-30x more computation. I'm certain GPT-4 is extremely profitable; they can charge whatever they want because nothing else like it exists. 3.5 turbo also isn't GPT-3/Davinci, which is still more expensive and often better for non chatbot functions. Additionally, 4 definitely provides at least 15x more value for me than 3.5. Turbo is a toy by comparison. The differences on complex tasks are subtle but profound. OpenAI just received $10B and had to pause new paid subscriptions because more people were signing up than their hardware could handle. Even if they think it is diminishing returns they can afford to try.