3 ms·

>So every time you're amazed by something chat-gpt4 says, remember that soon this will be in your pocket. I want to believe you, but I'm ignorant of the hardwa

>So every time you're amazed by something chat-gpt4 says, remember that soon this will be in your pocket. I want to believe you, but I'm ignorant of the hardware requirements for these things. How soon do you think we'd be able to run something reasonably gpt4-like on, say, a 4090?

LightMachine 4y ago
I feel like no less than 10 years if the singularity doesn't kick in before that. Hardware and energy isn't progressing as fast as we'd like, and that is the main bottleneck. As in, imagine a world where we actually had the same computing power required to train (not run) GPT-4 in 1s in a phone? That kind of world is way beyond AGI and the cure of cancer IMO. Which is great, because it gives us a very objective goal to achieve these things. Sadly, I don't think we're nowhere near that. What was even the total energy consumption of GPT-4 training? Very hard to imagine computers will get that much better anytime soon. IIRC we have some kind of data that the a smartphone today has the power of the best supercomputer... of 30 years ago, right? Don't remember the source, sadly.
- waboremo 4y ago
  I don't think 10 years for training such large models on phones is entirely feasible, if only because phones are mainly concerned with power drain and ergonomics before anything else. Nobody is going to buy an iPhone 15 if it's the size of a brick and lasts 2 hours just because you're training&running models on it. The focus instead should be on running expansive models locally on your desktop system at home. This is a better focus in two ways: one the power issue is a non-concern (and AMD has already stated power usage will reach 500-700W average by 2025), two once you have it running locally the avenues of use open up such as being able to access your local model through other devices without the heavy burden on those devices.
- colordrops 4y ago
  Your comment makes me wonder if it's not a coincidence that as we seem to be the hitting a limit in hardware power that human level intelligence begins to emerge.
  - pmoriarty 4y ago
    Such hardware problems might be overcome with new computer architectures and/or substrates, like DNA computing, quantum computing, etc... Current AI's could help us overcome such limits.
bloaf 4y ago
It's literally already here: https://twitter.com/thiteanish/status/1635678053853536256 https://twitter.com/thiteanish/status/1635678053853536256
- int_19h 4y ago
  llama-7b is nowhere even close to GPT-4. Not even llama-65b is. We'll see if fine tuning can improve this, but e.g. alpaca-30b is still inferior to ChatGPT-3.5.

LightMachine 4y ago

I feel like no less than 10 years if the singularity doesn't kick in before that. Hardware and energy isn't progressing as fast as we'd like, and that is the main bottleneck. As in, imagine a world where we actually had the same computing power required to train (not run) GPT-4 in 1s in a phone? That kind of world is way beyond AGI and the cure of cancer IMO. Which is great, because it gives us a very objective goal to achieve these things. Sadly, I don't think we're nowhere near that. What was even the total energy consumption of GPT-4 training? Very hard to imagine computers will get that much better anytime soon. IIRC we have some kind of data that the a smartphone today has the power of the best supercomputer... of 30 years ago, right? Don't remember the source, sadly.

waboremo 4y ago
I don't think 10 years for training such large models on phones is entirely feasible, if only because phones are mainly concerned with power drain and ergonomics before anything else. Nobody is going to buy an iPhone 15 if it's the size of a brick and lasts 2 hours just because you're training&running models on it. The focus instead should be on running expansive models locally on your desktop system at home. This is a better focus in two ways: one the power issue is a non-concern (and AMD has already stated power usage will reach 500-700W average by 2025), two once you have it running locally the avenues of use open up such as being able to access your local model through other devices without the heavy burden on those devices.
colordrops 4y ago
Your comment makes me wonder if it's not a coincidence that as we seem to be the hitting a limit in hardware power that human level intelligence begins to emerge.
- pmoriarty 4y ago
  Such hardware problems might be overcome with new computer architectures and/or substrates, like DNA computing, quantum computing, etc... Current AI's could help us overcome such limits.

bloaf 4y ago

It's literally already here: https://twitter.com/thiteanish/status/1635678053853536256 https://twitter.com/thiteanish/status/1635678053853536256

int_19h 4y ago
llama-7b is nowhere even close to GPT-4. Not even llama-65b is. We'll see if fine tuning can improve this, but e.g. alpaca-30b is still inferior to ChatGPT-3.5.