4 ms·

I guess this LoRA is the missing piece. NVIDIA stated recently that GPT bots will become one million times more powerful in ten years. Many people doubted that

I guess this LoRA is the missing piece. NVIDIA stated recently that GPT bots will become one million times more powerful in ten years. Many people doubted that. With LoRA, I see a much higher improvement. These guys claim a 10000 times reduction in parameter size. A different way to look at it, is that with the current hardware you can train a model that has 10000 times more parameters. If you add a 100x improvement in hardware in 10 years (not at all unrealistic), that's the million. But we will have significant improvements in training methods too.

flangola7 4y ago
Where do you find 10,000 more data?
- nathanasmith 4y ago
  Audio, video, and LLMs enabled for real world interaction will pave part of the way for sure.
  - flangola7 4y ago
    They have already ingested most of the internet and all popular and semi popular books. If your plan doesn't involve every person on Earth wearing a go pro and uploading it to OpenAI you will have difficulty finding 9,999 more internets and libraries of congress.
    - evilduck 4y ago
      Seems like Google might have the advantage, with all of YouTube to mine.
      - flangola7 4y ago
        Is that really a Google advantage? Anytime can watch or download YouTube videos.
- credit_guy 4y ago
  You don't need 10,000 more data to train a model with 10,000 more parameters. You can use the same data and the model will perform better. Much better. The problem is that currently it is nearly impossible to run and train the model with more parameters. ChatGPT has nearly 200 billion parameters. GPT-4 we don't know, there are rumors that it has 100 trillion parameters, but they are probably unfounded. In any case, we've seen how much more powerful GPT-4 is. Imagine a GPT-5 with 1 quadrillion parameters. And then imagine that after you've trained it to some reasonable level, you "downsample" the parameters, using the SVD approach described in LoRA, and get a GPT-5 with the same 200 billion parameters as ChatGPT, but with many, many times more power, even than GPT-4.
  - noonething 4y ago
    Parameters here are just like object attributes, correct?

flangola7 4y ago

Where do you find 10,000 more data?

nathanasmith 4y ago
Audio, video, and LLMs enabled for real world interaction will pave part of the way for sure.
- flangola7 4y ago
  They have already ingested most of the internet and all popular and semi popular books. If your plan doesn't involve every person on Earth wearing a go pro and uploading it to OpenAI you will have difficulty finding 9,999 more internets and libraries of congress.
  - evilduck 4y ago
    Seems like Google might have the advantage, with all of YouTube to mine.
    - flangola7 4y ago
      Is that really a Google advantage? Anytime can watch or download YouTube videos.
credit_guy 4y ago
You don't need 10,000 more data to train a model with 10,000 more parameters. You can use the same data and the model will perform better. Much better. The problem is that currently it is nearly impossible to run and train the model with more parameters. ChatGPT has nearly 200 billion parameters. GPT-4 we don't know, there are rumors that it has 100 trillion parameters, but they are probably unfounded. In any case, we've seen how much more powerful GPT-4 is. Imagine a GPT-5 with 1 quadrillion parameters. And then imagine that after you've trained it to some reasonable level, you "downsample" the parameters, using the SVD approach described in LoRA, and get a GPT-5 with the same 200 billion parameters as ChatGPT, but with many, many times more power, even than GPT-4.
- noonething 4y ago
  Parameters here are just like object attributes, correct?