3 ms·

The future is not known to us. But given how inefficient machine learning seems to be, algorithmic efficiency improvements may keep the scaling going for a whil

The future is not known to us. But given how inefficient machine learning seems to be, algorithmic efficiency improvements may keep the scaling going for a while? Maybe that's not a "major breakthrough" but it's improvement nonetheless. It's also going to take a while to learn to use the new toys we already have.

rented_mule 3y ago
I'd love to see more algorithmic efficiency in ML. But hasn't ML been going in the other direction for at least a decade now? It seems to me that it's been going aggressively towards brute force algorithms. Specifically, figuring out how to do as much matrix multiplication as possible.
- skybrian 3y ago
  Among other things, there seems to be progress in fixing attention to scale better.
czl 3y ago
> given how inefficient machine learning seems to be We emulate neurons mathematically but it is possible to build efficient analog circuits that emulate them physically. I doubt any biological system can ever learn all the information gpt4 has learned. The gpt4 learning may not have been power efficient but neither were the first airplanes compared to birds yet today biological flight is rather limited compared to flight that uses technology. Ever listen to Geoffrey Hinton speak about back propagation vs what biological systems use? Do you think he is wrong about this?
- imtringued 3y ago
  He isn't wrong, but with current GPU hardware back propagation is actually the superior algorithm compared to his latest forward forward algorithm. Swapping the training algorithm doesn't get around the fact that you need to perform lots and lots of forward passes and those need an insane amount of memory bandwidth. Back propagation isn't a significant bottleneck in computers and also not in terms of memory capacity. So the only benefit of the biologically plausible forward forward algorithm is that you could run it on a digital or analog NPU, but with Ryzen AI coming in Strix Point, every high end laptop is going to have 50 TOPS of AI compute and 200GB/s of memory bandwidth. Nothing except large datacenter GPUs or a hypothetical 5090 with 32 GB would be competitive against that anytime soon. Anything smaller and you will have to buy several 3090 on eBay for like $900 a pop. Analog circuits are far off for now.
- namaria 3y ago
  > I doubt any biological system can ever learn all the information gpt4 has learned The fact that human brains can become competent at language with training datasets with a size of a tiny fraction of common crawl points to how much more efficient they are
  - akasakahakada 3y ago
    Until you realize that ChatGPT is also fluent in 20 other languages by including minimal data into the training dataset.
    - namaria 3y ago
      Minimal as in the amount a person needs to learn the language or minimal as in tiny if compared to English in common crawl but still orders of magnitude more than a person goes through for infancy language acquisition? I think you know the difference.
      - skybrian 3y ago
        It’s anecdotal, but here [1] is an example of AI learning to translate a new language faster than a human could learn to do it, from a similar amount of data, using Claude 3 Opus. Translating a language isn’t quite the same as understanding it, but this is still impressively data-efficient. [1] https://twitter.com/hahahahohohe/status/1765088860592394250 https://twitter.com/hahahahohohe/status/1765088860592394250
        namaria 3y ago
        Did you read the "added context"? "The author has later responded and apologized a flawed / biased methodology which led them to believe that Opus has no prior knowledge of Circassian (i.e. wasn't trained on it) Apparently it was, and is able to speak Circassian just not perfectly"
        skybrian 3y ago

rented_mule 3y ago

I'd love to see more algorithmic efficiency in ML. But hasn't ML been going in the other direction for at least a decade now? It seems to me that it's been going aggressively towards brute force algorithms. Specifically, figuring out how to do as much matrix multiplication as possible.

skybrian 3y ago
Among other things, there seems to be progress in fixing attention to scale better.

czl 3y ago

> given how inefficient machine learning seems to be We emulate neurons mathematically but it is possible to build efficient analog circuits that emulate them physically. I doubt any biological system can ever learn all the information gpt4 has learned. The gpt4 learning may not have been power efficient but neither were the first airplanes compared to birds yet today biological flight is rather limited compared to flight that uses technology. Ever listen to Geoffrey Hinton speak about back propagation vs what biological systems use? Do you think he is wrong about this?

imtringued 3y ago
He isn't wrong, but with current GPU hardware back propagation is actually the superior algorithm compared to his latest forward forward algorithm. Swapping the training algorithm doesn't get around the fact that you need to perform lots and lots of forward passes and those need an insane amount of memory bandwidth. Back propagation isn't a significant bottleneck in computers and also not in terms of memory capacity. So the only benefit of the biologically plausible forward forward algorithm is that you could run it on a digital or analog NPU, but with Ryzen AI coming in Strix Point, every high end laptop is going to have 50 TOPS of AI compute and 200GB/s of memory bandwidth. Nothing except large datacenter GPUs or a hypothetical 5090 with 32 GB would be competitive against that anytime soon. Anything smaller and you will have to buy several 3090 on eBay for like $900 a pop. Analog circuits are far off for now.
namaria 3y ago
> I doubt any biological system can ever learn all the information gpt4 has learned The fact that human brains can become competent at language with training datasets with a size of a tiny fraction of common crawl points to how much more efficient they are
- akasakahakada 3y ago
  Until you realize that ChatGPT is also fluent in 20 other languages by including minimal data into the training dataset.
  - namaria 3y ago
    Minimal as in the amount a person needs to learn the language or minimal as in tiny if compared to English in common crawl but still orders of magnitude more than a person goes through for infancy language acquisition? I think you know the difference.
    - skybrian 3y ago
      It’s anecdotal, but here [1] is an example of AI learning to translate a new language faster than a human could learn to do it, from a similar amount of data, using Claude 3 Opus. Translating a language isn’t quite the same as understanding it, but this is still impressively data-efficient. [1] https://twitter.com/hahahahohohe/status/1765088860592394250 https://twitter.com/hahahahohohe/status/1765088860592394250
      - namaria 3y ago
        Did you read the "added context"? "The author has later responded and apologized a flawed / biased methodology which led them to believe that Opus has no prior knowledge of Circassian (i.e. wasn't trained on it) Apparently it was, and is able to speak Circassian just not perfectly"
        skybrian 3y ago