4 ms·
> given how inefficient machine learning seems to be We emulate neurons mathematically but it is possible to build efficient analog circuits that emulate them
by czl 3y ago
> given how inefficient machine learning seems to be
We emulate neurons mathematically but it is possible to build efficient analog circuits that emulate them physically.
I doubt any biological system can ever learn all the information gpt4 has learned. The gpt4 learning may not have been power efficient but neither were the first airplanes compared to birds yet today biological flight is rather limited compared to flight that uses technology.
Ever listen to Geoffrey Hinton speak about back propagation vs what biological systems use? Do you think he is wrong about this?
- imtringued 3y agoHe isn't wrong, but with current GPU hardware back propagation is actually the superior algorithm compared to his latest forward forward algorithm. Swapping the training algorithm doesn't get around the fact that you need to perform lots and lots of forward passes and those need an insane amount of memory bandwidth. Back propagation isn't a significant bottleneck in computers and also not in terms of memory capacity. So the only benefit of the biologically plausible forward forward algorithm is that you could run it on a digital or analog NPU, but with Ryzen AI coming in Strix Point, every high end laptop is going to have 50 TOPS of AI compute and 200GB/s of memory bandwidth. Nothing except large datacenter GPUs or a hypothetical 5090 with 32 GB would be competitive against that anytime soon. Anything smaller and you will have to buy several 3090 on eBay for like $900 a pop. Analog circuits are far off for now.
- namaria 3y ago> I doubt any biological system can ever learn all the information gpt4 has learned The fact that human brains can become competent at language with training datasets with a size of a tiny fraction of common crawl points to how much more efficient they are
- akasakahakada 3y agoUntil you realize that ChatGPT is also fluent in 20 other languages by including minimal data into the training dataset.
- namaria 3y agoMinimal as in the amount a person needs to learn the language or minimal as in tiny if compared to English in common crawl but still orders of magnitude more than a person goes through for infancy language acquisition? I think you know the difference.
- skybrian 3y agoIt’s anecdotal, but here [1] is an example of AI learning to translate a new language faster than a human could learn to do it, from a similar amount of data, using Claude 3 Opus. Translating a language isn’t quite the same as understanding it, but this is still impressively data-efficient. [1] https://twitter.com/hahahahohohe/status/1765088860592394250 https://twitter.com/hahahahohohe/status/1765088860592394250
- namaria 3y agoDid you read the "added context"? "The author has later responded and apologized a flawed / biased methodology which led them to believe that Opus has no prior knowledge of Circassian (i.e. wasn't trained on it) Apparently it was, and is able to speak Circassian just not perfectly"
- skybrian 3y agoI didn't see that! Thanks!
- akasakahakada 3y agoBro you are proven wrong. Baby FOV camera footage as the training dataset of LLM, the result is not bad. https://www.nyu.edu/about/news-publications/news/2024/february/ai-learns-through-the-eyes-and-ears-of-a-child.html https://www.nyu.edu/about/news-publications/news/2024/februa...
- Jensson 3y agoThe result is bad, that is just an image recognition model we know those can be trained with relatively little words and just a bunch of images, the model didn't learn grammar or sentences or relationships like a baby would. The only models we have that can do grammar and sentences and relationships are LLMs with massive amounts of data.