3 ms·

Your interesting take does ignore the fundamental point however that the human brain requires both far less energy and far less training data to learn. The take

Your interesting take does ignore the fundamental point however that the human brain requires both far less energy and far less training data to learn. The takeaway for me is that there is huge scope to improve our machine learning techniques.

lajamerr 4y ago
That energy efficiency was paid for by the total energy of the biological chain of events that eventually led to that human in the form of their DNA. We humans don't start from a blank/random slate like ANNs when we are born. We might be efficient in some regards but we surely didn't get there efficiently.
- hgomersall 4y ago
  And nor should ML systems be a blank slate, which is really my point.
boloust 4y ago
I didn't really want to get into the specifics of the article, since I believe the entire analysis is fundamentally misguided. But I think the article's specific claims about image recognition (they use the example of identifying pictures of cats) are completely false: - The article claims models must be retrained from scratch if they are to identify cartoon cats instead of real cats. This is absolutely false and transfer learning is old hat by machine learning standards. - The article cites the fact that an image recognition model "requires many weights and lots of multiplication", as an example of inefficiency, as if human brains don't encode trillions of weights in the connections between their billions of neurons. - If AI models could only identify a few hundred photos per hour, this would undoubtedly be used as evidence of inefficiency. By the same token, if we consider the cost of training amortized over the subsequent cost of image identification, image recognition models are are far more efficient at scale than humans, even if we grant massively overstated estimates of training costs. - All humans must go through an incredibly expensive, labor-intensive training process, before which they will completely fail to reliably identify a cat. - Humans required billions of years, the death and suffering of trillions of organisms, the terraforming of an entire planet, and vast amounts of energy to run a stupidly inefficient Monte Carlo "pretraining" phase. I can't help but come away with the feeling that I've been trolled. They managed to pick image recognition, an area where efficient AI models approach or exceed human performance, to prop up an absurdly myopic comparison that wouldn't make sense even if they got any of the details right.

lajamerr 4y ago

That energy efficiency was paid for by the total energy of the biological chain of events that eventually led to that human in the form of their DNA. We humans don't start from a blank/random slate like ANNs when we are born. We might be efficient in some regards but we surely didn't get there efficiently.

hgomersall 4y ago
And nor should ML systems be a blank slate, which is really my point.

boloust 4y ago

I didn't really want to get into the specifics of the article, since I believe the entire analysis is fundamentally misguided. But I think the article's specific claims about image recognition (they use the example of identifying pictures of cats) are completely false: - The article claims models must be retrained from scratch if they are to identify cartoon cats instead of real cats. This is absolutely false and transfer learning is old hat by machine learning standards. - The article cites the fact that an image recognition model "requires many weights and lots of multiplication", as an example of inefficiency, as if human brains don't encode trillions of weights in the connections between their billions of neurons. - If AI models could only identify a few hundred photos per hour, this would undoubtedly be used as evidence of inefficiency. By the same token, if we consider the cost of training amortized over the subsequent cost of image identification, image recognition models are are far more efficient at scale than humans, even if we grant massively overstated estimates of training costs. - All humans must go through an incredibly expensive, labor-intensive training process, before which they will completely fail to reliably identify a cat. - Humans required billions of years, the death and suffering of trillions of organisms, the terraforming of an entire planet, and vast amounts of energy to run a stupidly inefficient Monte Carlo "pretraining" phase. I can't help but come away with the feeling that I've been trolled. They managed to pick image recognition, an area where efficient AI models approach or exceed human performance, to prop up an absurdly myopic comparison that wouldn't make sense even if they got any of the details right.