4 ms·

ML benchmarks and scores (eg. FID, BLEU) correlate with model ability, but it's problematic to compare their absolute performance against humans. Convnets for e

ML benchmarks and scores (eg. FID, BLEU) correlate with model ability, but it's problematic to compare their absolute performance against humans. Convnets for example, are directly analogous to the visual cortex of humans. To get an apples to apples comparison you'd need to shut down all other areas of the brain, lest the human "cheat" by using deduction to figure out what's in the image, or using prior knowledge acquired outside the training data. imo the fact that ML models beat humans on benchmarks despite this handicap suggests that ANNs are better at absorbing and processing information compared to biology.

SiempreViernes 4y ago
Sure, and CCD chips are also better at recording information than the human visual cortex and and a shovel is better at digging holes than our hands: humans have built machines better than them at certain tasks for millennia. These machines have brought revolutions with them, but we've stayed human regardless.
t_mann 4y ago
It's not a given at all that being focused on one task alone should be a disadvantage at performing that specific task. A human might be reminded by a picture about what they need to organize for their kid's birthday party, start wondering why they're doing this exercise, or might become distracted in a million other ways.
- Jack000 4y ago
  Single-task ANNs like convnets are really quite different from the human brain as a whole. Without the "rest of the brain" it fundamentally doesn't understand that the pixels it sees are representations of a 3d world, with discrete objects and the passage of time.
jstx1 4y ago
I don’t think it’s problematic. Benchmarking to humans isn’t for the sake of an apples to apples comparison - that would be impossible since the human brain doesn’t work like a convnet, not even close. It’s because human performance gives a good baseline by showing what’s possible; and also because for some tasks you might want to replace humans with algorithms. It’s much more about being pragmatic than being fair.
- Jack000 4y ago
  the brain as a whole doesn't work the same way as ANNs, but the visual cortex in particular is extremely similar to a convnet. We even know which layers are responsible for which features, on a coarse scale: https://www.sciencedirect.com/science/article/pii/S089662730700774X https://www.sciencedirect.com/science/article/pii/S089662730... the visual cortex uses local receptive fields to organize features in a hierarchical manner, exactly like a convnet.

SiempreViernes 4y ago

Sure, and CCD chips are also better at recording information than the human visual cortex and and a shovel is better at digging holes than our hands: humans have built machines better than them at certain tasks for millennia. These machines have brought revolutions with them, but we've stayed human regardless.

t_mann 4y ago

It's not a given at all that being focused on one task alone should be a disadvantage at performing that specific task. A human might be reminded by a picture about what they need to organize for their kid's birthday party, start wondering why they're doing this exercise, or might become distracted in a million other ways.

Jack000 4y ago
Single-task ANNs like convnets are really quite different from the human brain as a whole. Without the "rest of the brain" it fundamentally doesn't understand that the pixels it sees are representations of a 3d world, with discrete objects and the passage of time.

jstx1 4y ago

I don’t think it’s problematic. Benchmarking to humans isn’t for the sake of an apples to apples comparison - that would be impossible since the human brain doesn’t work like a convnet, not even close. It’s because human performance gives a good baseline by showing what’s possible; and also because for some tasks you might want to replace humans with algorithms. It’s much more about being pragmatic than being fair.

Jack000 4y ago
the brain as a whole doesn't work the same way as ANNs, but the visual cortex in particular is extremely similar to a convnet. We even know which layers are responsible for which features, on a coarse scale: https://www.sciencedirect.com/science/article/pii/S089662730700774X https://www.sciencedirect.com/science/article/pii/S089662730... the visual cortex uses local receptive fields to organize features in a hierarchical manner, exactly like a convnet.