3 ms·
I think that the best way to address this potential ARC overfitting, would be to create more benchmarks - that are similar in concept, focusing on fluid intelli
by jononor 1y ago
I think that the best way to address this potential ARC overfitting, would be to create more benchmarks - that are similar in concept, focusing on fluid intelligence, but from another angle than ARC.
Of course it is quite costly and also requires some "marketing" to actually get it established.
- shawntan 1y agoThis would not help if no proper constraints are established on what data can and cannot be trained on. And maybe just figuring out what the goal of the benchmark is. If it is to test generalisation capability, then what data the model being evaluated is trained on is crucial to making any conclusions. Look at the construction of this synthetic dataset for example: https://arxiv.org/pdf/1711.00350 https://arxiv.org/pdf/1711.00350