4 ms·
I agree that for any given test, you could build a specific pipeline to optimize for that test. I supposed that's why it is helpful to have many tests. Howeve
by tylervigen 11mo ago
I agree that for any given test, you could build a specific pipeline to optimize for that test. I supposed that's why it is helpful to have many tests.
However, many people have worked hard to optimize tools specifically for ARC over many years, and it's proven to be a particularly hard test to optimize for. This is why I find it so interesting that LLMs can do it well at all, regardless of whether tests like it are included in training.
- stephc_int13 11mo agoThe real strength of current neural nets/transformers relies on huge datasets. ARC do not provide this kind of dataset, only a small public one and a private one where they do the benchmarks. Building your own large private ARC set does not seem too difficult if you have enough resources.
- deleted 11mo ago[deleted]
- egeozcan 11mo agoHow can they keep it private? It's not like they can run these models locally. Do the providers promise not to peak when they are testing?