4 ms·

So, this is a good idea. Having opinions about what AGI benchmarks should look like is a great way to argue about the kind of technology we want to build for th

So, this is a good idea. Having opinions about what AGI benchmarks should look like is a great way to argue about the kind of technology we want to build for the future. However, why are the 100 test tasks secret? I don't understand why how resisting “memorization” techniques requires it. Maybe someone can enlighten me.

muglug 2y ago
If the tasks were public then it would be trivial to have a human figure out the answers, and then to train an LLM to memorise those answers.
andoando 2y ago
Test date is always a secret no, otherwise you can train it on the test data and prod your algo to match the results closely as possible