3 ms·
Show the training set, and PROVE that the tasks and answers aren't in there. I don't understand why this is not a default first step for proving that this is cr
by lionkor 1y ago
Show the training set, and PROVE that the tasks and answers aren't in there. I don't understand why this is not a default first step for proving that this is creating new knowledge.
- tananaev 1y agoHow can you actually verify it, even if they provide something?
- lionkor 1y agoThat's my point; you can't. They have no idea if their model came up with any of this or not.
- Davidzheng 1y agoWell that's harder than maybe solving well-known open problems (whose soln's are presumably not in training set lol) but it seems that their examples are not clearly breaking sota, especially on matmul
- thatguysaguy 1y agoAre you claiming that for the open problems they give record-breaking solutions for, there were just answers on the web waiting to be found?
- lionkor 1y agoNo, I'm saying they have a massive database of solutions (the training set) and don't even bother proving that their solution isn't in there. I'm not claiming something, they are failing to provide some necessary information here
- thatguysaguy 1y agoMany of the problems (such as the matmul and packing problems) have existing "scoreboards". It would be quite surprising if so many records had already been broken without it coming to the attention of the people who care about the problem.
- lionkor 1y agoNot really; people solve problems all the time, rarely do I check it a solution I worked hard on is known by some scoreboard
- thw09j9m 1y agoIt's Google. Assume the training set contains, as a subset, the entirety of all public digitized information. How would you like to them to share it?
- kookamamie 1y agoAs a tar, please.
- lionkor 1y agoIf they wanna do research where they claim they did something novel, without showing that they didn't just "look it up" in their massive training set, then yes, they should share what is and what isn't contained within.