3 ms·
im very naive here but does anyone trust these benchmarks? do they mean anything to you? they seem far too easy to game and it doesn't feel like its an accurate
by czk 2y ago
im very naive here but does anyone trust these benchmarks? do they mean anything to you? they seem far too easy to game and it doesn't feel like its an accurate way to really tell how these models compare to one another. seems like benchmark performance declines quite a bit if you introduce a problem that's similar to those in benchmarks but one that the model hasn't seen before
- javaunsafe2019 2y agoSomeone could read this as: as soon as I don't like the results I doubt the test...
- czk 2y agothats far from the case here i really have no skin in the game...
- thorum 2y agoAIME has significant problems: https://x.com/DimitrisPapail/status/1888325914603516214 https://x.com/DimitrisPapail/status/1888325914603516214 > Problems near identical to the test set can be found online.
- janalsncm 2y agoI don’t see how that is relevant to a model trained with RL. If it had already memorized the answers then why was any training necessary?