You know what they say about LLM benchmarks nowadays? Pre-training on the test set is all you need. https://arxiv.org/abs/2309.08632 https://arxiv.org/abs/2309

You know what they say about LLM benchmarks nowadays? Pre-training on the test set is all you need. https://arxiv.org/abs/2309.08632 https://arxiv.org/abs/2309.08632 Many of the modern LLMs take entire copy of internet which includes the test set for many of these benchmarks. So if someone claims to beat ChatGPT and their model is trained on the test set, ofcourse they’ll do better. Even ChatGPT is likely trained on the test set. Even a hash table will get stellar results if trained on the test set. Their website provides no evidence that it did not train on the test set. Until we get to play with it, their claims are to be taken with a grain of salt.