3 ms·

>I also think most of the accuracy came from memorization of training set Oh yes..it memorized a 20 digit training set to solve 100 digit problems. That makes

>I also think most of the accuracy came from memorization of training set Oh yes..it memorized a 20 digit training set to solve 100 digit problems. That makes sense. Lol >(model didn't provide intermediate results, and started failing significantly at sligtly larger input). No it didn't. They tested up to 100 digits with very high accuracy. I don't think you even read the abstract of this, nevermind the actual paper.

riku_iki 2y ago
> No it didn't. They tested up to 100 digits with very high accuracy. I don't think you even read the abstract of this, nevermind the actual paper. they have two OOD (out of distribution) accuracies in the paper: OOD: up to 100 digits, and 100+ OOD: 100-160 digits. 100+ OOD accuracy is significantly worse: around 30%.