15 ms·
> The real issue is they tested on data in their training set. Hm, no. They trained on a part of their synthetic set and tested on another part of the set. Or
by littlestymaar 1y ago
> The real issue is they tested on data in their training set.
Hm, no.
They trained on a part of their synthetic set and tested on another part of the set. Or at least that's what they said they did:
> from which 1,000 were held out as a benchmark test set.
Emphasis mine.
- bangaladore 1y agoThanks, rereading it makes it clear that you are correct.
- _carltg 1y agoYes, but due to it being derived from the same underlying source dataset, it is effectively evaluating on the training dataset, not an independent validation/ test dataset. The difference is subtle but important. If we expect the model to truly outperform a general model, it should generalize to a completely independent set.