I was thinking they might have "overfit" by picking hyper-parameters that happened by chance to result in a high CV score but wouldn't perform well in a holdout

I was thinking they might have "overfit" by picking hyper-parameters that happened by chance to result in a high CV score but wouldn't perform well in a holdout set. I could be wrong though, that's just my intuition.