6 ms·
I do find it strange that their test WER is uniformly lower than their devset WER. I would expect validating on the devset to lead to the opposite effect, where
by yorwba 9y ago
I do find it strange that their test WER is uniformly lower than their devset WER. I would expect validating on the devset to lead to the opposite effect, where the model overfits to the devset and performance degrades during test.
Or do you mean by "tuning hyperparameters to small publicly available datasets" that the datasets do not have enough real-world variability and can be fit too easily? Are there no large, realistic datasets or are those just not available to the public?
- hiddencost 9y agoEven though they'll use the dev set to tune hyperparameters, they'll end up making a "is this a good model for the paper" decision based on the test set. So, e.g., they'll try different architectures, tune hyperparameters on the dev set, then decide whether to use that architecture by looking at the test set. So that's part of it. AKA "gradient descent via grad student" The other part is, as you say, relatively constrained data sets.