4 ms·

So, for people that aren't aware: There are a couple big shops that publish papers like this every year. The sole reason they do so is so that they can tell th

So, for people that aren't aware: There are a couple big shops that publish papers like this every year. The sole reason they do so is so that they can tell their managers and potential professional services customers that they have "the best ASR system". It's BS. IBM and Microsoft are among the most guilty. If you're seriously interested in ASR state-of-the-art performance, "systems" papers like this are largely nonsense. Usually they spend most of their time tuning hyperparameters to small publicly available datasets, to the point that the model will often not generalize well to real world settings because of how aggressively they've overfit the benchmark set.

ghusbands 9y ago
In a similar vein for IBM are releases claiming to have simulated a whole cat brain [1] and many claims about Watson [2]. In general, any claim that IBM has invented or done something useful, nowadays, should be taken with a pinch of salt. It's rarely anywhere near as good or complete as claimed. [1] http://www.lovemeow.com/ibm-cat-brain-simulation-a-hoax-1607973824.html http://www.lovemeow.com/ibm-cat-brain-simulation-a-hoax-1607... [2] https://www.linkedin.com/pulse/people-have-been-asking-why-ibms-lies-watson-upset-me-roger-schank https://www.linkedin.com/pulse/people-have-been-asking-why-i...
yorwba 9y ago
I do find it strange that their test WER is uniformly lower than their devset WER. I would expect validating on the devset to lead to the opposite effect, where the model overfits to the devset and performance degrades during test. Or do you mean by "tuning hyperparameters to small publicly available datasets" that the datasets do not have enough real-world variability and can be fit too easily? Are there no large, realistic datasets or are those just not available to the public?
- hiddencost 9y ago
  Even though they'll use the dev set to tune hyperparameters, they'll end up making a "is this a good model for the paper" decision based on the test set. So, e.g., they'll try different architectures, tune hyperparameters on the dev set, then decide whether to use that architecture by looking at the test set. So that's part of it. AKA "gradient descent via grad student" The other part is, as you say, relatively constrained data sets.

ghusbands 9y ago

In a similar vein for IBM are releases claiming to have simulated a whole cat brain [1] and many claims about Watson [2]. In general, any claim that IBM has invented or done something useful, nowadays, should be taken with a pinch of salt. It's rarely anywhere near as good or complete as claimed. [1] http://www.lovemeow.com/ibm-cat-brain-simulation-a-hoax-1607973824.html http://www.lovemeow.com/ibm-cat-brain-simulation-a-hoax-1607... [2] https://www.linkedin.com/pulse/people-have-been-asking-why-ibms-lies-watson-upset-me-roger-schank https://www.linkedin.com/pulse/people-have-been-asking-why-i...

yorwba 9y ago

I do find it strange that their test WER is uniformly lower than their devset WER. I would expect validating on the devset to lead to the opposite effect, where the model overfits to the devset and performance degrades during test. Or do you mean by "tuning hyperparameters to small publicly available datasets" that the datasets do not have enough real-world variability and can be fit too easily? Are there no large, realistic datasets or are those just not available to the public?

hiddencost 9y ago
Even though they'll use the dev set to tune hyperparameters, they'll end up making a "is this a good model for the paper" decision based on the test set. So, e.g., they'll try different architectures, tune hyperparameters on the dev set, then decide whether to use that architecture by looking at the test set. So that's part of it. AKA "gradient descent via grad student" The other part is, as you say, relatively constrained data sets.