4 ms·

At the PyData DE I just saw an excellent talk about GANs and data augmentation in image recognition: https://www.slideshare.net/FlorianWilhelm2/performance-eva

At the PyData DE I just saw an excellent talk about GANs and data augmentation in image recognition: https://www.slideshare.net/FlorianWilhelm2/performance-evaluation-of-gans-in-a-semisupervised-ocr-use-case https://www.slideshare.net/FlorianWilhelm2/performance-evalu... The authors were able to outperform Google ML by a large margin for a vision task that involved recognizing numbers from car registration documents. With just 160 manually collected training samples they were able to train a neural net that could recognize characters with 99.7 % accuracy. GoogleML performed very poorly in comparison, which I found very surprising because it didn't seem to be such a hard recognition task (clean, machine-written characters on a structured, green background).

deelowe 8y ago
Isn't that a bit like a synthetic benchmark though? This reads to me like those mongodb vs mysql comparisons that were made 7 years ago where they compared object store efficiencies for the two.
- ThePhysicist 8y ago
  I think it shows nicely that you can beat a "professional" ML toolkit for a specific problem. And most people don't want to solve many different ML problems but just a single one, so I think the result is quite interesting.
QML 8y ago
Isn’t this just the no free lunch theorem? Should you be expecting a more general framework to beat an specially trained algorithm? Another concern is generality: just because it performs well on this dataset does not mean it will perform well on another.
- ThePhysicist 8y ago
  It really doesn't matter that the model will probably not perform well on another dataset, as it was built for a specific task. It's also about flexibility: If Google ML doesn't provide you with a way to train their algorithms specifically for your use case it won't help you that they work well for generic text recognition tasks.

deelowe 8y ago

Isn't that a bit like a synthetic benchmark though? This reads to me like those mongodb vs mysql comparisons that were made 7 years ago where they compared object store efficiencies for the two.

ThePhysicist 8y ago
I think it shows nicely that you can beat a "professional" ML toolkit for a specific problem. And most people don't want to solve many different ML problems but just a single one, so I think the result is quite interesting.

QML 8y ago

Isn’t this just the no free lunch theorem? Should you be expecting a more general framework to beat an specially trained algorithm? Another concern is generality: just because it performs well on this dataset does not mean it will perform well on another.

ThePhysicist 8y ago
It really doesn't matter that the model will probably not perform well on another dataset, as it was built for a specific task. It's also about flexibility: If Google ML doesn't provide you with a way to train their algorithms specifically for your use case it won't help you that they work well for generic text recognition tasks.