4 ms·

This is what my dissertation was about, in a different field: Everyone using the same datasets, even though they are severely flawed. As it turned out, differe

This is what my dissertation was about, in a different field: Everyone using the same datasets, even though they are severely flawed. As it turned out, differences between datasets proved significantly larger (by a big margin) than differences between algorithms. And the most popular datasets in fact included biases and eccentricities that were bound to cause such problems. https://bastibe.github.io/Dissertation-Website/ https://bastibe.github.io/Dissertation-Website/ (figures 12.9-12.11, if you're interested)

fock 5y ago
thank you! I also found the same thing during my PhD and it took a basic-pay-topublish paper and 2 years to get my supervisor to agree that this is not really going anywhere with publications and switch to a field, where I am more comfortable with producing publications. Essentially my field has 1 big dataset, and then domain experts "improve" models by creating their own data (200 samples) and then claim a "novel" method (which is not transferable at all). The pay2publish-paper was sent back by 2 journals with reviews indicating exactly "our method is better, just use the right (our) thing" (which I _all_ refuted for the professor by doing it, but the editor wouldn't hear anymore...). And then there's papers predicting features through a complex preprocessing pipeline in these journals. Academia and big companies are just idiotic.
- Derbasti 5y ago
  Funny that you would mention trying to publish such findings in journals. I tried... more than two times, as well. Rejected with the most spurious of claims, despite my presenting evidence disproving them. Frankly, these journal submissions were the most frustrating and trying experiences in my professional life. Oh well. I'm glad I got my degree, and could leave academia relatively unharmed. Now I work in AI (engineering!), where most science is highschool-level "hey, I tried stuff, and things happened. Dunno why, though". It's just ridiculous.

fock 5y ago

thank you! I also found the same thing during my PhD and it took a basic-pay-topublish paper and 2 years to get my supervisor to agree that this is not really going anywhere with publications and switch to a field, where I am more comfortable with producing publications. Essentially my field has 1 big dataset, and then domain experts "improve" models by creating their own data (200 samples) and then claim a "novel" method (which is not transferable at all). The pay2publish-paper was sent back by 2 journals with reviews indicating exactly "our method is better, just use the right (our) thing" (which I _all_ refuted for the professor by doing it, but the editor wouldn't hear anymore...). And then there's papers predicting features through a complex preprocessing pipeline in these journals. Academia and big companies are just idiotic.

Derbasti 5y ago
Funny that you would mention trying to publish such findings in journals. I tried... more than two times, as well. Rejected with the most spurious of claims, despite my presenting evidence disproving them. Frankly, these journal submissions were the most frustrating and trying experiences in my professional life. Oh well. I'm glad I got my degree, and could leave academia relatively unharmed. Now I work in AI (engineering!), where most science is highschool-level "hey, I tried stuff, and things happened. Dunno why, though". It's just ridiculous.