3 ms·

Don't you think something is missing if we've defined "quality" as an independent and uncorrelated characteristic from importance or influentiality?

by johnfn 3y ago

SubiculumCode 3y ago
When an author refuses to address reasonable questions by the reviewers, what should you expect? There were legitimate questions and concerns about potential alternative explanations for the increase in accuracy raised by the reviewers, and the authors didn't play ball.
nicklecompte 3y ago
No, because "quality" means two different things here. I believe the main reason word2vec became important was purely on the software/engineering side, not because it was scientifically novel. Advancements in Python development, especially good higher-level constructs around numerical linear algebra, meant that the fairly shallow and simple tools of word2vec were available to almost every tech company. So I don't think word2vec was (or is) particularly good research, but it became good software for reasons beyond its own control In 2016 or so it was proven that word2vec is equivalent to the pointwise mutual information between the words in its training set, after doing some preprocessing. This means that Claude Shannon had things mostly figured out in the 60s, and some reviewers were quite critical of the word2vec paper for not citing similar developments in the 70s.
nybsjytm 3y ago
Quality in the sense I meant it (cogency and intellectual depth/rigor) should certainly be correlated with importance and influence!
godelski 3y ago
Yes and no. I think the larger issue is about the ambiguity of what publications mean and should be. There's a lot of ways to optimize this, and none of those has optimal solutions. I don't think you should be down voted for your different framing because I think we just need to be more open about this chaos and consider other values than our own or the road we're on. I think it is very unclear what we are trying to optimize and it is quite apparent that you're going to have many opinions on this and your group of reviewers may all seek to optimize different things. The only solution I can see is to stop pretending as if we know what each other is trying to do and be a bit more explicit about it. Because if we argue based on different assumptions we'll talk past one another if we assume the other is working on the same set of assumptions.
dr_kiszonka 3y ago
I believe many journals focusing on potentially influential papers is why we have a reproducibility crisis. Since it is very hard to publish null results, people often don't even bother trying. This leads to tons of wasted effort as multiple groups attempt the same thing not knowing that others before them have failed. Also, predicting whether a paper will be influential is very hard and inherently subjective, unless you are reviewing something truly groundbreaking. Quality-based reviews are also subjective, but less so.

Don't you think something is missing if we've defined "quality" as an independent and uncorrelated characteristic from importance or influentiality?

by johnfn 3y ago

Don't you think something is missing if we've defined "quality" as an independent and uncorrelated characteristic from importance or influentiality?

SubiculumCode 3y ago

When an author refuses to address reasonable questions by the reviewers, what should you expect? There were legitimate questions and concerns about potential alternative explanations for the increase in accuracy raised by the reviewers, and the authors didn't play ball.

nicklecompte 3y ago

No, because "quality" means two different things here. I believe the main reason word2vec became important was purely on the software/engineering side, not because it was scientifically novel. Advancements in Python development, especially good higher-level constructs around numerical linear algebra, meant that the fairly shallow and simple tools of word2vec were available to almost every tech company. So I don't think word2vec was (or is) particularly good research, but it became good software for reasons beyond its own control In 2016 or so it was proven that word2vec is equivalent to the pointwise mutual information between the words in its training set, after doing some preprocessing. This means that Claude Shannon had things mostly figured out in the 60s, and some reviewers were quite critical of the word2vec paper for not citing similar developments in the 70s.

nybsjytm 3y ago

Quality in the sense I meant it (cogency and intellectual depth/rigor) should certainly be correlated with importance and influence!

godelski 3y ago

Yes and no. I think the larger issue is about the ambiguity of what publications mean and should be. There's a lot of ways to optimize this, and none of those has optimal solutions. I don't think you should be down voted for your different framing because I think we just need to be more open about this chaos and consider other values than our own or the road we're on. I think it is very unclear what we are trying to optimize and it is quite apparent that you're going to have many opinions on this and your group of reviewers may all seek to optimize different things. The only solution I can see is to stop pretending as if we know what each other is trying to do and be a bit more explicit about it. Because if we argue based on different assumptions we'll talk past one another if we assume the other is working on the same set of assumptions.

dr_kiszonka 3y ago

I believe many journals focusing on potentially influential papers is why we have a reproducibility crisis. Since it is very hard to publish null results, people often don't even bother trying. This leads to tons of wasted effort as multiple groups attempt the same thing not knowing that others before them have failed. Also, predicting whether a paper will be influential is very hard and inherently subjective, unless you are reviewing something truly groundbreaking. Quality-based reviews are also subjective, but less so.