7 ms·

> (...) in fact, many prefer the status quo, usually those who rely on prestige over actual quality to advance their careers. Your comment doesn't read like on

> (...) in fact, many prefer the status quo, usually those who rely on prestige over actual quality to advance their careers. Your comment doesn't read like one from anyone with any relationship with academia. If you had, you'd know that the issue is not a vacuous "prestige" but funding being dependent on hard metrics such as impact factor, and in some cases with metrics being collected exclusively from a set of established peer-reviewed journals that must be whitelisted. And ArXiv is not one of them. This means that a big share of academia has their professional and future, as well as their institution's ability to raise funding, dependent on them publishing on a small set of non-open peer-reviewed journals. Reading your post, you make it sound like anyone can just upload a random PDF to a random file server and call it a publication. That ain't it. If you fail to understand the problem, you certainly ain't the solution.

dguest 2y ago
I all fairness, I don't think the grandparent post disagrees with anything that is in the parent post here. Yes, academia has tried to quantify prestige via impact factor and peer-reviewed journals. Yes, lots of people (even in Academia) feel that the system is being gamed, with by the publishing houses that own the journals being a common scapegoat. The system isn't broken, but it also keeps its integrity through some dynamic tension: a bit of criticism is a good thing.
JadeNB 2y ago
> And ArXiv is not one of them. But putting your papers on the arXiv, as your parent said, doesn't mean you only put them on the arXiv. I put all my papers on the arXiv, but I also submit them for publication in journals that will help me make the case for funding and promotion.
BeetleB 2y ago
> Your comment doesn't read like one from anyone with any relationship with academia. Your comment reads likewise. He didn't say he publishes them exclusively on Arxiv. It's quite common for professors to post it there as well as submit to journals. Many (most?) journals allow for it - they don't insist the ones in arxiv be taken down - as long as they're posting preprints and not the final (copyrighted) version. As an academic, you should also know that practices vary widely with discipline. As an example: > dependent on them publishing on a small set of non-open peer-reviewed journals. IIRC, NIH grants require publishing in open peer-reviewed journals. Also, lots of disciplines are not heavily reliant on funding. In both universities I attended, the bulk of math professors did not even apply for grants! It's not required to get tenure (unlike engineering/physics). Also often true in some economics departments. As an aside, your comment violates a number of HN guidelines.
- godelski 2y ago
  > Your comment reads likewise. FWIW I thought it read like reviewer 2, which makes me actually think they have a relationship with academia. The problem I have with their comment is that it rejects the critique by pointing to a different issue. As if there is a singular issue with academia that leads to the mess. So it comes off as a typical "reviewer 2" comment to me where the is more complaint and disagreement than critique. FWIW, I think we in academia need to embrace the fuzziness and noise of evaluation. I think the issue is that by trying to make sense out of an extremely noisy process we placed too much value in the (still noisy) signals we can use. It is a problem to think that these are objective answers and deny the existence of Goodhart's Law (this is especially ironic in ML where any intro class discusses reward hacking). And in this, I think there's a strong coupling between cgshep's and chipdart's complaints. As for publishing, I think we also lost sight of the main reason we publish: to communicate with our peers. Publishers played an important role since not even a century ago we could not make our works trivially available to our peers. But now the problem is information overload, not lack of information. And I think in part the review process is getting worse and worse each year as we place so little value on the act of reviewing, do not hold anyone accountable for a low quality review[0,1], do not hold ACs or Metas accountable for, and we have so many papers to review that I don't think we can expect high quality reviewing even if we actually incentivized it. I mean in ML we expect a few hundred ACs to oversee over ten thousand submissions? My question is if we'll learn that the noisiness of the process and the randomness of rejection creates a negative feedback of papers. Where you "roll the dice" on the next conference. You resubmit without changes, as well as your new work (publish or perish baby). If we had quality reviewing at least this would push pressure for papers to get incrementally better instead of just being recycled. But recycling is a very efficient strategy right now and we've seen plenty of data to suggest it is. [0] I understand the reasons for this. It is a tricky problem [1] I'd actually argue we incentivize bad reviews. No one questions you when you reject a work and finding reasons to reject a work are far easier to accept one. There's always legitimate reasons to reject any work. Not to mention that the whole process is zero sum, since venue prestige is for some reason based on the percentage of papers rejected. As if there isn't even variance in year to year quality.
godelski 2y ago
> the issue is not a vacuous "prestige" but funding being dependent on hard metrics such as impact factor These things are not in contention. There is no singular problem to be solved, which is why it is so difficult. No smoking gun. > And ArXiv is not one of them. ArXiv has a large impact on metrics and so called impact factor. But let's also not be delusional to the fact that a paper from a prestigious institution will always receive more citations (or any other metric) than an equal quality paper from a less prestigious institution. All our metrics can be hacked through publicity. > Reading your post Reading yours, it sounds like you stand in the way of resolving issues in academia. Not because you don't have issues with it that you want to solve, but because you have already found the answer. Your comment reads like someone who has a relationship with academia.

dguest 2y ago

I all fairness, I don't think the grandparent post disagrees with anything that is in the parent post here. Yes, academia has tried to quantify prestige via impact factor and peer-reviewed journals. Yes, lots of people (even in Academia) feel that the system is being gamed, with by the publishing houses that own the journals being a common scapegoat. The system isn't broken, but it also keeps its integrity through some dynamic tension: a bit of criticism is a good thing.

JadeNB 2y ago

> And ArXiv is not one of them. But putting your papers on the arXiv, as your parent said, doesn't mean you only put them on the arXiv. I put all my papers on the arXiv, but I also submit them for publication in journals that will help me make the case for funding and promotion.

BeetleB 2y ago

> Your comment doesn't read like one from anyone with any relationship with academia. Your comment reads likewise. He didn't say he publishes them exclusively on Arxiv. It's quite common for professors to post it there as well as submit to journals. Many (most?) journals allow for it - they don't insist the ones in arxiv be taken down - as long as they're posting preprints and not the final (copyrighted) version. As an academic, you should also know that practices vary widely with discipline. As an example: > dependent on them publishing on a small set of non-open peer-reviewed journals. IIRC, NIH grants require publishing in open peer-reviewed journals. Also, lots of disciplines are not heavily reliant on funding. In both universities I attended, the bulk of math professors did not even apply for grants! It's not required to get tenure (unlike engineering/physics). Also often true in some economics departments. As an aside, your comment violates a number of HN guidelines.

godelski 2y ago
> Your comment reads likewise. FWIW I thought it read like reviewer 2, which makes me actually think they have a relationship with academia. The problem I have with their comment is that it rejects the critique by pointing to a different issue. As if there is a singular issue with academia that leads to the mess. So it comes off as a typical "reviewer 2" comment to me where the is more complaint and disagreement than critique. FWIW, I think we in academia need to embrace the fuzziness and noise of evaluation. I think the issue is that by trying to make sense out of an extremely noisy process we placed too much value in the (still noisy) signals we can use. It is a problem to think that these are objective answers and deny the existence of Goodhart's Law (this is especially ironic in ML where any intro class discusses reward hacking). And in this, I think there's a strong coupling between cgshep's and chipdart's complaints. As for publishing, I think we also lost sight of the main reason we publish: to communicate with our peers. Publishers played an important role since not even a century ago we could not make our works trivially available to our peers. But now the problem is information overload, not lack of information. And I think in part the review process is getting worse and worse each year as we place so little value on the act of reviewing, do not hold anyone accountable for a low quality review[0,1], do not hold ACs or Metas accountable for, and we have so many papers to review that I don't think we can expect high quality reviewing even if we actually incentivized it. I mean in ML we expect a few hundred ACs to oversee over ten thousand submissions? My question is if we'll learn that the noisiness of the process and the randomness of rejection creates a negative feedback of papers. Where you "roll the dice" on the next conference. You resubmit without changes, as well as your new work (publish or perish baby). If we had quality reviewing at least this would push pressure for papers to get incrementally better instead of just being recycled. But recycling is a very efficient strategy right now and we've seen plenty of data to suggest it is. [0] I understand the reasons for this. It is a tricky problem [1] I'd actually argue we incentivize bad reviews. No one questions you when you reject a work and finding reasons to reject a work are far easier to accept one. There's always legitimate reasons to reject any work. Not to mention that the whole process is zero sum, since venue prestige is for some reason based on the percentage of papers rejected. As if there isn't even variance in year to year quality.

godelski 2y ago

> the issue is not a vacuous "prestige" but funding being dependent on hard metrics such as impact factor These things are not in contention. There is no singular problem to be solved, which is why it is so difficult. No smoking gun. > And ArXiv is not one of them. ArXiv has a large impact on metrics and so called impact factor. But let's also not be delusional to the fact that a paper from a prestigious institution will always receive more citations (or any other metric) than an equal quality paper from a less prestigious institution. All our metrics can be hacked through publicity. > Reading your post Reading yours, it sounds like you stand in the way of resolving issues in academia. Not because you don't have issues with it that you want to solve, but because you have already found the answer. Your comment reads like someone who has a relationship with academia.