5 ms·

Idk if we're automating humans out of the publishing loop as much as rapidly automating the production of crap. I had a very similar experience reviewing for EM

Idk if we're automating humans out of the publishing loop as much as rapidly automating the production of crap. I had a very similar experience reviewing for EMNLP recently. We are nowhere near AI being able to judge the quality of research (in fact, one might reasonably state that even most humans can't really judge the quality of research). Most things in society are not like math: we can't automate (via verification) our way out of noise overwhelming the signal. Folks are willing to entirely abuse the public resource that is faithful, honest reviewing. (This is unsurprising; the abuse of the commons / public resources has been rising for a long time). There isn't a good solution other than something akin to draconian social scoring to limit access to the reviewing system.

boplicity 2mo ago
> draconian social scoring This is a legitimate question: should people have reputations? Should their behavior be made more visible publicly, both good and bad? How? In a small contained society, where consequences are more directly affecting individuals immedately, these questions don't need to be asked, because they're inherantly answered. We now are a society with billions of people, and dire consequences sometimes deferred for a generation, or more. Part of our general failing is the lack of good answers to the above questions. For many people, there are rarely negative consequences for causing harm to others, and the rewards can be very great indeed.
- onetokeoverthe 2mo ago
  [dead]
- ryandrake 2mo ago
  "Consequences and reputations stemming from one's actions" isn't necessarily draconian social scoring. Without a structure for imposing consequences on wrongdoers, we're not a society, we're just monkeys flinging poo at each other.
  - cindyllm 2mo ago
    [dead]
  - blackqueeriroh 2mo ago
    The problem is that everyone imagines some sort of just and fair arbiter of these things, when the reality is all of the social scoring and consequences and reputations rarely actually stem from one’s actions, and far more often stem from how much money someone has put in someone’s pocket, who someone knows, the color of someone’s skin, or what’s between someone’s legs. Until we’re actually serious about treating people equitably, (not equally, as that would simply leave the lopsided power structure we have in place) we aren’t getting out of this.
    - somenameforme 2mo ago
      I don't understand the desire to argue against this in identity politics terms, which I assume you realize is going to be extremely divisive. There's a far more simple argument that people, independent of ideological framing, would tend to agree upon. And that's what is valuable to one person isn't valuable, to the same degree, to another. For things that are completely illegal, like murder, such a system would be pretty useless because we already have systems in place to punish that sort of behavior and defacto social scoring on it as well. So this is going to come down to things that aren't usually illegal or 'that illegal', but otherwise affect society. So for instance one group might want to reasonably punish the directors of a company that causes emissions. Another group might want to reasonable reward the directors of a company that creates a large number of desirable jobs. And even in this one example you immediately end up with a weird scenario. Is it okay to pollute as long as you make enough jobs? Who gets to decide that? There's no need for cartoon villainy for this to be a bad idea. Even completely well intentioned, it just doesn't work out well. And the more diverse a society, the worse it's going to work.
      - knollimar 2mo ago
        How do you combat review bombing or someone just being bigoted?
- attila-lendvai 2mo ago
  in short: our social sructure has built out so far beyond the Dunbar's number that it has become a haven for psychopaths.
- BobbyTables2 2mo ago
  Isn’t China doing something like this ? Wonder if it could degrade to something worse than the status quo where reputations are falsely tarnished by competitors purely as a weapon to get ahead…
  - nneonneo 2mo ago
    The "Chinese social credit" system is vastly overblown. I'm living here (China, Beijing and Jiangsu) now, and there are exactly two cases in the past eight months where the "social credit" system has had any impact at all: - To open "take first pay after" vending machines. These are vending machines which are basically big locked fridges; if your credit score is high enough, you can open the machines, take whatever drinks/snacks you want, and they'll use (I presume) computer vision to charge you afterwards. If you don't have a high enough credit score, tough, you can't use them. (But there's almost always normal pay-first vending machines nearby). - To borrow mobile charging packs (powerbanks). Some operators will let you "swipe your credit" (check your credit score) to take one without paying a deposit. If your credit score isn't high enough, you first pay e.g. a ¥99 deposit (~$15USD) which gets returned when you return the powerbank, not a big deal. That's...it. My credit score is high enough on only one platform (Alipay), so I get to try what happens with both "high" and "low" credit, and I can confidently say these are the *only* two cases where I have even been asked to show the credit score. I have taken multiple train trips, bought lots of stuff in stores and restaurants, etc. without ever touching the "social credit score". P.S. I literally don't even know how to raise my credit score, and neither do many of the folks who live here - again, not that it matters, because the score is simply not that important.
    - rogerrogerr 2mo ago
      I remember in the west a few years ago we were seeing purported pictures of electronic billboards in China shaming low-social-credit-havers. Was that all a weird psyop?
      - scns 2mo ago
        > electronic billboards in China shaming low-social-credit-havers It is people who did not pay their debts. Dr Jonathan Tam has a video on YT explaining it called: Why the World Fell for China's Fake Dystopia https://www.youtube.com/watch?v=Ecx45eOuc0E https://www.youtube.com/watch?v=Ecx45eOuc0E
- gspr 2mo ago
  There's a huge difference between a completely decentralized reputation system like the interpersonal one that has existed since humans first became sentient, and an artificially built top-down reputation system run by a powerful entity (be it the state like in China or megacorps like in the West).
hellohello2 2mo ago
I think you are correct, its so hard to judge the quality of research that we have been using publication record/count as a proxy. Ultimately, having papers being easy to write is good, provided we find a better way of judging quality.
azan_ 2mo ago
People should get paid for reviewing. That’s the solution. Publishing companies rack in billions of dollars in pure profit exploring free labor. Once people actually get paid for reviewing, it becomes much faster and higher quality and you won’t need AI triage.
- Sharlin 2mo ago
  Doesn’t work anymore because people would just have ChatGPT generate their reviews and get paid for doing nothing.
  - zx8080 2mo ago
    So in sum: - it did not work before because greed - will not work anymore because AI Is human science dead?
    - inigyou 2mo ago
      Science as an activity will never die, but science as an institution is in big trouble.
    - StableAlkyne 2mo ago
      Nah, people will just come up with different (probably broader) heuristics when reviewing. We have a past example of this in the US, too. Certain countries have historically had a huge problem with paper mills. Because of this, most people in the US/EU do have a negative bias once they see the author affiliations of those countries. Yes, it isn't fair to researchers who aren't pulling some shenaniganry. But it is a common shortcut; the sixth or seventh time you've spent a few hours reviewing a paper filled with bullshit, you probably would develop it too. I imagine we will see similar things come up: the most obvious I can think of is, if it's not a well-known institution then it might be bullshit. Hell, we are already seeing something similar in FOSS, where many high profile projects have completely banned AI contributions due to all the low effort slop PRs.
    - NetMageSCW 2mo ago
      That should be will not work anymore because of greed.
whatever1 2mo ago
Exactly. We invented a pump. We can use it as vacuum to clean things up. The same pump can be used as a shit fire hose. The effort we need to clean things up is significantly higher than the effort needed to make a mess.
ludicrousdispla 2mo ago
The 'named' reviewers typically pass on the review responsibilty to grad students or less-senior colleagues. This is currently a generally accepted and sanctioned/encouraged practice. A social scoring system would need to eliminate that for true accountability.
- weiliddat 2mo ago
  Yeah it's a well-known secret, and I think any sort of scoring system misses the point. The scentific method (and system, incl. peer reviews) is a public good (like good governance) that requires moral acknowledgement, the necessary social/cultural/political/academic pressures, and discipline of taking care of it by participating parties. If you add a score/metric this incentivizes the wrong thing, like how money, funding, and career already incentivizes all the wrong non-scientific behaviors (like faking data, p-hacking, etc.)
  - StableAlkyne 2mo ago
    > If you add a score/metric this incentivizes the wrong thing, Don't we already have all of these things because academia decided to use H-index as a metric for career impact?
    - weiliddat 2mo ago
      Yeah exactly. One more metric isn't going to solve it.
- gradstudent 2mo ago
  Reviews are typically in conjunction with junior colleagues and the senior person is involved in the discussion and held responsible for the outcome. The junior folks then end up in the proceedings as external reviewers. In my experience anyway. I don't see an issue here. How else do you train junior researchers to review? Its how I learned and all those around me as well.
Rebuff5007 2mo ago
I think the good solution is to introduce a bit of "shame" into the system -- de-anonymized submissions and reviews. You'd be less willing to publish ai slop if your name had to be tied to it for years to come.

boplicity 2mo ago

> draconian social scoring This is a legitimate question: should people have reputations? Should their behavior be made more visible publicly, both good and bad? How? In a small contained society, where consequences are more directly affecting individuals immedately, these questions don't need to be asked, because they're inherantly answered. We now are a society with billions of people, and dire consequences sometimes deferred for a generation, or more. Part of our general failing is the lack of good answers to the above questions. For many people, there are rarely negative consequences for causing harm to others, and the rewards can be very great indeed.

onetokeoverthe 2mo ago
[dead]
ryandrake 2mo ago
"Consequences and reputations stemming from one's actions" isn't necessarily draconian social scoring. Without a structure for imposing consequences on wrongdoers, we're not a society, we're just monkeys flinging poo at each other.
- cindyllm 2mo ago
  [dead]
- blackqueeriroh 2mo ago
  The problem is that everyone imagines some sort of just and fair arbiter of these things, when the reality is all of the social scoring and consequences and reputations rarely actually stem from one’s actions, and far more often stem from how much money someone has put in someone’s pocket, who someone knows, the color of someone’s skin, or what’s between someone’s legs. Until we’re actually serious about treating people equitably, (not equally, as that would simply leave the lopsided power structure we have in place) we aren’t getting out of this.
  - somenameforme 2mo ago
    I don't understand the desire to argue against this in identity politics terms, which I assume you realize is going to be extremely divisive. There's a far more simple argument that people, independent of ideological framing, would tend to agree upon. And that's what is valuable to one person isn't valuable, to the same degree, to another. For things that are completely illegal, like murder, such a system would be pretty useless because we already have systems in place to punish that sort of behavior and defacto social scoring on it as well. So this is going to come down to things that aren't usually illegal or 'that illegal', but otherwise affect society. So for instance one group might want to reasonably punish the directors of a company that causes emissions. Another group might want to reasonable reward the directors of a company that creates a large number of desirable jobs. And even in this one example you immediately end up with a weird scenario. Is it okay to pollute as long as you make enough jobs? Who gets to decide that? There's no need for cartoon villainy for this to be a bad idea. Even completely well intentioned, it just doesn't work out well. And the more diverse a society, the worse it's going to work.
    - knollimar 2mo ago
      How do you combat review bombing or someone just being bigoted?
attila-lendvai 2mo ago
in short: our social sructure has built out so far beyond the Dunbar's number that it has become a haven for psychopaths.
BobbyTables2 2mo ago
Isn’t China doing something like this ? Wonder if it could degrade to something worse than the status quo where reputations are falsely tarnished by competitors purely as a weapon to get ahead…
- nneonneo 2mo ago
  The "Chinese social credit" system is vastly overblown. I'm living here (China, Beijing and Jiangsu) now, and there are exactly two cases in the past eight months where the "social credit" system has had any impact at all: - To open "take first pay after" vending machines. These are vending machines which are basically big locked fridges; if your credit score is high enough, you can open the machines, take whatever drinks/snacks you want, and they'll use (I presume) computer vision to charge you afterwards. If you don't have a high enough credit score, tough, you can't use them. (But there's almost always normal pay-first vending machines nearby). - To borrow mobile charging packs (powerbanks). Some operators will let you "swipe your credit" (check your credit score) to take one without paying a deposit. If your credit score isn't high enough, you first pay e.g. a ¥99 deposit (~$15USD) which gets returned when you return the powerbank, not a big deal. That's...it. My credit score is high enough on only one platform (Alipay), so I get to try what happens with both "high" and "low" credit, and I can confidently say these are the *only* two cases where I have even been asked to show the credit score. I have taken multiple train trips, bought lots of stuff in stores and restaurants, etc. without ever touching the "social credit score". P.S. I literally don't even know how to raise my credit score, and neither do many of the folks who live here - again, not that it matters, because the score is simply not that important.
  - rogerrogerr 2mo ago
    I remember in the west a few years ago we were seeing purported pictures of electronic billboards in China shaming low-social-credit-havers. Was that all a weird psyop?
    - scns 2mo ago
      > electronic billboards in China shaming low-social-credit-havers It is people who did not pay their debts. Dr Jonathan Tam has a video on YT explaining it called: Why the World Fell for China's Fake Dystopia https://www.youtube.com/watch?v=Ecx45eOuc0E https://www.youtube.com/watch?v=Ecx45eOuc0E
gspr 2mo ago
There's a huge difference between a completely decentralized reputation system like the interpersonal one that has existed since humans first became sentient, and an artificially built top-down reputation system run by a powerful entity (be it the state like in China or megacorps like in the West).

hellohello2 2mo ago

I think you are correct, its so hard to judge the quality of research that we have been using publication record/count as a proxy. Ultimately, having papers being easy to write is good, provided we find a better way of judging quality.

azan_ 2mo ago

People should get paid for reviewing. That’s the solution. Publishing companies rack in billions of dollars in pure profit exploring free labor. Once people actually get paid for reviewing, it becomes much faster and higher quality and you won’t need AI triage.

Sharlin 2mo ago
Doesn’t work anymore because people would just have ChatGPT generate their reviews and get paid for doing nothing.
- zx8080 2mo ago
  So in sum: - it did not work before because greed - will not work anymore because AI Is human science dead?
  - inigyou 2mo ago
    Science as an activity will never die, but science as an institution is in big trouble.
  - StableAlkyne 2mo ago
    Nah, people will just come up with different (probably broader) heuristics when reviewing. We have a past example of this in the US, too. Certain countries have historically had a huge problem with paper mills. Because of this, most people in the US/EU do have a negative bias once they see the author affiliations of those countries. Yes, it isn't fair to researchers who aren't pulling some shenaniganry. But it is a common shortcut; the sixth or seventh time you've spent a few hours reviewing a paper filled with bullshit, you probably would develop it too. I imagine we will see similar things come up: the most obvious I can think of is, if it's not a well-known institution then it might be bullshit. Hell, we are already seeing something similar in FOSS, where many high profile projects have completely banned AI contributions due to all the low effort slop PRs.
  - NetMageSCW 2mo ago
    That should be will not work anymore because of greed.

whatever1 2mo ago

Exactly. We invented a pump. We can use it as vacuum to clean things up. The same pump can be used as a shit fire hose. The effort we need to clean things up is significantly higher than the effort needed to make a mess.

ludicrousdispla 2mo ago

The 'named' reviewers typically pass on the review responsibilty to grad students or less-senior colleagues. This is currently a generally accepted and sanctioned/encouraged practice. A social scoring system would need to eliminate that for true accountability.

weiliddat 2mo ago
Yeah it's a well-known secret, and I think any sort of scoring system misses the point. The scentific method (and system, incl. peer reviews) is a public good (like good governance) that requires moral acknowledgement, the necessary social/cultural/political/academic pressures, and discipline of taking care of it by participating parties. If you add a score/metric this incentivizes the wrong thing, like how money, funding, and career already incentivizes all the wrong non-scientific behaviors (like faking data, p-hacking, etc.)
- StableAlkyne 2mo ago
  > If you add a score/metric this incentivizes the wrong thing, Don't we already have all of these things because academia decided to use H-index as a metric for career impact?
  - weiliddat 2mo ago
    Yeah exactly. One more metric isn't going to solve it.
gradstudent 2mo ago
Reviews are typically in conjunction with junior colleagues and the senior person is involved in the discussion and held responsible for the outcome. The junior folks then end up in the proceedings as external reviewers. In my experience anyway. I don't see an issue here. How else do you train junior researchers to review? Its how I learned and all those around me as well.

Rebuff5007 2mo ago

I think the good solution is to introduce a bit of "shame" into the system -- de-anonymized submissions and reviews. You'd be less willing to publish ai slop if your name had to be tied to it for years to come.