4 ms·

Maybe it's time for a reputation system. E.g. every author publishes a public PGP key along with their work. Not sure about the details but this is about CS, so

Maybe it's time for a reputation system. E.g. every author publishes a public PGP key along with their work. Not sure about the details but this is about CS, so I'm sure they will figure something out.

SoftTalker 1y ago
People are already putting their names on the LLM slop, why would they hesitate to PGP-sign it?
- caymanjim 1y ago
  They've also been putting their names on their grad students' work for eternity as well. It's not like the person whose name is at the top actually writes the paper.
  - jvanderbot 1y ago
    Not reviewing an upload which turns out to be LLM slop is precisely the kind of thing you want to track with a reputation system
jvanderbot 1y ago
Their name, orcid, and email isn't enough?
- gcr 1y ago
  You can’t get an arXiv account without a referral anyway. Edit: For clarification I’m agreeing with OP
  - deleted 1y ago
    [deleted]
  - hiddencost 1y ago
    Not quite true. If you've got an email associated with a known organization you can submit. Which includes some very large ones like @google.com
  - mindcrime 1y ago
    You can create an arXiv.org account with basically any email address whatsoever[0], with no referral. What you can't necessarily do is upload papers to arXiv without an "endorsement"[1]. Some accounts are given automatic endorsements for some domains (eg, math, cs, physics, etc) depending on the email address and other factors. Loosely speaking, the "received wisdom" has generally been that if you have a .edu address, you can probably publish fairly freely. But my understanding is that the rules are a little more nuanced than that. And I think there are other, non .edu domains, where you will also get auto-endorsed. But they don't publish a list of such things for obvious reasons. [0]: Unless things have changed since I created my account, which was originally created with my personal email address. That was quite some time ago, so I guess it's possible changes have happened that I'm not aware of. [1]: https://info.arxiv.org/help/endorsement.html https://info.arxiv.org/help/endorsement.html
jfengel 1y ago
I had been kinda hoping for a web-of-trust system to replace peer review. Anyone can endorse an article. You can decide which endorsers you trust, and do some network math to find what you think is reading. With hashes and signatures and all that rot. Not as gate-keepy as journals and not as anarchic as purely open publishing. Should be cheap, too.
- raddan 1y ago
  The problem with an endorsement scheme is citation rings, ie groups of people who artificially inflate the perceived value of some line of work by citing each other. This is a problem even now, but it is kept in check by the fact that authors do not usually have any control over who reviews their paper. Indeed, in my area, reviews are double blind, and despite claims that “you can tell who wrote this anyway” research done by several chairs in our SIG suggests that this is very much not the case. Fundamentally, we want research that offers something new (“what did we learn?”) and presents it in a way that at least plausibly has a chance of becoming generalizable knowledge. You call it gate-keeping, but I call it keeping published science high-quality.
  - geysersam 1y ago
    But you can choose to not trust people that are part of citation rings.
    - dmoy 1y ago
      It is a non trivial problem to do just that. It's related to the same problems you have with e.g. Sybil attacks: https://en.wikipedia.org/wiki/Sybil_attack https://en.wikipedia.org/wiki/Sybil_attack I'm not saying it wouldn't be worthwhile to try, just that I expect there to be a lot of very difficult problems to solve there.
      - yorwba 1y ago
        Sybil attacks are a problem when you care about global properties of permissionless networks. If you only care about local properties in a subnetwork where you hand-pick the nodes, the problem goes away. I.e. you can't use such a scheme to find the best paper in the whole world, but you can use it to rank papers in a small subdiscipline where you personally recognize most of the important authors.
losvedir 1y ago
Maybe arXiv could keep the free preprints but offer a service on top. Humans, experts in the field, would review submissions, and arXiv would curate and publish the high quality ones, and offer access to these via a subscription or fee per paper....
- nunez 1y ago
  I'm guessing this is why they are mandating that submitted position or review papers get published in a journal first.
- raddan 1y ago
  Of course we already have a system that does this: journals and conferences. They’re peer-reviewed venues for showing the world your work.
uniqueuid 1y ago
I got that suggestion recently talking to a colleague from a prestigious university. Her suggestion was simple: Kick out all non-ivy league and most international researchers. Then you have a working reputation system. Make of that what you will ...
- eesmith 1y ago
  Ahh, your colleague wants a higher concentration of "that comet might be an interstellar spacecraft" articles.
  - uniqueuid 1y ago
    If your goal is exclusively reducing strain of overloaded editors, then that's just a side effect that you might tolerate :)
- internetguy 1y ago
  all non-ivy league researchers? that seems a little harsh IMO. i've read some amazing papers from T50 or even some T100 universities.
- Ekaros 1y ago
  Maybe there should be some type of strike rules. Say 3 bad articles from any institution and they get 10 year ban. Whatever their prestige or monetary value is. You let people under your name to release bad articles you are out for a while. Treat everyone equally. After 10 years of only quality you get chance to get back. Before that though luck.
  - uniqueuid 1y ago
    I'm not sure everyone got my hint that the proposal is obviously very bad, (1) because ivy league also produces a lot of work that's not so great (i.e. wrong (looking at you, Ariely) or un-ambitious) and (2) because from time to time, some really important work comes out of surprising places. I don't think we have a good verdict on the Orthega hypothesis yet, but I'm not a professional meta scientist. That said, your proposal seems like a really good idea, I like it! Except I'd apply it to individuals and/or labs.
hermannj314 1y ago
I didn't agree with this idea, but then I looked at how much HN karma you have and now I think that maybe this is a good idea.
- SyrupThinker 1y ago
  Ignoring the actual proposal or user, just looking at karma is probably a pretty terrible metric. High karma accounts tend to just interact more frequently, for long periods of time. Often with less nuanced takes, that just play into what is likely to be popular within a thread. Having a Userscript that just places the karma and comment count next to a username is pretty eye opening.
  - elashri 1y ago
    I have a userscript to actually hide my own karma because I always think it is useless but your point is good actually. But also I think that karma/comment ratio is better than absolute karma. It has its own problems but it is just better. And I would ask if you can share the userscript. And to bring this back to the original arxiv topic. I think reputation system is going to face problems with some people outside CS lack of enough technical abilities. It also introduce biases in that you would endorse people who you like for other reasons. Actually some of the problems are solved and you would need careful proposal. But the change for publishing scheme needs push from institutions and funding agencies. Authors don't oppose changes but you have a lobby of the parasitic publishing cartel that will oppose these changes.
  - amelius 1y ago
    Yes, HN should probably publish karma divided by #comments. Or at least show both numbers.
    - amelius 1y ago
      (an added complication is that posting articles also increases karma)
- fn-mote 1y ago

SoftTalker 1y ago

People are already putting their names on the LLM slop, why would they hesitate to PGP-sign it?

caymanjim 1y ago
They've also been putting their names on their grad students' work for eternity as well. It's not like the person whose name is at the top actually writes the paper.
- jvanderbot 1y ago
  Not reviewing an upload which turns out to be LLM slop is precisely the kind of thing you want to track with a reputation system

jvanderbot 1y ago

Their name, orcid, and email isn't enough?

gcr 1y ago
You can’t get an arXiv account without a referral anyway. Edit: For clarification I’m agreeing with OP
- deleted 1y ago
  [deleted]
- hiddencost 1y ago
  Not quite true. If you've got an email associated with a known organization you can submit. Which includes some very large ones like @google.com
- mindcrime 1y ago
  You can create an arXiv.org account with basically any email address whatsoever[0], with no referral. What you can't necessarily do is upload papers to arXiv without an "endorsement"[1]. Some accounts are given automatic endorsements for some domains (eg, math, cs, physics, etc) depending on the email address and other factors. Loosely speaking, the "received wisdom" has generally been that if you have a .edu address, you can probably publish fairly freely. But my understanding is that the rules are a little more nuanced than that. And I think there are other, non .edu domains, where you will also get auto-endorsed. But they don't publish a list of such things for obvious reasons. [0]: Unless things have changed since I created my account, which was originally created with my personal email address. That was quite some time ago, so I guess it's possible changes have happened that I'm not aware of. [1]: https://info.arxiv.org/help/endorsement.html https://info.arxiv.org/help/endorsement.html

jfengel 1y ago

I had been kinda hoping for a web-of-trust system to replace peer review. Anyone can endorse an article. You can decide which endorsers you trust, and do some network math to find what you think is reading. With hashes and signatures and all that rot. Not as gate-keepy as journals and not as anarchic as purely open publishing. Should be cheap, too.

raddan 1y ago
The problem with an endorsement scheme is citation rings, ie groups of people who artificially inflate the perceived value of some line of work by citing each other. This is a problem even now, but it is kept in check by the fact that authors do not usually have any control over who reviews their paper. Indeed, in my area, reviews are double blind, and despite claims that “you can tell who wrote this anyway” research done by several chairs in our SIG suggests that this is very much not the case. Fundamentally, we want research that offers something new (“what did we learn?”) and presents it in a way that at least plausibly has a chance of becoming generalizable knowledge. You call it gate-keeping, but I call it keeping published science high-quality.
- geysersam 1y ago
  But you can choose to not trust people that are part of citation rings.
  - dmoy 1y ago
    It is a non trivial problem to do just that. It's related to the same problems you have with e.g. Sybil attacks: https://en.wikipedia.org/wiki/Sybil_attack https://en.wikipedia.org/wiki/Sybil_attack I'm not saying it wouldn't be worthwhile to try, just that I expect there to be a lot of very difficult problems to solve there.
    - yorwba 1y ago
      Sybil attacks are a problem when you care about global properties of permissionless networks. If you only care about local properties in a subnetwork where you hand-pick the nodes, the problem goes away. I.e. you can't use such a scheme to find the best paper in the whole world, but you can use it to rank papers in a small subdiscipline where you personally recognize most of the important authors.

losvedir 1y ago

Maybe arXiv could keep the free preprints but offer a service on top. Humans, experts in the field, would review submissions, and arXiv would curate and publish the high quality ones, and offer access to these via a subscription or fee per paper....

nunez 1y ago
I'm guessing this is why they are mandating that submitted position or review papers get published in a journal first.
raddan 1y ago
Of course we already have a system that does this: journals and conferences. They’re peer-reviewed venues for showing the world your work.

uniqueuid 1y ago

I got that suggestion recently talking to a colleague from a prestigious university. Her suggestion was simple: Kick out all non-ivy league and most international researchers. Then you have a working reputation system. Make of that what you will ...

eesmith 1y ago
Ahh, your colleague wants a higher concentration of "that comet might be an interstellar spacecraft" articles.
- uniqueuid 1y ago
  If your goal is exclusively reducing strain of overloaded editors, then that's just a side effect that you might tolerate :)
internetguy 1y ago
all non-ivy league researchers? that seems a little harsh IMO. i've read some amazing papers from T50 or even some T100 universities.
Ekaros 1y ago
Maybe there should be some type of strike rules. Say 3 bad articles from any institution and they get 10 year ban. Whatever their prestige or monetary value is. You let people under your name to release bad articles you are out for a while. Treat everyone equally. After 10 years of only quality you get chance to get back. Before that though luck.
- uniqueuid 1y ago
  I'm not sure everyone got my hint that the proposal is obviously very bad, (1) because ivy league also produces a lot of work that's not so great (i.e. wrong (looking at you, Ariely) or un-ambitious) and (2) because from time to time, some really important work comes out of surprising places. I don't think we have a good verdict on the Orthega hypothesis yet, but I'm not a professional meta scientist. That said, your proposal seems like a really good idea, I like it! Except I'd apply it to individuals and/or labs.

hermannj314 1y ago

I didn't agree with this idea, but then I looked at how much HN karma you have and now I think that maybe this is a good idea.

SyrupThinker 1y ago
Ignoring the actual proposal or user, just looking at karma is probably a pretty terrible metric. High karma accounts tend to just interact more frequently, for long periods of time. Often with less nuanced takes, that just play into what is likely to be popular within a thread. Having a Userscript that just places the karma and comment count next to a username is pretty eye opening.
- elashri 1y ago
  I have a userscript to actually hide my own karma because I always think it is useless but your point is good actually. But also I think that karma/comment ratio is better than absolute karma. It has its own problems but it is just better. And I would ask if you can share the userscript. And to bring this back to the original arxiv topic. I think reputation system is going to face problems with some people outside CS lack of enough technical abilities. It also introduce biases in that you would endorse people who you like for other reasons. Actually some of the problems are solved and you would need careful proposal. But the change for publishing scheme needs push from institutions and funding agencies. Authors don't oppose changes but you have a lobby of the parasitic publishing cartel that will oppose these changes.
- amelius 1y ago
  Yes, HN should probably publish karma divided by #comments. Or at least show both numbers.
  - amelius 1y ago
    (an added complication is that posting articles also increases karma)
fn-mote 1y ago