6 ms·

Prediction: as a result of fake research and it’s ilk (fake credentials, fake trade-journal pubs, fake reviews, etc.), we will observe a renewed interest in ref

Prediction: as a result of fake research and it’s ilk (fake credentials, fake trade-journal pubs, fake reviews, etc.), we will observe a renewed interest in referrals and meatspace networking. It will become correspondingly difficult for outsiders to enter professional circles.

godelski 3y ago
On the other hand, maybe we'll abandon peer review and return to a pre 1950's like style? Everyone in ML just uses arxiv because the field is moving so fast. All work is realistically peer reviewed once it is out there. We treat conferences (more important than journals) as very noisy signals. Unfortunately, we still use those as metrics for completing degrees or hiring people. The way I would try to make this better is just to post to OpenReview rather than arxiv or integrate the two. That way discussions can be had on the works, in the open, and public.
- stubybubs 3y ago
  It's easy to do this with anything in computer science as others can implement or recreate whatever is being discussed easily. Not so with a 6 month experiment with several groups of humans, or even a 2 week one with rats.
  - godelski 3y ago
    This is not always true. ML has reproducibility issues despite the status quo being open models. Even if we don't include proprietary datasets there are still issues. Compute is one issue, but we can even ignore that[0]. The Lottery Ticket[1,2] plays a big role, in that you can just get lucky. This really should have resulted in treating benchmarks as weaker indicators but see other comments about reviewing incentives. Another issue is that it is status quo to optimize hyperparameters on test data and this results in information leakage. While this won't affect results of running a checkpoint there are issues in reproducing the work. It also adds noise. Generative papers also have a large issue in not showing random uncurated samples, which introduces huge biases and we can argue this is a reproducibility issue as you can't reproduce the results show in the works. Anyone that's played with things like Stable Diffusion (or any generative model) will be familiar with this. There are more nuanced issues and things that require intimate domain knowledge to fully understand, but I wanted to push back at this comment because I see a lot of people just brush off reproducibility concerns by pointing to GitHub. While it helps, it definitely doesn't make reproducibility "easy" or concerns nonexistent. [0] Sometimes we can't though as scaling has more effects than obvious. e.g. GANs can't scale batch on a per GPU level but scaling by multiple GPUs/nodes does give an advantage in quality (not just training times). This is non-obvious and many things can play a role. [1] https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635 [2] https://arxiv.org/abs/2109.08203 https://arxiv.org/abs/2109.08203
- omginternets 3y ago
  I think peer review exists for reasons beyond building or affirming the reputation of a scholar. In particular, it’s regarded as a filter for eliminating the very worst-quality research. And judging from the manuscripts that came across my desk when I was still in academia, I’d say it’s performing this particular duty reasonably well. As such, the connection between AI fakery and peer-review isn’t all that clear cut to me. I might expect the peer-review pipeline to become inundated with prima facie credible manuscripts, which would overwhelm the reviewer’s ability to process publications even more than is already the case. At this point, I’d expect scholarly reputation to become even more important for getting published, and I’d expect to see an increase in signaling and other out-of-band communications (e.g. talking about your draft at a conference). Critically, I’d expect this to generally avoid AI generated “studies”, despite the obvious drawbacks. And moreover, I actually have serious reservations about the project of purging the scientific landscape of informal processes. While there are some obvious and notorious drawbacks to the way things currently work, I am also wary of top-down, “authoritarian high-modernist” (to borrow the turn of phrase from James C. Scott’s Seeing Like a State) projects in all their forms, especially in science. The production of scientific knowledge is not pure technae; it cannot be reduced to a well-behaved formula. A large part of it is a craft requiring a significant share of informal processes in order to be truly creative. But that is a separate topic, and indeed, one might argue that doing away with formal peer-review altogether would be a step in the direction of informality. So at the end of the day, I’m not convinced by any predictions that relate the effects of AI on peer-review. I think it could go either way :)
  - godelski 3y ago
    I mostly agree with your assessment of peer review, but feel that it's gone too far. It's decades of metric hacking going unresolved. My research area is ML and it is definitely far out of hand. When I was in physics I'd probably be closer to your statement. But right now we have measurements that conclude "reviewers are good at identifying bad papers, but bad at identifying good papers." Either I think we need to accept more works (as long as they are valid and useful) and/or reducing the value of a top tier venue (which right now is top or none). I think papers should get multiple rounds, as our goal is to improve works, and submissions should always be rolling. I think this can help, because I agree that peer review should be about eliminating the worst, not identifying the best (far too difficult, and hundreds of years of evidence). I think one important thing that venues could do is host data and code/tools for works. Harddrive space is rather cheap now (especially with tax benefits and donations) and can only result in high value to the community. The other thing is having formats like OpenReview, where works can continuously be discussed in the open. With authors and others being able to defend/criticize/question works. But I think there should be some filter, even if low, and rules for control of quality. For the most part, I do actually think getting rid of venue based review would be a step in the right direction. I use these words because I like to encourage the idea that open publication still leads to peer reviewing. In ML a lot is done with arxiv + twitter, I just think we should better formalize this. It allows for a lot of freedom and for high speed. It has problems, but I don't see them as any worse than the current system (I see an overall decrease in problems). Good science requires risks and a lot of creativity. The modern word advantage is that we have higher numbers of researchers and monte carlo sampling in parallel helps optimize. You want to encourage tail end samples too to escape local minima. The history of science is a history of upsets. You don't get upsets by doing what everyone is doing. I agree that top down hierarchies discourage such thinking and are an overall net negative to science and advancing human knowledge. Creativity is critical. For predictions, I agree. I lean towards predicting disruption, but I'm not even certain of that. And disruption can go many different ways. I'm glad we're starting the discussions, but I think they need to be deeper and more honest.
- jltsiren 3y ago
  I don't think ML is a good example, because it's an outlier within an outlier. CS is already a weird field, because it leans so heavily on conference papers that see a single round of reviews. Journal papers with multiple rounds of reviews are more common in other STEM fields. Because of this difference, peer review in CS is more focused on accepting/rejecting the paper than on improving it. And then the CS model breaks down in ML, which is far too popular for its own good. There are too many people working on the same topics, writing too many papers and submitting them to too few conferences, which have too many people attending. CS conferences are supposed to be more like community meetings than formal conferences. Informal discussions are the main point, but that doesn't scale beyond a few hundred attendees.
  - godelski 3y ago
    Honestly, I agree with every word here. I'd rather see conferences as networking events than quality filters. Conferences should invite published works. The conference system just adds noise, and like you are suggesting, the single round results in a zero sum game and no advocates for the paper to be accepted/improved. Thus reviewers optimize their strategy: reject everything. While I still have problems with peer review and still advocate for open publication, I do think many the the specific issues of ML could be resolved by focusing on journals rather than conferences.

godelski 3y ago

On the other hand, maybe we'll abandon peer review and return to a pre 1950's like style? Everyone in ML just uses arxiv because the field is moving so fast. All work is realistically peer reviewed once it is out there. We treat conferences (more important than journals) as very noisy signals. Unfortunately, we still use those as metrics for completing degrees or hiring people. The way I would try to make this better is just to post to OpenReview rather than arxiv or integrate the two. That way discussions can be had on the works, in the open, and public.

stubybubs 3y ago
It's easy to do this with anything in computer science as others can implement or recreate whatever is being discussed easily. Not so with a 6 month experiment with several groups of humans, or even a 2 week one with rats.
- godelski 3y ago
  This is not always true. ML has reproducibility issues despite the status quo being open models. Even if we don't include proprietary datasets there are still issues. Compute is one issue, but we can even ignore that[0]. The Lottery Ticket[1,2] plays a big role, in that you can just get lucky. This really should have resulted in treating benchmarks as weaker indicators but see other comments about reviewing incentives. Another issue is that it is status quo to optimize hyperparameters on test data and this results in information leakage. While this won't affect results of running a checkpoint there are issues in reproducing the work. It also adds noise. Generative papers also have a large issue in not showing random uncurated samples, which introduces huge biases and we can argue this is a reproducibility issue as you can't reproduce the results show in the works. Anyone that's played with things like Stable Diffusion (or any generative model) will be familiar with this. There are more nuanced issues and things that require intimate domain knowledge to fully understand, but I wanted to push back at this comment because I see a lot of people just brush off reproducibility concerns by pointing to GitHub. While it helps, it definitely doesn't make reproducibility "easy" or concerns nonexistent. [0] Sometimes we can't though as scaling has more effects than obvious. e.g. GANs can't scale batch on a per GPU level but scaling by multiple GPUs/nodes does give an advantage in quality (not just training times). This is non-obvious and many things can play a role. [1] https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635 [2] https://arxiv.org/abs/2109.08203 https://arxiv.org/abs/2109.08203
omginternets 3y ago
I think peer review exists for reasons beyond building or affirming the reputation of a scholar. In particular, it’s regarded as a filter for eliminating the very worst-quality research. And judging from the manuscripts that came across my desk when I was still in academia, I’d say it’s performing this particular duty reasonably well. As such, the connection between AI fakery and peer-review isn’t all that clear cut to me. I might expect the peer-review pipeline to become inundated with prima facie credible manuscripts, which would overwhelm the reviewer’s ability to process publications even more than is already the case. At this point, I’d expect scholarly reputation to become even more important for getting published, and I’d expect to see an increase in signaling and other out-of-band communications (e.g. talking about your draft at a conference). Critically, I’d expect this to generally avoid AI generated “studies”, despite the obvious drawbacks. And moreover, I actually have serious reservations about the project of purging the scientific landscape of informal processes. While there are some obvious and notorious drawbacks to the way things currently work, I am also wary of top-down, “authoritarian high-modernist” (to borrow the turn of phrase from James C. Scott’s Seeing Like a State) projects in all their forms, especially in science. The production of scientific knowledge is not pure technae; it cannot be reduced to a well-behaved formula. A large part of it is a craft requiring a significant share of informal processes in order to be truly creative. But that is a separate topic, and indeed, one might argue that doing away with formal peer-review altogether would be a step in the direction of informality. So at the end of the day, I’m not convinced by any predictions that relate the effects of AI on peer-review. I think it could go either way :)
- godelski 3y ago
  I mostly agree with your assessment of peer review, but feel that it's gone too far. It's decades of metric hacking going unresolved. My research area is ML and it is definitely far out of hand. When I was in physics I'd probably be closer to your statement. But right now we have measurements that conclude "reviewers are good at identifying bad papers, but bad at identifying good papers." Either I think we need to accept more works (as long as they are valid and useful) and/or reducing the value of a top tier venue (which right now is top or none). I think papers should get multiple rounds, as our goal is to improve works, and submissions should always be rolling. I think this can help, because I agree that peer review should be about eliminating the worst, not identifying the best (far too difficult, and hundreds of years of evidence). I think one important thing that venues could do is host data and code/tools for works. Harddrive space is rather cheap now (especially with tax benefits and donations) and can only result in high value to the community. The other thing is having formats like OpenReview, where works can continuously be discussed in the open. With authors and others being able to defend/criticize/question works. But I think there should be some filter, even if low, and rules for control of quality. For the most part, I do actually think getting rid of venue based review would be a step in the right direction. I use these words because I like to encourage the idea that open publication still leads to peer reviewing. In ML a lot is done with arxiv + twitter, I just think we should better formalize this. It allows for a lot of freedom and for high speed. It has problems, but I don't see them as any worse than the current system (I see an overall decrease in problems). Good science requires risks and a lot of creativity. The modern word advantage is that we have higher numbers of researchers and monte carlo sampling in parallel helps optimize. You want to encourage tail end samples too to escape local minima. The history of science is a history of upsets. You don't get upsets by doing what everyone is doing. I agree that top down hierarchies discourage such thinking and are an overall net negative to science and advancing human knowledge. Creativity is critical. For predictions, I agree. I lean towards predicting disruption, but I'm not even certain of that. And disruption can go many different ways. I'm glad we're starting the discussions, but I think they need to be deeper and more honest.
jltsiren 3y ago
I don't think ML is a good example, because it's an outlier within an outlier. CS is already a weird field, because it leans so heavily on conference papers that see a single round of reviews. Journal papers with multiple rounds of reviews are more common in other STEM fields. Because of this difference, peer review in CS is more focused on accepting/rejecting the paper than on improving it. And then the CS model breaks down in ML, which is far too popular for its own good. There are too many people working on the same topics, writing too many papers and submitting them to too few conferences, which have too many people attending. CS conferences are supposed to be more like community meetings than formal conferences. Informal discussions are the main point, but that doesn't scale beyond a few hundred attendees.
- godelski 3y ago
  Honestly, I agree with every word here. I'd rather see conferences as networking events than quality filters. Conferences should invite published works. The conference system just adds noise, and like you are suggesting, the single round results in a zero sum game and no advocates for the paper to be accepted/improved. Thus reviewers optimize their strategy: reject everything. While I still have problems with peer review and still advocate for open publication, I do think many the the specific issues of ML could be resolved by focusing on journals rather than conferences.