8 ms·

>made it past peer review I don’t think that’s an especially difficult thing to do. It seems any paper dealing with social sciences or history that fits the cu

>made it past peer review I don’t think that’s an especially difficult thing to do. It seems any paper dealing with social sciences or history that fits the current narrative is lauded and “reviewed” favorably. I think the mechanism which facilitates this is relatively simple and similar to how USSR officials would rather bring good news than bad facts

cauch 3y ago
But how could the reviewers have found the errors in this article? The article can tell "Cort was informed about the new techniques from a family member back from Jamaica". How do you verify that? The only way is to redo the all work from scratch, refound all the historical references and cross-validate each of them. It's a huge work. This article was later debunked by experts who have taken time and effort to do that. But peer-reviewing does not allow reviewers to pause they own work to do such extensive checks. More realistically, the article passes the peer-reviewing process because the peer-reviewing process does what it is supposed to do: the article is "believable", it does not have things that looks not coming from the scientific process. The reviewers don't have time and it is in fact not even their jobs to redo the study. All they do is to check if the guidelines seem to be respected. If you want more than that, you want a "replication study", which is the next step in the scientific process of building trust on a study (and which also have difficulties). And, sure, they may have flagged that some conclusions may need more convincing demonstration, but it is a Gaussian curve: sometimes they do, sometimes they don't, some passes through the gaps. I think there is a problem with layman people who don't understand that the peer review process is not a magical tool that remove all the incorrect studies (on top of that, due to statistical fluctuation, some studies are false while the authors have done 100% everything perfectly). I'm not saying it's the case here, but this is one explanation that should not be ignored. Ironically, it is a bit funny that for your conclusion you jumped immediately on the story you wish to be true: social sciences or history are ideologically biased. This is a valid hypothesis, but not the only one explaining what we observe.
- Apocryphon 3y ago
  > The article can tell "Cort was informed about the new techniques from a family member back from Jamaica". How do you verify that? The only way is to redo the all work from scratch, refound all the historical references and cross-validate each of them. It's a huge work. Since this is about checking textual references, as opposed to laboratory work, would it be possible for an LLM to do that? Seems like the hardest thing would be for the A.I. to log into the requisite gateways for the databases hosting the papers.
  - cauch 3y ago
    I doubt it is so naively simple as "querying a database". For example, before the publication of this article, the majority of the "databases" were simply saying "the inventor is Cort", which is one example of thing that the A.I. will get trivially "wrong" when reviewing the paper. And if it is true, then this A.I. review method can also be applied to mathematics, theoretical physics, computer science, ... And even if it is doable, you will still have things that will pass the review process when ideally it should not. It is just an hard limitation, same as the one in "justice" (impossible to not sometimes judge a guilty person "innocent" or an innocent person "guilty", anyone who thinks otherwise just don't understand how complicated it is)
  - nvm0n2 3y ago
    It probably can be done, but does the academic establishment really want word to get out that their great research is being auto-reviewed because the "peers" were so bad at their job that an AI can do better? They've spent decades pushing the idea that peer review is a gold standard on the world, it would be quite a climbdown to admit ChatGPT can do it better.
- whatshisface 3y ago
  >How do you verify that? The only way is to redo the all work from scratch, refound all the historical references and cross-validate each of them. It's a huge work. Sorry, but if the paper doesn't make its case, it isn't an academic work. If it needs to be 1,000 papers in order to explain why its assertions are reliable, great, that's 1,000 good papers to write. :)*
OfSanguineFire 3y ago
I provide language revision for academics who are non-native speakers of English, so I read a hell of a lot of papers, theses, and grant applications, while also continuing to publish and peer-review in my own field. My impression is that peer review in the social sciences and history isn’t necessarily the major force for conformance to whatever contemporary social narratives. Papers often deal with minutiae, and both author and peer reviewers alike are nerds who like delving into those minutiae. They don’t necessarily want to be drawn into any wider topic like in the case of the paper discussed in this article. Rather, the force for ideological conformity may instead be funding bodies. I have seen so many grant applications where the author(s) clearly want to explore minutiae, but are forced to appeal to some grand social-justice cause in order to secure funding. I’ve participated through series of revisions of grant applications where 2–3 paragraphs claiming the research would benefit some minority or another, are inserted at a late stage after someone mentions that the application would stand no chance without them.
- davidktr 3y ago
  Interesting perspective, thanks for sharing.
- paganel 3y ago
  > claiming the research would benefit some minority or another, are inserted at a late stage after someone mentions that the application would stand no chance without them. Funny, it's similar to "forced" mentions of Marxism inserted apparently at will in social science books/papers written in Romanian back when the Wall was still up. Towards the later years of the communist government being in power those Marx insertions had been replaced by quotes to what Ceausescu had said.
  - beebeepka 3y ago
    Was Marx anything like Ceausescu?
    - dsign 3y ago
      I guess the OP is referring to how communist countries are/were big in personality cult and general brainwashing. Marx and his ideology when they were happy with the soviets, Ceausescu later when the political winds changed. Generally though, people have a tendency to seek flags to rally behind, be it personalities, ideologies or religions. There is nothing surprising on that, given that the success of our species relies on coordinating large numbers of people. The problem is that, sometimes, it is very difficult or impossible for the individual to choose the "right" flag, whatever the definition of right may be. In communist and totalitarian countries, the problem is solved for the individual by allowing only one flag. In countries with good democracy, e.g. USA[^1], it sometimes happens that the population divides evenly between the flags available to them (i.e. Republicans and Democrats). [^1]: The US' democracy has many faults, but it's a shining thing compared with a lot of countries in the world.
watwut 3y ago
Except you made that accusation up. In all likelihood, you do not know how censorship in USSR functioned (because the hints you provide suggest so). And in all likelihood, you know zero about what is in historical articles or how peer review functions.

cauch 3y ago

But how could the reviewers have found the errors in this article? The article can tell "Cort was informed about the new techniques from a family member back from Jamaica". How do you verify that? The only way is to redo the all work from scratch, refound all the historical references and cross-validate each of them. It's a huge work. This article was later debunked by experts who have taken time and effort to do that. But peer-reviewing does not allow reviewers to pause they own work to do such extensive checks. More realistically, the article passes the peer-reviewing process because the peer-reviewing process does what it is supposed to do: the article is "believable", it does not have things that looks not coming from the scientific process. The reviewers don't have time and it is in fact not even their jobs to redo the study. All they do is to check if the guidelines seem to be respected. If you want more than that, you want a "replication study", which is the next step in the scientific process of building trust on a study (and which also have difficulties). And, sure, they may have flagged that some conclusions may need more convincing demonstration, but it is a Gaussian curve: sometimes they do, sometimes they don't, some passes through the gaps. I think there is a problem with layman people who don't understand that the peer review process is not a magical tool that remove all the incorrect studies (on top of that, due to statistical fluctuation, some studies are false while the authors have done 100% everything perfectly). I'm not saying it's the case here, but this is one explanation that should not be ignored. Ironically, it is a bit funny that for your conclusion you jumped immediately on the story you wish to be true: social sciences or history are ideologically biased. This is a valid hypothesis, but not the only one explaining what we observe.

Apocryphon 3y ago
> The article can tell "Cort was informed about the new techniques from a family member back from Jamaica". How do you verify that? The only way is to redo the all work from scratch, refound all the historical references and cross-validate each of them. It's a huge work. Since this is about checking textual references, as opposed to laboratory work, would it be possible for an LLM to do that? Seems like the hardest thing would be for the A.I. to log into the requisite gateways for the databases hosting the papers.
- cauch 3y ago
  I doubt it is so naively simple as "querying a database". For example, before the publication of this article, the majority of the "databases" were simply saying "the inventor is Cort", which is one example of thing that the A.I. will get trivially "wrong" when reviewing the paper. And if it is true, then this A.I. review method can also be applied to mathematics, theoretical physics, computer science, ... And even if it is doable, you will still have things that will pass the review process when ideally it should not. It is just an hard limitation, same as the one in "justice" (impossible to not sometimes judge a guilty person "innocent" or an innocent person "guilty", anyone who thinks otherwise just don't understand how complicated it is)
- nvm0n2 3y ago
  It probably can be done, but does the academic establishment really want word to get out that their great research is being auto-reviewed because the "peers" were so bad at their job that an AI can do better? They've spent decades pushing the idea that peer review is a gold standard on the world, it would be quite a climbdown to admit ChatGPT can do it better.
whatshisface 3y ago
>How do you verify that? The only way is to redo the all work from scratch, refound all the historical references and cross-validate each of them. It's a huge work. Sorry, but if the paper doesn't make its case, it isn't an academic work. If it needs to be 1,000 papers in order to explain why its assertions are reliable, great, that's 1,000 good papers to write. :)*

OfSanguineFire 3y ago

I provide language revision for academics who are non-native speakers of English, so I read a hell of a lot of papers, theses, and grant applications, while also continuing to publish and peer-review in my own field. My impression is that peer review in the social sciences and history isn’t necessarily the major force for conformance to whatever contemporary social narratives. Papers often deal with minutiae, and both author and peer reviewers alike are nerds who like delving into those minutiae. They don’t necessarily want to be drawn into any wider topic like in the case of the paper discussed in this article. Rather, the force for ideological conformity may instead be funding bodies. I have seen so many grant applications where the author(s) clearly want to explore minutiae, but are forced to appeal to some grand social-justice cause in order to secure funding. I’ve participated through series of revisions of grant applications where 2–3 paragraphs claiming the research would benefit some minority or another, are inserted at a late stage after someone mentions that the application would stand no chance without them.

davidktr 3y ago
Interesting perspective, thanks for sharing.
paganel 3y ago
> claiming the research would benefit some minority or another, are inserted at a late stage after someone mentions that the application would stand no chance without them. Funny, it's similar to "forced" mentions of Marxism inserted apparently at will in social science books/papers written in Romanian back when the Wall was still up. Towards the later years of the communist government being in power those Marx insertions had been replaced by quotes to what Ceausescu had said.
- beebeepka 3y ago
  Was Marx anything like Ceausescu?
  - dsign 3y ago
    I guess the OP is referring to how communist countries are/were big in personality cult and general brainwashing. Marx and his ideology when they were happy with the soviets, Ceausescu later when the political winds changed. Generally though, people have a tendency to seek flags to rally behind, be it personalities, ideologies or religions. There is nothing surprising on that, given that the success of our species relies on coordinating large numbers of people. The problem is that, sometimes, it is very difficult or impossible for the individual to choose the "right" flag, whatever the definition of right may be. In communist and totalitarian countries, the problem is solved for the individual by allowing only one flag. In countries with good democracy, e.g. USA[^1], it sometimes happens that the population divides evenly between the flags available to them (i.e. Republicans and Democrats). [^1]: The US' democracy has many faults, but it's a shining thing compared with a lot of countries in the world.

watwut 3y ago

Except you made that accusation up. In all likelihood, you do not know how censorship in USSR functioned (because the hints you provide suggest so). And in all likelihood, you know zero about what is in historical articles or how peer review functions.