5 ms·

But how could the reviewers have found the errors in this article? The article can tell "Cort was informed about the new techniques from a family member back f

by cauch 3y ago

But how could the reviewers have found the errors in this article? The article can tell "Cort was informed about the new techniques from a family member back from Jamaica". How do you verify that? The only way is to redo the all work from scratch, refound all the historical references and cross-validate each of them. It's a huge work. This article was later debunked by experts who have taken time and effort to do that. But peer-reviewing does not allow reviewers to pause they own work to do such extensive checks. More realistically, the article passes the peer-reviewing process because the peer-reviewing process does what it is supposed to do: the article is "believable", it does not have things that looks not coming from the scientific process. The reviewers don't have time and it is in fact not even their jobs to redo the study. All they do is to check if the guidelines seem to be respected. If you want more than that, you want a "replication study", which is the next step in the scientific process of building trust on a study (and which also have difficulties). And, sure, they may have flagged that some conclusions may need more convincing demonstration, but it is a Gaussian curve: sometimes they do, sometimes they don't, some passes through the gaps. I think there is a problem with layman people who don't understand that the peer review process is not a magical tool that remove all the incorrect studies (on top of that, due to statistical fluctuation, some studies are false while the authors have done 100% everything perfectly). I'm not saying it's the case here, but this is one explanation that should not be ignored. Ironically, it is a bit funny that for your conclusion you jumped immediately on the story you wish to be true: social sciences or history are ideologically biased. This is a valid hypothesis, but not the only one explaining what we observe.

Apocryphon 3y ago
> The article can tell "Cort was informed about the new techniques from a family member back from Jamaica". How do you verify that? The only way is to redo the all work from scratch, refound all the historical references and cross-validate each of them. It's a huge work. Since this is about checking textual references, as opposed to laboratory work, would it be possible for an LLM to do that? Seems like the hardest thing would be for the A.I. to log into the requisite gateways for the databases hosting the papers.
- cauch 3y ago
  I doubt it is so naively simple as "querying a database". For example, before the publication of this article, the majority of the "databases" were simply saying "the inventor is Cort", which is one example of thing that the A.I. will get trivially "wrong" when reviewing the paper. And if it is true, then this A.I. review method can also be applied to mathematics, theoretical physics, computer science, ... And even if it is doable, you will still have things that will pass the review process when ideally it should not. It is just an hard limitation, same as the one in "justice" (impossible to not sometimes judge a guilty person "innocent" or an innocent person "guilty", anyone who thinks otherwise just don't understand how complicated it is)
- nvm0n2 3y ago
  It probably can be done, but does the academic establishment really want word to get out that their great research is being auto-reviewed because the "peers" were so bad at their job that an AI can do better? They've spent decades pushing the idea that peer review is a gold standard on the world, it would be quite a climbdown to admit ChatGPT can do it better.
whatshisface 3y ago
>How do you verify that? The only way is to redo the all work from scratch, refound all the historical references and cross-validate each of them. It's a huge work. Sorry, but if the paper doesn't make its case, it isn't an academic work. If it needs to be 1,000 papers in order to explain why its assertions are reliable, great, that's 1,000 good papers to write. :)*
robertlagrant 3y ago
> This article was later debunked by experts who have taken time and effort to do that. But peer-reviewing does not allow reviewers to pause they own work to do such extensive checks. Is this not a problem? Reviews from non-experts?
slowmovintarget 3y ago
> But how could the reviewers have found the errors in this article? The same way this guy wrote the thorough debunking of the paper: https://osf.io/preprints/socarxiv/rp5ae/ https://osf.io/preprints/socarxiv/rp5ae/ ...by actually reviewing the citations and noticing they do not, at all, support the claims she made. > ...such extensive checks... The checks didn't need to be extensive, they needed to be cursory. The problem is they were absent because the reviewers enjoyed the same political bent as the author of the paper.
nvm0n2 3y ago
> This article was later debunked by experts who have taken time and effort to do that It was debunked by a single non-academic with no publishing track record, who is currently completing a master's thesis. In other words it could have been debunked by anyone. > the article passes the peer-reviewing process because the peer-reviewing process does what it is supposed to do Indeed, it passed the process because the process is designed to create the illusion of consensus and credibility without actually blocking false claims. It worked wonderfully here. > I think there is a problem with layman people Yes that's right, those unwashed masses who expect scholars to not make shit up, who are these scoundrels? Do they not understand how the great processes of discovery work? > social sciences or history are ideologically biased. This is a valid hypothesis, but not the only one explaining what we observe. Yes it is the only one explaining what we observe. The claims were made up, completed with fake references, there is no scope for statistical P-hacking "oh bad luck old boy" explanations here (not that they're acceptable anyway). It's not just scholars, the whole left is like this. Black Woman With Magical Powers is by now a Netflix trope. You can guess the plot of a lot of post-2015 Hollywood output just by looking at the race of the characters. This sort of scam is exactly what we'd expect from an academic culture in a state of advanced decay.
- msrenee 3y ago
  I was with you on most of your points until you extrapolated this to "the left" and rolled on into Netflix shows as evidence of the degradation of academia due to politics you don't agree with.
  - nvm0n2 3y ago
    Do you think this attempt to rewrite British history isn't ideological in nature?
- cauch 3y ago
  [flagged]
msrenee 3y ago
>The article can tell "Cort was informed about the new techniques from a family member back from Jamaica". How do you verify that? The only way is to redo the all work from scratch, refound all the historical references and cross-validate each of them. It's a huge work. You start by going through the citations the author has listed and seeing if those support their reasoning. Sounds like that's what the non-expert in the linked article did. It turned out that some of what she claimed was not supported by or was in direct opposition to the sources in her citations. That's what citations are for. They're listed at the end of the paper in a standardized way so they can be easily accessed by anyone who wants to know where an author got a piece of information from. If this wasn't possible, there would be no point to peer review. I can understand an article getting through where the author falsified experimental data. That's hard to refute unless you can find evidence of the false data or you run the experiment again yourself. A history article, though? If you're not double-checking dubious claims by see what sources the author is using to support them, you may as well not bother reviewing.
- cauch 3y ago
  I agree with checking the references, but again, my point is that as a general rule it's really not that easy. You can think it's easy because you have had the work done for you for this particular article and that it is obvious in this particular article, but in general, it is way more complicated. Properly checking that the citation is indeed saying that often requires studying the quoted paper quite in details. Again, it may not have been the case here, but it is an amount of work that a reviewer should plan in advance. But more importantly, and it makes your point moot: if the article is misrepresentating the reference, then, rather than in the peer-review process, it will be the reference author that will notice it. I think it is my point: people here are seeing peer-review and are thinking that it is the ultimate method to decide what is good or bad (or what is "consensus"). This is ridiculous. A peer-reviewed article has NEVER been considered as "obviously correct and uncriticable". A peer-review is just one of the step in the long process. If indeed the references were not saying what the article is saying they say, it will end up being discovered. The peer-review is the first step before submitting the article to the whole community, and it is then that the majority of the discussion happens. > If you're not double-checking dubious claims While I agree with the previous things, this bit makes me pause: careful with that, historically, the majority of correct claims were, at first, sounding dubious, and the majority of incorrect claims were, at first, sounding totally not dubious. Let's rather say: let's double-check everything.

by cauch 3y ago

Apocryphon 3y ago

cauch 3y ago
I doubt it is so naively simple as "querying a database". For example, before the publication of this article, the majority of the "databases" were simply saying "the inventor is Cort", which is one example of thing that the A.I. will get trivially "wrong" when reviewing the paper. And if it is true, then this A.I. review method can also be applied to mathematics, theoretical physics, computer science, ... And even if it is doable, you will still have things that will pass the review process when ideally it should not. It is just an hard limitation, same as the one in "justice" (impossible to not sometimes judge a guilty person "innocent" or an innocent person "guilty", anyone who thinks otherwise just don't understand how complicated it is)
nvm0n2 3y ago
It probably can be done, but does the academic establishment really want word to get out that their great research is being auto-reviewed because the "peers" were so bad at their job that an AI can do better? They've spent decades pushing the idea that peer review is a gold standard on the world, it would be quite a climbdown to admit ChatGPT can do it better.

whatshisface 3y ago

>How do you verify that? The only way is to redo the all work from scratch, refound all the historical references and cross-validate each of them. It's a huge work. Sorry, but if the paper doesn't make its case, it isn't an academic work. If it needs to be 1,000 papers in order to explain why its assertions are reliable, great, that's 1,000 good papers to write. :)*

robertlagrant 3y ago

> This article was later debunked by experts who have taken time and effort to do that. But peer-reviewing does not allow reviewers to pause they own work to do such extensive checks. Is this not a problem? Reviews from non-experts?

slowmovintarget 3y ago

> But how could the reviewers have found the errors in this article? The same way this guy wrote the thorough debunking of the paper: https://osf.io/preprints/socarxiv/rp5ae/ https://osf.io/preprints/socarxiv/rp5ae/ ...by actually reviewing the citations and noticing they do not, at all, support the claims she made. > ...such extensive checks... The checks didn't need to be extensive, they needed to be cursory. The problem is they were absent because the reviewers enjoyed the same political bent as the author of the paper.

nvm0n2 3y ago

> This article was later debunked by experts who have taken time and effort to do that It was debunked by a single non-academic with no publishing track record, who is currently completing a master's thesis. In other words it could have been debunked by anyone. > the article passes the peer-reviewing process because the peer-reviewing process does what it is supposed to do Indeed, it passed the process because the process is designed to create the illusion of consensus and credibility without actually blocking false claims. It worked wonderfully here. > I think there is a problem with layman people Yes that's right, those unwashed masses who expect scholars to not make shit up, who are these scoundrels? Do they not understand how the great processes of discovery work? > social sciences or history are ideologically biased. This is a valid hypothesis, but not the only one explaining what we observe. Yes it is the only one explaining what we observe. The claims were made up, completed with fake references, there is no scope for statistical P-hacking "oh bad luck old boy" explanations here (not that they're acceptable anyway). It's not just scholars, the whole left is like this. Black Woman With Magical Powers is by now a Netflix trope. You can guess the plot of a lot of post-2015 Hollywood output just by looking at the race of the characters. This sort of scam is exactly what we'd expect from an academic culture in a state of advanced decay.

msrenee 3y ago
I was with you on most of your points until you extrapolated this to "the left" and rolled on into Netflix shows as evidence of the degradation of academia due to politics you don't agree with.
- nvm0n2 3y ago
  Do you think this attempt to rewrite British history isn't ideological in nature?
cauch 3y ago
[flagged]

msrenee 3y ago

>The article can tell "Cort was informed about the new techniques from a family member back from Jamaica". How do you verify that? The only way is to redo the all work from scratch, refound all the historical references and cross-validate each of them. It's a huge work. You start by going through the citations the author has listed and seeing if those support their reasoning. Sounds like that's what the non-expert in the linked article did. It turned out that some of what she claimed was not supported by or was in direct opposition to the sources in her citations. That's what citations are for. They're listed at the end of the paper in a standardized way so they can be easily accessed by anyone who wants to know where an author got a piece of information from. If this wasn't possible, there would be no point to peer review. I can understand an article getting through where the author falsified experimental data. That's hard to refute unless you can find evidence of the false data or you run the experiment again yourself. A history article, though? If you're not double-checking dubious claims by see what sources the author is using to support them, you may as well not bother reviewing.

cauch 3y ago
I agree with checking the references, but again, my point is that as a general rule it's really not that easy. You can think it's easy because you have had the work done for you for this particular article and that it is obvious in this particular article, but in general, it is way more complicated. Properly checking that the citation is indeed saying that often requires studying the quoted paper quite in details. Again, it may not have been the case here, but it is an amount of work that a reviewer should plan in advance. But more importantly, and it makes your point moot: if the article is misrepresentating the reference, then, rather than in the peer-review process, it will be the reference author that will notice it. I think it is my point: people here are seeing peer-review and are thinking that it is the ultimate method to decide what is good or bad (or what is "consensus"). This is ridiculous. A peer-reviewed article has NEVER been considered as "obviously correct and uncriticable". A peer-review is just one of the step in the long process. If indeed the references were not saying what the article is saying they say, it will end up being discovered. The peer-review is the first step before submitting the article to the whole community, and it is then that the majority of the discussion happens. > If you're not double-checking dubious claims While I agree with the previous things, this bit makes me pause: careful with that, historically, the majority of correct claims were, at first, sounding dubious, and the majority of incorrect claims were, at first, sounding totally not dubious. Let's rather say: let's double-check everything.