12 ms·

ChatGPT and other AI tools could disrupt scientific publishing

RandomLensman 3y ago
In the end, it will increase standardization and conformity, while at the same time helping non-English speakers to write things. Whether that is net good or bad is difficult to say: broader perimeter of publishing vs narrower way to portray findings.
- phist_mcgee 3y ago
  Not sure about you, but I sometimes simply stop reading an article if the tone or language is 1 SD away from "Standard English". It can be hard enough understanding the content of a paper without having to interpret and parse the language they're using.
  - RandomLensman 3y ago
    Exactly, it sort of increases the effort and decreases likelihood of reading, so support for authors there is a good thing. At the same time, the LLM tools will do more than just enhance the language in a narrow way but rather also "streamline" it, which reduces what can be expressed. That probably will lead to a drop in the maximum complexity that can be expressed. We already see that in other areas where language is already down to really simple stuff with all complexity being removed (whether it is there or not.
    - falcor84 3y ago
      >That probably will lead to a drop in the maximum complexity that can be expressed. I'm not following your argument here. Are you saying that rather than getting help from an LLM, future researchers will be forced to use the LLM's output as-is?
      - RandomLensman 3y ago
        No. However, there will be expectations forming around the language in papers based on how LLMs use language: People using LLMs to enhance readability will have an impact as the language would gravitate towards something like "maximum likelihood" text (incl text length). It will be very easy to read but also limits what can be expressed within such a "corset".
        falcor84 3y ago
        If editors make such a decision, how would that be any different than a publication deciding to set a readability bar based on something like Flesch–Kincaid?
  - DoingIsLearning 3y ago
    You can request an LLM to write in a particular style. You can request a paragraph to be written in an Allan Poe style (complex prose) or in the style of US Navy Technical Manuals (simple to the point language). What we associate with chatGPT style prose is just the model's 'default voice' but can arguably be curbed.
    - phist_mcgee 3y ago
      I'd never actually thought to get chatgpt to rewrite papers for me, there's some very badly written ones (fully aware that English may not be easy for many people to write in) papers that I have been setting aside for ages.
    - RandomLensman 3y ago
      While true, scientific papers are written with an aim to be reasonably easy to parse. Turning scientific papers into literary works is likely to fail (novelty bits aside), same as it fails in business settings etc. We already have pretty streamlined language because people demand it in a lot of places - LLMs will accelerate that because it reduces the effort to get there.
      - DoingIsLearning 3y ago
        I think you misinterpreted. My point is that the output style of LLMs like chatGPT is not static. The 'default voice' uses fairly embellished language but there is nothing stopping a user from requesting a plain language output, such as what you would find in a US Navy manual (whose target audience is a huge cohort) .
        RandomLensman 3y ago
        I know, but my point is that people will converge on a certain compact and simple style for scientific papers because that will be favored for fast and easy reading (as seen in business, for example). Such a style isn't necessarily easy to write but the LLMs will help people to get there.
arpa 3y ago
less bullshit would be nice than more but we're so good at polluting, we'll just start shitting in the noosphere as well until it also becomes uninhabitable for humans just like our own planet.
jasfi 3y ago
There's a wrong way and a right way to use ChatGPT/AI tools for scientific publishing. The right way is as a source of ideas and for feedback. The wrong way asking ChatGPT to write anything for you, which is sent off to be published. I'm quite sure that people who use the wrong way will end up regretting it.
- paulluuk 3y ago
  And yet, if your article gets rejected when you write it yourself, and gets published when you have AI rewrite it for you to use all the right buzzwords based on previously published articles -- that's a good (financial) reason to use AI for writing.
  - jrimbault 3y ago
    Fascinating how we (society) have let KPIs get into research/education and healthcare.
    - oefrha 3y ago
      Research inherently requires attention and attention is inherently scarce, so unfortunately people do and will compete for attention however they can, metrics or not.
    - JumpCrisscross 3y ago
      > let KPIs get into research/education and healthcare Research, sure, that’s a creative profession. But education and healthcare have obviously measurable outcomes, some of which are predictably desired ex ante.
      - jrimbault 3y ago
        The implicit here is "business/for profit" KPIs.
    - godelski 3y ago
      This is why I've started to argue against "peer review." Because no one can decide a paper's correctness/legitimacy by sitting down and reading the work. The method just doesn't scale and certainly doesn't in settings that are zero sum like conferences (see the absolute shitshow that is ML peer review). It's become a entire waste of time and money. The alternative I suggest is just submitting works to Open Review so there can be open discussions on the works. It removes the hostility from the setting and self biases towards reviewers/commenters being parties that are experts in that sub-niche as they're going to tend to be the only ones reading those papers, and especially reading closely. Metrics are fucking hard and we don't improve systems by removing nuance from metrics. Yet as complexity has increased in our world, this has seemed to be the direction we've gone. It's bad for science and the current methods have far too many false negatives (improper rejects) and false positives (improper accepts) to validate it as a meaningful signal. We've tried so many ways to fix it while maintaining the institution that it's time to just try letting go of it. I honestly cannot find a great argument for keeping these journals around (yes, there are arguments, but not many ,,strong,, arguments, and specifically ones that lead to better science).
test77777 3y ago
[dead]
raister 3y ago
I can imagine in the future authors selecting a checkbox with "I authored this manuscript without the help of any IA." before submitting a paper for review. AND if caught, they would have the 'power' to lift the paper from their databases or mark it somehow (brand it with a (IA) logo perhaps?)
- dagw 3y ago
  "I authored this manuscript without the help of any IA." Where do you draw the line? Tools like Grammarly and similar are AI. Using google translate to translate a phrase from your native language to English to is AI. Hell the spellchecker that just autocorrected a typo I made in the previous sentence is probably AI. Should they all be banned?
jillesvangurp 3y ago
People are focusing on the negative stuff here (lazy researchers "cheating"). But there's also a positive side to this. Generative AI is the ultimate research tool. It already knows the vast majority of research that you'll never get around to reading. You could spend your entire life reading every last second of it and you wouldn't come close to catching up. A lot of that stuff may be irrelevant (to your context) but it still knows about it. And you can ask it questions about anything it ingested. Or ask it to criticize your text, find analogies to other work, or generally perform the role of a really diligent peer reviewer and editor before you even submit the work. That's all highly useful and it should lead to a higher quality of work for those researchers that use their tools well. I use gpt for Google docs and it's definitely helping me improve my text. I don't let it write whole sections/paragraphs. But I do use it to criticize, critique, and suggest improvements. I imagine a lot of students and researchers have been doing the same for the last year or so. The same goes for reviewers. They can ask to extract key points, analyze the argumentation, find related work that the author might have missed, figure out where the authors are taking a few liberties with the facts/literature, etc. much more easily. I've reviewed a fair amount of mostly badly written papers back in the day. This is not fun and rather laborious work. A lot of academic life is basically about reading each other's work and providing (hopefully) constructive criticism for articles that ultimately don't make the cut. I got rather good at that when I was still doing that. Any workshop, conference, or journal ends up rejecting way more articles than they accept. Especially the better publications. Some poor souls have to read all the rejected stuff. The price you pay for getting accepted is helping out with the peer reviews. The process can be biased, political, unfair, and sometimes harsh. But it's better than not having a process. Generative AI can help with challenging unfair peer reviews, help reviewers extract key points, and zoom in on novel ideas/theories. Ultimately what you look for in an article is: Does it contribute something novel? Is the work contextualized properly relative to prior work? Is the work sound in its reasoning? Etc. Answering such questions positively basically means it's a good article. A generative AI can save a lot of time with this. Weeding out the bad articles is not that hard but a lot of work.
- hashtag-til 3y ago
  > Generative AI can help with challenging unfair peer reviews Until people figure out which words to add in order to game the AI peer reviewer, or good work is missed because the lack of buzzwords.
oersted 3y ago
Just to plug that we have built MirrorThink.ai: an AI assistant specifically designed to help professional researchers in their day-to-day work. (I am aware that self-promotion is allowed but somewhat frowned-upon in HN. I believe this is relevant to the context and I am particularly qualified to comment on it. But I'll strive to be as transparent as possible.) We have intentionally steered away from assisting with writing work for the reasons outlined in the article. Our focus has been to save time in keeping up with their field: making literature reviews much faster, summarizing recent discoveries in each researchers' niche area of interest, and gathering how others have tackled similar highly-specialized research problems. Grounding LLMs in scientific fact and always providing verifiable sources. We have spent many years before this building a business around our "Map of Science" at Scitodate, so we have strong connections with the community and a lot of experience in this domain. Both in terms of how science is done and how to reliably extract valuable knowledge from millions of documents with NLP. We now have one of the largest datasets of papers, patents, funding grants... Updated on a daily basis. With plenty of enrichment, connections and aggregations: we have done a lot of in-house research on disambiguation and entity-linking. We particularly have focused on mapping out the expertise, interests and achievements of all professional scientists, as well as the institutions they have worked at.
- icapybara 3y ago
  How do you get the chatbot to tell the truth more? I’m a scientist and this is my biggest showstopper with AI tools. I just can’t know if I can trust what it’s telling me.
  - unblough 3y ago
    You are unable to. “LLMs can’t self-correct in reasoning tasks, DeepMind study finds“ https://news.ycombinator.com/item?id=37823543 https://news.ycombinator.com/item?id=37823543 Anyone who says otherwise is either ignorant of the underlying function of llms or trying to sell you something.
    - mistermann 3y ago
hashtag-til 3y ago
I think using any AI tool that reports your findings for you represents failure and laziness in science. It will become a circle: somebody uses AI to generate boring text (probably not even themselves will read it), then somebody else puts the boring text for AI to summarise. Nobody learns anything. Repeat.
lynx23 3y ago
This reminds me of M-x write-thesis RET, a usenet joke from the 90s IIRC. Apparently, since google is no longer indexing the past, I couldn't find a link. However, I still remember the post vividly, and it makes me sad that 25 years later, we have to have this discussion for real now. Back then, everyone in their right mind knew why automated scientific publication writing would be hilariously bad. These days, we have articles like this. Weird.
otabdeveloper4 3y ago
SaaS Sokal-as-a-Service.
streamfunk191 3y ago
It’s a shame the article doesn’t talk more about the possible benefits of AI in academia. The ability for researchers to spot new patterns, discover new areas of research. The way information is generated, disseminated and then read by other researchers in this space is fairly prosaic if you think about it. Causally is a company I think is trying something in this space
- izacus 3y ago
  That's because it's not how it's used right now. Just like elsewhere, it's primarly used to generate bullsh*t and flood journals with low quality generated content. Which now need to deal with this deluge of sewage drowning good articles.
  - hashtag-til 3y ago
    Yep, I know quite a few gaming the system just like this, selling the AI dream...
- aleph_minus_one 3y ago
  > It’s a shame the article doesn’t talk more about the possible benefits of AI in academia. The ability for researchers to spot new patterns, discover new areas of research. Honestly, which publicly available AIs are capable of doing that? At least not the publicly available LLM AIs: I asked those quite many questions about scientific topics, but they could not give me any answer that helped me - it was a complete waste of time.
dist-epoch 3y ago
In the spirit of open, reproducible science, I think the prompts should appear in the final papers. Something like: [Summarize the prior research in paper XXX.YYY focusing on the DNA synthesis section] Bla BLA BLA [Describe in two paragraphs how DNA synthesis works] BLA BLA BLA
Moldoteck 3y ago
imo scientific publishing was already ruined by lack of reproducibility for a lot of papers, even for software/machine learning-focused ones
hoseja 3y ago
Elsevier and other corporate tools could disrupt scientific publishing
lapama 3y ago
Today I wrote a budget description for a scientific proposal that an AI would not be able to match. The reason? My knowledge of the social context.

In the end, it will increase standardization and conformity, while at the same time helping non-English speakers to write things. Whether that is net good or bad is difficult to say: broader perimeter of publishing vs narrower way to portray findings.

phist_mcgee 3y ago
Not sure about you, but I sometimes simply stop reading an article if the tone or language is 1 SD away from "Standard English". It can be hard enough understanding the content of a paper without having to interpret and parse the language they're using.
- RandomLensman 3y ago
  Exactly, it sort of increases the effort and decreases likelihood of reading, so support for authors there is a good thing. At the same time, the LLM tools will do more than just enhance the language in a narrow way but rather also "streamline" it, which reduces what can be expressed. That probably will lead to a drop in the maximum complexity that can be expressed. We already see that in other areas where language is already down to really simple stuff with all complexity being removed (whether it is there or not.
  - falcor84 3y ago
    >That probably will lead to a drop in the maximum complexity that can be expressed. I'm not following your argument here. Are you saying that rather than getting help from an LLM, future researchers will be forced to use the LLM's output as-is?
    - RandomLensman 3y ago
      No. However, there will be expectations forming around the language in papers based on how LLMs use language: People using LLMs to enhance readability will have an impact as the language would gravitate towards something like "maximum likelihood" text (incl text length). It will be very easy to read but also limits what can be expressed within such a "corset".
      - falcor84 3y ago
        If editors make such a decision, how would that be any different than a publication deciding to set a readability bar based on something like Flesch–Kincaid?
- DoingIsLearning 3y ago
  You can request an LLM to write in a particular style. You can request a paragraph to be written in an Allan Poe style (complex prose) or in the style of US Navy Technical Manuals (simple to the point language). What we associate with chatGPT style prose is just the model's 'default voice' but can arguably be curbed.
  - phist_mcgee 3y ago
    I'd never actually thought to get chatgpt to rewrite papers for me, there's some very badly written ones (fully aware that English may not be easy for many people to write in) papers that I have been setting aside for ages.
  - RandomLensman 3y ago
    While true, scientific papers are written with an aim to be reasonably easy to parse. Turning scientific papers into literary works is likely to fail (novelty bits aside), same as it fails in business settings etc. We already have pretty streamlined language because people demand it in a lot of places - LLMs will accelerate that because it reduces the effort to get there.
    - DoingIsLearning 3y ago
      I think you misinterpreted. My point is that the output style of LLMs like chatGPT is not static. The 'default voice' uses fairly embellished language but there is nothing stopping a user from requesting a plain language output, such as what you would find in a US Navy manual (whose target audience is a huge cohort) .
      - RandomLensman 3y ago
        I know, but my point is that people will converge on a certain compact and simple style for scientific papers because that will be favored for fast and easy reading (as seen in business, for example). Such a style isn't necessarily easy to write but the LLMs will help people to get there.

arpa 3y ago

less bullshit would be nice than more but we're so good at polluting, we'll just start shitting in the noosphere as well until it also becomes uninhabitable for humans just like our own planet.

jasfi 3y ago

There's a wrong way and a right way to use ChatGPT/AI tools for scientific publishing. The right way is as a source of ideas and for feedback. The wrong way asking ChatGPT to write anything for you, which is sent off to be published. I'm quite sure that people who use the wrong way will end up regretting it.

paulluuk 3y ago
And yet, if your article gets rejected when you write it yourself, and gets published when you have AI rewrite it for you to use all the right buzzwords based on previously published articles -- that's a good (financial) reason to use AI for writing.
- jrimbault 3y ago
  Fascinating how we (society) have let KPIs get into research/education and healthcare.
  - oefrha 3y ago
    Research inherently requires attention and attention is inherently scarce, so unfortunately people do and will compete for attention however they can, metrics or not.
  - JumpCrisscross 3y ago
    > let KPIs get into research/education and healthcare Research, sure, that’s a creative profession. But education and healthcare have obviously measurable outcomes, some of which are predictably desired ex ante.
    - jrimbault 3y ago
      The implicit here is "business/for profit" KPIs.
  - godelski 3y ago
    This is why I've started to argue against "peer review." Because no one can decide a paper's correctness/legitimacy by sitting down and reading the work. The method just doesn't scale and certainly doesn't in settings that are zero sum like conferences (see the absolute shitshow that is ML peer review). It's become a entire waste of time and money. The alternative I suggest is just submitting works to Open Review so there can be open discussions on the works. It removes the hostility from the setting and self biases towards reviewers/commenters being parties that are experts in that sub-niche as they're going to tend to be the only ones reading those papers, and especially reading closely. Metrics are fucking hard and we don't improve systems by removing nuance from metrics. Yet as complexity has increased in our world, this has seemed to be the direction we've gone. It's bad for science and the current methods have far too many false negatives (improper rejects) and false positives (improper accepts) to validate it as a meaningful signal. We've tried so many ways to fix it while maintaining the institution that it's time to just try letting go of it. I honestly cannot find a great argument for keeping these journals around (yes, there are arguments, but not many ,,strong,, arguments, and specifically ones that lead to better science).

test77777 3y ago

[dead]

raister 3y ago

I can imagine in the future authors selecting a checkbox with "I authored this manuscript without the help of any IA." before submitting a paper for review. AND if caught, they would have the 'power' to lift the paper from their databases or mark it somehow (brand it with a (IA) logo perhaps?)

dagw 3y ago
"I authored this manuscript without the help of any IA." Where do you draw the line? Tools like Grammarly and similar are AI. Using google translate to translate a phrase from your native language to English to is AI. Hell the spellchecker that just autocorrected a typo I made in the previous sentence is probably AI. Should they all be banned?

jillesvangurp 3y ago

People are focusing on the negative stuff here (lazy researchers "cheating"). But there's also a positive side to this. Generative AI is the ultimate research tool. It already knows the vast majority of research that you'll never get around to reading. You could spend your entire life reading every last second of it and you wouldn't come close to catching up. A lot of that stuff may be irrelevant (to your context) but it still knows about it. And you can ask it questions about anything it ingested. Or ask it to criticize your text, find analogies to other work, or generally perform the role of a really diligent peer reviewer and editor before you even submit the work. That's all highly useful and it should lead to a higher quality of work for those researchers that use their tools well. I use gpt for Google docs and it's definitely helping me improve my text. I don't let it write whole sections/paragraphs. But I do use it to criticize, critique, and suggest improvements. I imagine a lot of students and researchers have been doing the same for the last year or so. The same goes for reviewers. They can ask to extract key points, analyze the argumentation, find related work that the author might have missed, figure out where the authors are taking a few liberties with the facts/literature, etc. much more easily. I've reviewed a fair amount of mostly badly written papers back in the day. This is not fun and rather laborious work. A lot of academic life is basically about reading each other's work and providing (hopefully) constructive criticism for articles that ultimately don't make the cut. I got rather good at that when I was still doing that. Any workshop, conference, or journal ends up rejecting way more articles than they accept. Especially the better publications. Some poor souls have to read all the rejected stuff. The price you pay for getting accepted is helping out with the peer reviews. The process can be biased, political, unfair, and sometimes harsh. But it's better than not having a process. Generative AI can help with challenging unfair peer reviews, help reviewers extract key points, and zoom in on novel ideas/theories. Ultimately what you look for in an article is: Does it contribute something novel? Is the work contextualized properly relative to prior work? Is the work sound in its reasoning? Etc. Answering such questions positively basically means it's a good article. A generative AI can save a lot of time with this. Weeding out the bad articles is not that hard but a lot of work.

hashtag-til 3y ago
> Generative AI can help with challenging unfair peer reviews Until people figure out which words to add in order to game the AI peer reviewer, or good work is missed because the lack of buzzwords.

oersted 3y ago

Just to plug that we have built MirrorThink.ai: an AI assistant specifically designed to help professional researchers in their day-to-day work. (I am aware that self-promotion is allowed but somewhat frowned-upon in HN. I believe this is relevant to the context and I am particularly qualified to comment on it. But I'll strive to be as transparent as possible.) We have intentionally steered away from assisting with writing work for the reasons outlined in the article. Our focus has been to save time in keeping up with their field: making literature reviews much faster, summarizing recent discoveries in each researchers' niche area of interest, and gathering how others have tackled similar highly-specialized research problems. Grounding LLMs in scientific fact and always providing verifiable sources. We have spent many years before this building a business around our "Map of Science" at Scitodate, so we have strong connections with the community and a lot of experience in this domain. Both in terms of how science is done and how to reliably extract valuable knowledge from millions of documents with NLP. We now have one of the largest datasets of papers, patents, funding grants... Updated on a daily basis. With plenty of enrichment, connections and aggregations: we have done a lot of in-house research on disambiguation and entity-linking. We particularly have focused on mapping out the expertise, interests and achievements of all professional scientists, as well as the institutions they have worked at.

icapybara 3y ago
How do you get the chatbot to tell the truth more? I’m a scientist and this is my biggest showstopper with AI tools. I just can’t know if I can trust what it’s telling me.
- unblough 3y ago
  You are unable to. “LLMs can’t self-correct in reasoning tasks, DeepMind study finds“ https://news.ycombinator.com/item?id=37823543 https://news.ycombinator.com/item?id=37823543 Anyone who says otherwise is either ignorant of the underlying function of llms or trying to sell you something.
  - mistermann 3y ago

hashtag-til 3y ago

I think using any AI tool that reports your findings for you represents failure and laziness in science. It will become a circle: somebody uses AI to generate boring text (probably not even themselves will read it), then somebody else puts the boring text for AI to summarise. Nobody learns anything. Repeat.

lynx23 3y ago

This reminds me of M-x write-thesis RET, a usenet joke from the 90s IIRC. Apparently, since google is no longer indexing the past, I couldn't find a link. However, I still remember the post vividly, and it makes me sad that 25 years later, we have to have this discussion for real now. Back then, everyone in their right mind knew why automated scientific publication writing would be hilariously bad. These days, we have articles like this. Weird.

otabdeveloper4 3y ago

SaaS Sokal-as-a-Service.

streamfunk191 3y ago

It’s a shame the article doesn’t talk more about the possible benefits of AI in academia. The ability for researchers to spot new patterns, discover new areas of research. The way information is generated, disseminated and then read by other researchers in this space is fairly prosaic if you think about it. Causally is a company I think is trying something in this space

izacus 3y ago
That's because it's not how it's used right now. Just like elsewhere, it's primarly used to generate bullsh*t and flood journals with low quality generated content. Which now need to deal with this deluge of sewage drowning good articles.
- hashtag-til 3y ago
  Yep, I know quite a few gaming the system just like this, selling the AI dream...
aleph_minus_one 3y ago
> It’s a shame the article doesn’t talk more about the possible benefits of AI in academia. The ability for researchers to spot new patterns, discover new areas of research. Honestly, which publicly available AIs are capable of doing that? At least not the publicly available LLM AIs: I asked those quite many questions about scientific topics, but they could not give me any answer that helped me - it was a complete waste of time.

dist-epoch 3y ago

In the spirit of open, reproducible science, I think the prompts should appear in the final papers. Something like: [Summarize the prior research in paper XXX.YYY focusing on the DNA synthesis section] Bla BLA BLA [Describe in two paragraphs how DNA synthesis works] BLA BLA BLA

Moldoteck 3y ago

imo scientific publishing was already ruined by lack of reproducibility for a lot of papers, even for software/machine learning-focused ones

hoseja 3y ago

Elsevier and other corporate tools could disrupt scientific publishing

lapama 3y ago

Today I wrote a budget description for a scientific proposal that an AI would not be able to match. The reason? My knowledge of the social context.