5 ms·

Stack Overflow bans ChatGPT as 'substantially harmful'

33 points by redbell 4y ago

2pEXgD0fZ5cF 4y ago
One of the scariest and most depressing aspects when it comes to ChatGPT (so far) is that people already try to use it in such a way. Same for people that are already talking about using it for medical diagnoses and such, without ever having seen a single proper independent and peer reviewed study when it comes to chatgpts accuracies in specific fields (and that is ignoring all the other potential short term and long term problems when it comes to even considering something like that).
- europeanguy 4y ago
  On the bright side, dumb people have always been around. It's not like it's getting worse, in fact I would bet it's getting better. The meta-phenomenon is not something new here.
- 2OEH8eoCRo0 4y ago
  That people ask it programming questions or they use it to poison and spam online communities for internet points?
- selfhoster11 4y ago
  I think the problem is that people cannot envision (or cannot believe, when told) that something that can speak human language would outright bullshit its way through every single conversation. And because GPTs and chatbots are built as very convincing bullshit generators, some percent of the time (majority of the time) they will say things that check out - and that provides enough fuel to start building false assumptions that everything else it says is true.
  - saurik 4y ago
    > I think the problem is that people cannot envision (or cannot believe, when told) that something that can speak human language would outright bullshit its way through every single conversation. Which is always extremely confusing to me because I know people who do this also ;P.
  - justinclift 4y ago
    > I think the problem is that people cannot envision (or cannot believe, when told) that something that can speak human language would outright bullshit its way through every single conversation. <cough> politicians <cough/>
    - frumper 4y ago
      ChatGPT 2024!
- deleted 4y ago
  [deleted]
Imnimo 4y ago
It's interesting that OpenAI slow-rolled the release of some of their other projects due to a worry that they'd cause harm in the wrong hands, but seems to have just let ChatGPT rip. I don't think "makes it easy to pollute StackOverflow" rises to the level of a serious enough harm to justify keeping the tech locked away, but it's interesting that we're at least starting to get a glimpse of actual harms.
- bioemerl 4y ago
  They didn't care about the harm, they just needed an excuse to close source all their models.
LandR 4y ago
ChatGPT probably makes stack overflow unecessesary, I can see why they don't like it.
- shadowofneptune 4y ago
  The article already states why this is such an issue: The results could be correct or incorrect (the model has no idea either way), but are delivered in the style of a well-researched answer. They can also be automated, so there's tons of these deceptive answers.
  - blacksmith_tb 4y ago
    Right, it sounds like the scale is a particular problem, you'd expect the feedback mechanisms to gradually weed out nonsense answers, but if 90% of all answers were nonsense, humans couldn't keep up with.
- ludamad 4y ago
  I have trialled live coding with it and while amazing it suddenly generates half outputs and gets stuck. Right no, no, its swamping their volunteers, not suddenly making their service better.
- StreamBright 4y ago
  I don't think that you have the data to back this up.
- Micoloth 4y ago
  I am big fan of Gpt and not a fan of restricting its usage, but i think this decision is completely correct. Considering that, stack overflow is definitely one of the source materials for Gpt training, and probably a very significant one for like Codex. If i ask on SO it’s because i want a human answer. What is the point of getting a Gpt one? If I wanted the Got opinion, I can __just ask gpt__ (and btw I already do it always, now). So the decision is very reasonable.. If anything, it’s sad that i cannot see any way in which it can actually be enforced
- 4y ago
2OEH8eoCRo0 4y ago
As opposed to the multitude of horrible or incorrect answers written by humans? I think anybody who uses this thing for a few minutes understands that it's not infallible.
- orphea 4y ago
  This thing allows to produce those horrible or incorrect answers order of magnitude easier.
laacz 4y ago
I posted two answers to two different threads just to play with it. One got accepted, another didnt. I validated both answers before posting them. Answers were posted well before the ban on AI generated answers was even considered. Nevertheless, Stackoverflow suspended my account for a week. Of course, it's only for a week, I'll survive and lesson learned, but at the time answer was posted, it did not violate any rules.
- deleted 4y ago
  [deleted]
- laacz 4y ago
  I have no idea.
koromak 4y ago
I've been messing around with it for a few days trying random interview style questions, and its almost always subtly wrong. Its still very cool but clearly not there yet.
Karawebnetwork 4y ago
How do they detect that an answer has been generated by ChatGPT?
- hxugufjfjf 4y ago
  https://huggingface.co/openai-detector https://huggingface.co/openai-detector
  - Karawebnetwork 4y ago
    Interesting, it seem to work most of the time using texts I've generated before. Any idea how it works? I can't seem to find a blog post.
    - hxugufjfjf 4y ago
      They link to both the code and the research paper on the site.
jmakov 4y ago
*to their business model.
PeterStuer 4y ago
I doubt this ban will succeed. Automation makes the production cost fall close to zero, while the detection cost remains relatibely high. More ardent reputation based vetting could occur, but that would also impact genuine new contribitors and cut of the inflow of those while further boosting an already bolstered activist gatekeeper contingent on the platform scaring of well intentioned fresh posters.

33 points by redbell 4y ago

One of the scariest and most depressing aspects when it comes to ChatGPT (so far) is that people already try to use it in such a way. Same for people that are already talking about using it for medical diagnoses and such, without ever having seen a single proper independent and peer reviewed study when it comes to chatgpts accuracies in specific fields (and that is ignoring all the other potential short term and long term problems when it comes to even considering something like that).

europeanguy 4y ago
On the bright side, dumb people have always been around. It's not like it's getting worse, in fact I would bet it's getting better. The meta-phenomenon is not something new here.
2OEH8eoCRo0 4y ago
That people ask it programming questions or they use it to poison and spam online communities for internet points?
selfhoster11 4y ago
I think the problem is that people cannot envision (or cannot believe, when told) that something that can speak human language would outright bullshit its way through every single conversation. And because GPTs and chatbots are built as very convincing bullshit generators, some percent of the time (majority of the time) they will say things that check out - and that provides enough fuel to start building false assumptions that everything else it says is true.
- saurik 4y ago
  > I think the problem is that people cannot envision (or cannot believe, when told) that something that can speak human language would outright bullshit its way through every single conversation. Which is always extremely confusing to me because I know people who do this also ;P.
- justinclift 4y ago
  > I think the problem is that people cannot envision (or cannot believe, when told) that something that can speak human language would outright bullshit its way through every single conversation. <cough> politicians <cough/>
  - frumper 4y ago
    ChatGPT 2024!
deleted 4y ago
[deleted]

Imnimo 4y ago

It's interesting that OpenAI slow-rolled the release of some of their other projects due to a worry that they'd cause harm in the wrong hands, but seems to have just let ChatGPT rip. I don't think "makes it easy to pollute StackOverflow" rises to the level of a serious enough harm to justify keeping the tech locked away, but it's interesting that we're at least starting to get a glimpse of actual harms.

bioemerl 4y ago
They didn't care about the harm, they just needed an excuse to close source all their models.

LandR 4y ago

ChatGPT probably makes stack overflow unecessesary, I can see why they don't like it.

shadowofneptune 4y ago
The article already states why this is such an issue: The results could be correct or incorrect (the model has no idea either way), but are delivered in the style of a well-researched answer. They can also be automated, so there's tons of these deceptive answers.
- blacksmith_tb 4y ago
  Right, it sounds like the scale is a particular problem, you'd expect the feedback mechanisms to gradually weed out nonsense answers, but if 90% of all answers were nonsense, humans couldn't keep up with.
ludamad 4y ago
I have trialled live coding with it and while amazing it suddenly generates half outputs and gets stuck. Right no, no, its swamping their volunteers, not suddenly making their service better.
StreamBright 4y ago
I don't think that you have the data to back this up.
Micoloth 4y ago
I am big fan of Gpt and not a fan of restricting its usage, but i think this decision is completely correct. Considering that, stack overflow is definitely one of the source materials for Gpt training, and probably a very significant one for like Codex. If i ask on SO it’s because i want a human answer. What is the point of getting a Gpt one? If I wanted the Got opinion, I can __just ask gpt__ (and btw I already do it always, now). So the decision is very reasonable.. If anything, it’s sad that i cannot see any way in which it can actually be enforced
4y ago

2OEH8eoCRo0 4y ago

As opposed to the multitude of horrible or incorrect answers written by humans? I think anybody who uses this thing for a few minutes understands that it's not infallible.

orphea 4y ago
This thing allows to produce those horrible or incorrect answers order of magnitude easier.

laacz 4y ago

I posted two answers to two different threads just to play with it. One got accepted, another didnt. I validated both answers before posting them. Answers were posted well before the ban on AI generated answers was even considered. Nevertheless, Stackoverflow suspended my account for a week. Of course, it's only for a week, I'll survive and lesson learned, but at the time answer was posted, it did not violate any rules.

deleted 4y ago
[deleted]
laacz 4y ago
I have no idea.

koromak 4y ago

I've been messing around with it for a few days trying random interview style questions, and its almost always subtly wrong. Its still very cool but clearly not there yet.

Karawebnetwork 4y ago

How do they detect that an answer has been generated by ChatGPT?

hxugufjfjf 4y ago
https://huggingface.co/openai-detector https://huggingface.co/openai-detector
- Karawebnetwork 4y ago
  Interesting, it seem to work most of the time using texts I've generated before. Any idea how it works? I can't seem to find a blog post.
  - hxugufjfjf 4y ago
    They link to both the code and the research paper on the site.

jmakov 4y ago

*to their business model.

PeterStuer 4y ago

I doubt this ban will succeed. Automation makes the production cost fall close to zero, while the detection cost remains relatibely high. More ardent reputation based vetting could occur, but that would also impact genuine new contribitors and cut of the inflow of those while further boosting an already bolstered activist gatekeeper contingent on the platform scaring of well intentioned fresh posters.