12 ms·
Use of ChatGPT generated text for posts on Stack Overflow is temporarily banned
- mudrockbestgirl 4y agoAre there some examples of the answers by ChatGPT? I'd really like to look at them.
- rogerb 4y agoprompt: write a short essay as a scientist which is better, celsius or fahrentheit partial answer which some faulty reasoning about precision: First, the Celsius scale is based on a more intuitive and logical reference point. The Celsius scale sets the freezing point of water at 0 degrees and the boiling point of water at 100 degrees, whereas the Fahrenheit scale sets the freezing point of water at 32 degrees and the boiling point at 212 degrees. This means that the Celsius scale has a smaller unit interval, which allows for more precise temperature measurement.
- samcheng 4y agoPerfect. This kind of junk is going to litter our collective knowledge bases for years. It looks just good enough to be authoritative, but is fundamentally incorrect.
- saagarjha 4y agoSo a Hacker News comment.
- Ygg2 4y agoOr a Stack Overflow answer. Or any post on Internet.
- tonis2 4y agoYep, they trained the AI on hackernews
- DemocracyFTW2 4y agoOh no, not at all. In fact, hackernews got trained on the AI output of the Previous Turn.
- kasabali 4y agoThat's basically Reddit
- fragmede 4y agoHere's one: https://stackoverflow.com/a/74675088/19877232 https://stackoverflow.com/a/74675088/19877232
- insanitybit 4y agoI don't know react. Is this correct? It looks correct. And a good example of how to use ChatGPT - a human asks for the answer and then augments it with their expertise.
- freeone3000 4y agoThe post itself contains a correction, that the example code is out-of-date and incorrect. The human answer is valid, but would be valid with or without the assistance. The assistance requires constant supervision by a subject-matter expert.
- m0x 4y agohave they released any information regarding the number of recent submissions that were correct versus incorrect, or even how they are detecting ChatGPT?
- colesantiago 4y agoStackOverflow and Google are finished. This is expected and it may not look like it now, but banning an advancement in technology (even temporary) is usually the first step to irrelevance of those pushing back. In a few years from now when ChatGPT and perhaps the much awaited GPT-4 are released, get better, (or even an implementation is open sourced) it will further plummet the usage for SO and Google. The genie is out of the bottle and there is no going back, I'll give it less than 15 years.
- mrg3_2013 4y agowell, nothing stops google from baking this into their search. But it feels harsh to claim SO is finished. SO is human curated and has an edge (today). And for all we know, chatGPT learnt from SO and can synthesize results that are almost good.
- BoorishBears 4y agoIt feels especially harsh when SO is having to ban the answers it generates for being wrong...
- N3cr0ph4g1st 4y agoGoogle is most likely already working on its own version. They have the resources and ML chops to probably build one far more powerful. I'd bet they'll stay dominant
- dzink 4y agoAny tool that doesn’t force you to click on many page views to get an answer is bad for Google search. They want you to peruse and click on sites riddled with google ads as many times as possible. These incentives have led to quality of results getting worse over time. The only way Google is launching this type of model would be if they could embed ads in the answers and even potentially bake them in. Which would also make showing as many iterations as possible more lucrative, and thus disincentivize good performance. The problem with Ad platforms is that good user experience is a bug for the business model.
- vxNsr 4y agoDo they have a plan for identifying chatGPT posts aside from “looks like it”?
- liuliu 4y agoSince it is a centralized service provided by OpenAI, they should really provide services (paid or not) that given a text, check if it is generated by ChatGPT recently.
- mormegil 4y agoWhile a great idea, how could this work so that it would be tolerant to publishing just excerpts of the full output or to ignore trivial edits (either way, it could be fooled easily, I guess)?
- TeMPOraL 4y agoAdapt some kind of "perceptual hashing" that's used for audio and video fingerprinting - one that works when dealing with partial content.
- insanitybit 4y agoI assume it's just for cases of people saying "ChatGPT produced this output", not guessing.
- insanitybit 4y ago> Overall, because the average rate of getting correct answers from ChatGPT is too low, the posting of answers created by ChatGPT is substantially harmful to the site and to users who are asking or looking for correct answers. Would like to learn more about this. Tons of SO posts are wrong, it's almost always the case that you have to scroll past the accepted answer to find one that's actually right. Is ChatGPT much worse?
- nextlevelwizard 4y agoEven if it is the same why should it be allowed? We want better answers not same shit or something even marginally worse
- insanitybit 4y agoI'm asking what makes it worse.
- 9wzYQbTYsAIc 4y agoFrom the post: “The volume of these answers (thousands) and the fact that the answers often require a detailed read by someone with at least some subject matter expertise in order to determine that the answer is actually bad has effectively swamped our volunteer-based quality curation infrastructure.”
- insanitybit 4y agoChrist, I'm asking for more information. As I said, SO has tons of terrible and incorrect answers, so I want to know more about the impact of ChatGPT answers. Information like the rate at which they're correct vs incorrect, the rate at which they are submitted, and how that compares to normal answers.
- kop316 4y agoThe way I read it, ChatGPT acts as a force multiplier: https://en.wikipedia.org/wiki/Force_multiplication https://en.wikipedia.org/wiki/Force_multiplication Yes, there are terrible and incorrect answers, but adding ChatGPT allows this to happen at a much higher rate than previously. And it allows folks to put together coherent or correct sounding answers without knowing anything about the subject.
- esperent 4y agoThis seems reasonable to me and I expect this will become a permanent, although hard to enforce, rule on most forums. StackOverflow is supposed to be a place to ask programming questions and receive help from your human peers. It's not a place to ask questions and receive help from an AI. That would be a different product. It might in the future even be a better product, who knows? But it is a different product and there needs to be a clear line between them.
- numbsafari 4y ago> That would be a different product. That product is called "Google Cloud Support".
- FooBarWidget 4y agoIt wouldn't surprise me at all if ChatGPT was trained on data originating from Stack Overflow. Not familiar with deep learning algorithms but I can't imagine that it would be a good idea to have an unintended training data loop.
- msla 4y agoAn intended double-loop might be interesting: Train one GPT on the output of another.
- fragmede 4y agoOpenAI isn't giving out this service from the goodness of their hearts just for the fun of it. I bet they've gotten it to a place where they can feed its output back into itself for self improvement.
- maxbond 4y agoI dunno, is that really possible/sustainable? I suppose we add information to the system by poking it with prompts, and to some extent that gets encoded in the outputs. But I feel like if you're feeding it things that don't contain a new, fresh signal, you're ultimately going to fit to noise; the more of it's own input it ingests, the less meaningful the network would be, and presumably the more nonsensical the output. But I don't actually know how this stuff works, I'm genuinely asking. I agree OpenAI have something going on here, I'd assumed it was that they wanted to know the strategies we came up with for bypassing their filters, and to get into a cat and mouse game with us so that they could improve them to the point of being productized. Also, marketing, both sales, recruitment, and mindshare. ETA: I think I underestimated before the extent to which the output is often just your own words reflected back to you, in which case, it may well be adding more signal then noise. https://news.ycombinator.com/item?id=33863413 https://news.ycombinator.com/item?id=33863413 Still, some prompts return a walk of text from a single sentence, and I have to imagine those are worse then useless for training.
- 4y ago
- arthurofbabylon 4y agoWhat is the mechanism by which a text is determined to originate from ChatGPT? This will be crucial for preventing unfavorable training loops among a myriad of other human-led control efforts.
- autotune 4y agoTo determine if a post on StackOverflow is generated by ChatGPT, you could look for certain characteristics that are typical of posts generated by the model. For example, ChatGPT posts may contain repetitive or nonsensical phrases, lack coherence or context, or have a distinctive "machine-like" writing style. Additionally, you can check the user profile of the poster to see if it indicates that the user is a language model, such as ChatGPT. However, keep in mind that not all posts generated by ChatGPT will necessarily exhibit these characteristics, and some human-generated posts may also exhibit similar traits. Ultimately, the best way to determine if a post is generated by ChatGPT is to look for signs of artificiality in the content and context of the post. - obviously generated by ChatGPT
- gremlinsinc 4y agoThis has become a meme, at this point. - my opinion, not generated by ChatGPT.
- autotune 4y agoLet's hope it stays as absurd as a meme and not something that any of us actually have to worry about. - same as you
- spaceman_2020 4y agoThis needs to be a copypasta.
- archon1410 4y agookay, I did not realise that was GPT until the second last line, GPT seems to have a tendency to "hedge" all it's opinion in a certain way—it very rarely says one thing without adding a "however", "some people" etc.
- lovecg 4y agoI think the ban on ChatGPT answers on Stack Overflow is a mistake. Sure, some of the answers might not be correct, but that's true for any tool. And the fact that ChatGPT answers are easy to produce means that more people can contribute, which can only be a good thing for the community. Plus, it's not like the moderators have to accept every answer - they can still weed out the bad ones. Let's not punish everyone because of a few bad apples. — ChatGPT arguing its own case
- NetOpWibby 4y agoCompelling argument
- ignoramous 4y agoChatGPT misses the point that it mostly trains and hallucinates on info exclusively produced by humans. When sites like SO, Reddit, HN, Wikipedia begin to be dominated by GPT-3, it is then the humanity commences its ill-advised journey into the Matrix. As long as Chicken tastes good, I guess there will be few complaints (worry about the futility of staunch vegans, though). — Human who watches SciFi movies
- hgomersall 4y agoIt tastes quite like chicken initially, but lacks any sauce, or variability. You soon crave a piece that has some novel texture.
- the_gipsy 4y agoHave you ever tasted a wild chicken?
- hgomersall 4y agoIt's not like that.
- joe_the_user 4y agoGiven the faulty argument of GPT trained on human speech, I assume that GPT trained on GPT will be distinctly incoherent but somehow persuasive. But by that point, the GPT administering the sites won't be troubled by the situation.
- anon291 4y agoWell I asked chat gpt about a small Haskell library I maintain, and it correctly gave answers with code samples. I input several emails I've received in private and it correctly answered those. This is not a huge library ecosystem or even a large language . It then translated several Haskell code samples I have into c++. I'm not sure if I'm in awe or just in shock to be honest.
- fabian2k 4y agoI tried ChatGPT with a few SO questions as input, it was completely missing the point of the questions more often than not. I got 3 bad answers and one correct one out of it. The scary part was just how confidently incorrect it was. The text looked very good, but there were big errors in there. I also tried to ask it a scientific question on a pretty niche field, and it produced a very reasonably looking answer. Most of the text was actually correct, if a bit generic. But there was one big factual mistake in there, it gave a value range that is simply wrong for this question.
- CGamesPlay 4y agoScientific questions are, without being disparaging, mostly trivia. ChatGPT knows all of Wikipedia. If an answer could be found by extracting a passage from a Wikipedia article, ChatGPT will do a good job of answering it. Ask ChatGPT to do any engineering work, and it falls flat. It's poor mathematical skills and lack of ability to create complex plans mean that it is relying purely on it's coherent-sounding dialog.
- mach1ne 4y agoSlight correction: ChatGPT will PROBABLY do a good job of answering factual questions. It still sometimes hallucinates even when asking known facts, thus it inherently can't be trusted.
- TillE 4y agoYeah, but this is a problem where I can think of plausible solutions without hand-waving radical AI advancements. For a factual claim, it could look at each sentence and see if the equivalent information exists on Wikipedia, or another online source, or even a book. And then link to that source. Google has already digested all that information, so it's just a matter of finding clever ways to connect them. I don't think it would ever be flawless, but it could be turned into an extremely capable research assistant.
- krainboltgreene 4y agoYou are perfect for HN.
- roomey 4y agoJust finished Neil Stephenson's Anethem. In it (I'm paraphrasing here) the open internet is destroyed by media companies who flood the internet with "good garbage", that is stuff that is _almost_ correct. It forced people to pay for curated accurate content (that is, pay the same people who generated the good garbage in the first place).
- amanzi 4y agoThis is the scary thing about this - I keep asking it questions, and it gives confident, lengthy responses that look perfect. But when you read it thoroughly, you start noticing small but important mistakes that are not obvious at first.
- steve_adams_86 4y agoI tried to get it to dig into some ethical questions and it was disturbing to see how cogent the argument was while mostly overlooking the ethical concerns of the question. I know it’s not actually meant to do that in the first place, but it’s quite excellent at it despite that shortcoming. I’m not looking forward to AIs learning to become even more convincing and authoritative.
- crabmusket 4y agoIt's like the ultimate solipsistic debate-club point of view, distilled and targeted in whatever direction you choose.
- wnkrshm 4y agoWhy have this mountain of noise? It'll end in: a) Wow such a nice tool that generates nice prose for emails and communication! b) Wow, such a nice tool that filters and summarizes the mountains of prose coming into my communication channels! It's as if we're just making the channel broader with less information for no reason.
- TeMPOraL 4y ago
- Markoff 4y agoI tried to create openai account, but seems e-mail is not enough for them and they require also phone number, not sure why, so I gave up, I provide my phone number only where it's essential.
- amanzi 4y agoThis might not be essential, but you're missing out on playing with some amazing technology.
- yesenadam 4y agoThey send an SMS code to sign up with.
- yjftsjthsd-h 4y agoThat sounds like a mechanism for verifying the number, not a reason.
- maxbond 4y agoCharitably, so that it's easier to ban people who turn out to be spammers, and more expensive for them to make another account.
- charcircuit 4y ago>not sure why It is to reduce abuse of their API because these requests are very expensive to serve as it requires 100s of gigabytes of VRAM to serve a request, requests take a few seconds at a time, and they are handling a ton of free requests. If you were paying for their API you could be paying up to about $0.10 per text generation.
- Markoff 4y agoI wish HN would be showing who downvotes the post, I am more and more experiecing brigading even on completely innocent posts like sharing my experience with account registration.
- johnchristopher 4y agoAnd (at the moment) the first answer is a ChatGPT answer. https://meta.stackoverflow.com/a/421832 https://meta.stackoverflow.com/a/421832 I guess that cat won't go back in its bottle [0]. [0] Attempt at humor was not AI generated. [1] [1] And neither is this footnote.
- b800h 4y agoYet another example of AI driving us back to analogue. There's no obvious way of identifying AI content on StackOverflow, and there's a financial imperative to having a high score on there. So one would expect it to get gamed into oblivion with bots. Once that's happened, there ceases to be any social payoff in posting tech info on the net. And so we end of hoarding knowledge and training people in person.
- sgu999 4y agoInteresting thought. But what makes you think that it won't be good enough for the majority of people? To some extent, online-only bootcamps are already good enough for a large crowd of devs, while others went to universities and labs to get a much more in depth education.
- theamk 4y agoPeople remember bad experiences much more than good ones, so if half of the answers on website look reasonable but do not work, the overall impression would be "website with garbage answers" And it seems that AI has much more than 50% answers wrong.
- ZephyrBlu 4y agoEven 20% wrong answers probably = "website with garbage answers".
- pydave 4y agoWhat's the financial imperative to having a high score on SO? Better job offers because you have high rep?
- hooby 4y agoI think it could work - but you'd have to find a way to let ChatGPT exclusively deal with the sort of repeat questions that are rather annoying to human posters - while all the actually interesting, new questions (which are also more likely to cause ChatGPT to create a flawed answer), are reserved for humans. (Also don't let ChatGPT pose as a human - give it it's own user name and automate it.)
- charcircuit 4y agoRepeat questions should not be answered. They get marked as duplicate of the original question.
- afdbcreid 4y agoIf only ChatGPT could close as a duplicate... But there is an answer suggesting using it: https://meta.stackoverflow.com/a/421836/7884305 https://meta.stackoverflow.com/a/421836/7884305.
- DarthNebo 4y agoA solution should have a parallel answers thread, system-gen vs bio-gen
- booleandilemma 4y agoGood. I'm wondering when HN will follow suit. Begun the AI war has.
- Traubenfuchs 4y agoI actually got emailed personally by Dan last weekend to stop AI-posting after doing this for more than a year.
- deleted 4y ago[deleted]
- swader999 4y agoThis has the potential to break every message board out there. No offense, but I'll be outta here if it's mostly robots to talk to.
- l0b0 4y agoIf it's cheap enough, it'll be just like email spam - basically everywhere worth influencing in any way will be flooded. And it'll be much harder to filter than email. One possible solution would be to change basically any community-driven site to be invite-only, and to wield the ban hammer ruthlessly.
- ilaksh 4y agoIts definitely a problem if people are not testing the answers. There often are bugs or nonsense. But it also routinely provides very useful answers, and you can ask it to fix bugs you find. I assume that within the next two years, Stack Overflow will be overtaken by a similar site that defaults to Chat-GPT-like AI answers, and has incentives for human programmers to train it by checking or improving its answers. Unless the Stack Overflow people decide to do something like that themselves. But even this version of ChatGPT often has amazing code answers, and if you could put it in a loop with a compiler/JavaScript runtime that could go pretty far. I imagine a couple more years of larger/better models (maybe with some visual understanding of UIs) etc., and the website will just be "write, test and deploy this entire program for me, here is a list of requirements" -- taking out a lot of not only Stack Overflow's business but also Upwork etc.
- nedt 4y agoFinally the "Not a Robot" badge for us ~1000 users pays off. https://stackoverflow.com/help/badges/6381/not-a-robot https://stackoverflow.com/help/badges/6381/not-a-robot Yeah I know, humans will team up with bots and praise the new chat overlords.
- floor_ 4y agoThey're basically banning it's direct competitor. Makes business sense.
- hrpnk 4y agoWould be nice if SO added the ChatGPT response by default that's hidden by default and prevent others from adding sufficiently similar responses.
- hackerman123469 4y agoMight as well ban normal posts and responses then. A lot of them are wrong too. The majority of the time you get wrong or misleading information, but sometimes that is enough for you to solve the issue anyway.
- raji_sankar 4y agoI will definitely say that ChatGPT comes up with really good combination of words that can coherently stick together. But, it should be understood that "we" make sense from the sentence it generates, not the Chat software itself, it is just stringing together words based on the patterns it has learnt. If answering a question was all about generating the correct subsequent/pattern of words based on prior context or prompts, then we can argue that sales consultant, marketing personnel, journalists etc can write and design software or come up with scientific theories? After all, they also can use buzz words they have learnt contextually very well and hence they sell the product, isn't it?