6 ms·

Moderation Strike: Results of Negotiations

70 points by thyrsus 3y ago

roenxi 3y ago
What are the circumstances where StackOverflow is still competitive against ChatGPT? As far as I can tell SO.com and google are both obsolete for getting answers to programming questions. I haven't used either in a while and as locally hosted LLMs get more pervasive I don't see the situation tilting back towards those services.
- dwheeler 3y ago
  I think it's very likely that StackExchange is the key data source for ChatGPT answers of this kind. If you want LLMs to give good answers, it's important to have their sources correct.
- viraptor 3y ago
  Time passing is the circumstance. ChatGPT knows the answers from SO, but for new languages the quality will degrade over time. ChatGPT already isn't great at answering questions about new version of Swift for example. It won't be able to use new languages / APIs either. Given how rarely the model is updated (officially anyway), it will always lag behind the new tech. It may be actually in the interest of OpenAI to start paying SO for the data and make sure it flows to the community in some way. Otherwise they will be starved.
- layer8 3y ago
  This is just one aspect, but some people strongly prefer typing only a few search terms and then exploring links over holding a linear synchronous conversation with a chatbot. It’s a significantly different usage mode. It’s a bit like browsing through a book vs. interviewing a random person who read the book.
- treis 3y ago
  A SO post usually has an answer that works and is a reasonably good idea. ChatGPT lacks the ability to see if its responses pass even a basic sanity test. Sometimes it's answers are good and sometimes they are totally wrong. Hard for a person asking the question to determine which answer they got.
  - rolisz 3y ago
    With code interpreter, ChatGPT can check that it's code at least runs without errors
    - dharmab 3y ago
      rm -rf $INSTALL_DIR/* this line of code has at least two gotchas. if you run it to test it out, there is a good chance you will destroy your data. It is otherwise a valid line of code. This is not a theoretical problem: https://github.com/valvesoftware/steam-for-linux/issues/3671 https://github.com/valvesoftware/steam-for-linux/issues/3671
      - rolisz 3y ago
        ChatGPT warns about gotchas with that line.
        danShumway 3y ago
        The point being that: > With code interpreter, ChatGPT can check that it's code at least runs without errors is not a solution to gotchas/caveats that build up over time that ChatGPT won't know about. ChatGPT can warn about caveats from that line of code because it's old code. It does not have a magical defense that will allow it to warn about caveats, errors, security risks, or deprecations that are introduced after its knowledge cutoff. The best solution that we've been able to find so far is for ChatGPT to do a search and feed the results into its context. But that has its own problems, and importantly it only works if the underlying search engines are still returning up-to-date relevant results and documentation from sites like StackOverflow.
    - treis 3y ago
      Yes, but ultimately it's still a "statistically likely" answer and not the "right" one. Often the two overlap and maybe one day they'll make it an overlapping venn diagram. But that day is not today.
- wodenokoto 3y ago
  If SO can’t compete on hard problems then we should all change job titles to prompt engineers.
  - twelve40 3y ago
    That's a strange claim. SO's short q-and-a format doesn't even begin to tackle hard problems.
- remram 3y ago
  We can't keep training ChatGPT on SO and then pretend that SO is made obsolete by ChatGPT. That makes no sense.
- dpkirchner 3y ago
  IME ChatGPT provides more useful results than SO, especially because ChatGPT doesn't try to guess what I'm asking and takes my request at face value (by its very nature). If ChatGPT struggles with a particular topic, companies like OpenAI could hire some subject matter experts (SMEs) to write material that could be used for future training. I don't see how SO can survive much longer. Two things SO could do to beat ChatGPT as it exists now: 1) Require all questions and answers to include the names of tools and their version numbers. Hide Qs and As until those are added. This is so critical, I have no idea why it wasn't required since day 1. 2) Discourage answers that don't answer the question, through moderation. If you don't know the answer, don't post, or at least leave your notes as a comment. Bonus: Stop removing chats. The number of times we see someone link to a chat that 404s is too damned high. Cf: complaints from folks about the unsearchability of Discord chats wrt software tools.
  - em-bee 3y ago
    i never understood why new users are allowed to post answers but not comments. it doesn't make sense. i mean it makes sense to restrict what new users can do, but the current mode results in new users posting comments as answers, and it irritates me every time i see one. on the other hand maybe it avoids answers as comments. but in the end maybe it should just be easier to convert between the two. i like your idea of including versions.
treis 3y ago
If five years ago I would have read this as a parody I would have dismissed it as unbelievable. Unpaid volunteers go on strike and the resulting agreement is a bureaucratic masterpiece of procedures that (probably) don't solve the problem and create working groups to establish more of them. What a world we live in.
- deleted 3y ago
  [deleted]
- pessimizer 3y ago
  A world where intentional communities of peers deliberate with each other to iron out past problems and agree on new procedures going forward? It's always easier to have a dictator come in and tell people what to do, but people complain about it.
- tmpX7dMeXU 3y ago
  Unpaid volunteers just want stuff to be involved in. The weird notion that unpaid volunteers are some pure force for good is falsely perpetuated by internet greybeards who only say it because they themselves were those people. Ask any young person their opinion of people in these positions and they’ll rightfully paint them in a negative light. Somehow we took being on body corporate and fetishised it.
  - bombcar 3y ago
    When the jannies are unpaid you have to wonder why they’re doing it. They will always say it’s from love for the community, but that can often be suspect. (We known dang moderates us out of the goodness of his paycheck, which means he’s free to love us all :)
  - em-bee 3y ago
    The weird notion that unpaid volunteers are some pure force for good is falsely perpetuated by internet greybeards who only say it because they themselves were those people that is an interesting argument, but i don't understand how it is relevant here. as far as i can tell the moderators were acting with good intentions as a force for good as it were. if anything the problem was not that but that the company seemed to think that they can dictate what volunteers do. but really, they can't. reddit is doing the same, and if they don't listen to the volunteers they will eventually pay the price.
Piskvorrr 3y ago
"particularly about the lack of communication ahead of time" Yup. SE Inc. gonna SE Inc. Not coming back this time either.
jefftk 3y ago
This looks really good; thanks Cody, Makyen, and Mithical, and thanks SE for engaging!
comex 3y ago
Note: Judging from the timestamps, this seems to be about three weeks old.
- gumshoegarry 3y ago
  And, judging by the words, it seems to be written in English. Very mysterious.
  - natpalmer1776 3y ago
    Got a good chuckle from this, thank you. Judging by how your comment appears to already have been flagged by other users though, you may want to include your actual point after the punchline.
  - stevehawk 3y ago
    lol this was made insanely better by your handle
next_xibalba 3y ago
My faith in the bureaucracy that maintains SE was irrevocably destroyed when Monica Cellio was terminated. An act of complete insanity and cowardice will do that.
- deleted 3y ago
  [deleted]

70 points by thyrsus 3y ago

What are the circumstances where StackOverflow is still competitive against ChatGPT? As far as I can tell SO.com and google are both obsolete for getting answers to programming questions. I haven't used either in a while and as locally hosted LLMs get more pervasive I don't see the situation tilting back towards those services.

dwheeler 3y ago
I think it's very likely that StackExchange is the key data source for ChatGPT answers of this kind. If you want LLMs to give good answers, it's important to have their sources correct.
viraptor 3y ago
Time passing is the circumstance. ChatGPT knows the answers from SO, but for new languages the quality will degrade over time. ChatGPT already isn't great at answering questions about new version of Swift for example. It won't be able to use new languages / APIs either. Given how rarely the model is updated (officially anyway), it will always lag behind the new tech. It may be actually in the interest of OpenAI to start paying SO for the data and make sure it flows to the community in some way. Otherwise they will be starved.
layer8 3y ago
This is just one aspect, but some people strongly prefer typing only a few search terms and then exploring links over holding a linear synchronous conversation with a chatbot. It’s a significantly different usage mode. It’s a bit like browsing through a book vs. interviewing a random person who read the book.
treis 3y ago
A SO post usually has an answer that works and is a reasonably good idea. ChatGPT lacks the ability to see if its responses pass even a basic sanity test. Sometimes it's answers are good and sometimes they are totally wrong. Hard for a person asking the question to determine which answer they got.
- rolisz 3y ago
  With code interpreter, ChatGPT can check that it's code at least runs without errors
  - dharmab 3y ago
    rm -rf $INSTALL_DIR/* this line of code has at least two gotchas. if you run it to test it out, there is a good chance you will destroy your data. It is otherwise a valid line of code. This is not a theoretical problem: https://github.com/valvesoftware/steam-for-linux/issues/3671 https://github.com/valvesoftware/steam-for-linux/issues/3671
    - rolisz 3y ago
      ChatGPT warns about gotchas with that line.
      - danShumway 3y ago
        The point being that: > With code interpreter, ChatGPT can check that it's code at least runs without errors is not a solution to gotchas/caveats that build up over time that ChatGPT won't know about. ChatGPT can warn about caveats from that line of code because it's old code. It does not have a magical defense that will allow it to warn about caveats, errors, security risks, or deprecations that are introduced after its knowledge cutoff. The best solution that we've been able to find so far is for ChatGPT to do a search and feed the results into its context. But that has its own problems, and importantly it only works if the underlying search engines are still returning up-to-date relevant results and documentation from sites like StackOverflow.
  - treis 3y ago
    Yes, but ultimately it's still a "statistically likely" answer and not the "right" one. Often the two overlap and maybe one day they'll make it an overlapping venn diagram. But that day is not today.
wodenokoto 3y ago
If SO can’t compete on hard problems then we should all change job titles to prompt engineers.
- twelve40 3y ago
  That's a strange claim. SO's short q-and-a format doesn't even begin to tackle hard problems.
remram 3y ago
We can't keep training ChatGPT on SO and then pretend that SO is made obsolete by ChatGPT. That makes no sense.
dpkirchner 3y ago
IME ChatGPT provides more useful results than SO, especially because ChatGPT doesn't try to guess what I'm asking and takes my request at face value (by its very nature). If ChatGPT struggles with a particular topic, companies like OpenAI could hire some subject matter experts (SMEs) to write material that could be used for future training. I don't see how SO can survive much longer. Two things SO could do to beat ChatGPT as it exists now: 1) Require all questions and answers to include the names of tools and their version numbers. Hide Qs and As until those are added. This is so critical, I have no idea why it wasn't required since day 1. 2) Discourage answers that don't answer the question, through moderation. If you don't know the answer, don't post, or at least leave your notes as a comment. Bonus: Stop removing chats. The number of times we see someone link to a chat that 404s is too damned high. Cf: complaints from folks about the unsearchability of Discord chats wrt software tools.
- em-bee 3y ago
  i never understood why new users are allowed to post answers but not comments. it doesn't make sense. i mean it makes sense to restrict what new users can do, but the current mode results in new users posting comments as answers, and it irritates me every time i see one. on the other hand maybe it avoids answers as comments. but in the end maybe it should just be easier to convert between the two. i like your idea of including versions.

treis 3y ago

If five years ago I would have read this as a parody I would have dismissed it as unbelievable. Unpaid volunteers go on strike and the resulting agreement is a bureaucratic masterpiece of procedures that (probably) don't solve the problem and create working groups to establish more of them. What a world we live in.

deleted 3y ago
[deleted]
pessimizer 3y ago
A world where intentional communities of peers deliberate with each other to iron out past problems and agree on new procedures going forward? It's always easier to have a dictator come in and tell people what to do, but people complain about it.
tmpX7dMeXU 3y ago
Unpaid volunteers just want stuff to be involved in. The weird notion that unpaid volunteers are some pure force for good is falsely perpetuated by internet greybeards who only say it because they themselves were those people. Ask any young person their opinion of people in these positions and they’ll rightfully paint them in a negative light. Somehow we took being on body corporate and fetishised it.
- bombcar 3y ago
  When the jannies are unpaid you have to wonder why they’re doing it. They will always say it’s from love for the community, but that can often be suspect. (We known dang moderates us out of the goodness of his paycheck, which means he’s free to love us all :)
- em-bee 3y ago
  The weird notion that unpaid volunteers are some pure force for good is falsely perpetuated by internet greybeards who only say it because they themselves were those people that is an interesting argument, but i don't understand how it is relevant here. as far as i can tell the moderators were acting with good intentions as a force for good as it were. if anything the problem was not that but that the company seemed to think that they can dictate what volunteers do. but really, they can't. reddit is doing the same, and if they don't listen to the volunteers they will eventually pay the price.

Piskvorrr 3y ago

"particularly about the lack of communication ahead of time" Yup. SE Inc. gonna SE Inc. Not coming back this time either.

jefftk 3y ago

This looks really good; thanks Cody, Makyen, and Mithical, and thanks SE for engaging!

comex 3y ago

Note: Judging from the timestamps, this seems to be about three weeks old.

gumshoegarry 3y ago
And, judging by the words, it seems to be written in English. Very mysterious.
- natpalmer1776 3y ago
  Got a good chuckle from this, thank you. Judging by how your comment appears to already have been flagged by other users though, you may want to include your actual point after the punchline.
- stevehawk 3y ago
  lol this was made insanely better by your handle

next_xibalba 3y ago

My faith in the bureaucracy that maintains SE was irrevocably destroyed when Monica Cellio was terminated. An act of complete insanity and cowardice will do that.

deleted 3y ago
[deleted]