3 ms·
It's flat wrong to suggest SO had the right answer all the time, and in fact in my experience for trickier work it was often wrong or missing entirely. LLMs ha
by ianbutler 10mo ago
It's flat wrong to suggest SO had the right answer all the time, and in fact in my experience for trickier work it was often wrong or missing entirely.
LLMs have a better hit rate with me.
- danver0 10mo ago[flagged]
- johnsmith1840 10mo agoI mean when I was pure SO debugging I would say a solid 70% of the responses did not work. Different stack different bug ect. LLMs even if they can't guess the right answer can search the documentation really well.
- ianbutler 10mo ago[flagged]
- GaryBluto 10mo agoI'm not sure he intended it as an insult. Despite not using LLMs for HTML or JavaScript often (if at all), I was under the impression that it was one of their strong areas.
- ianbutler 10mo agoIn my experience, at least in circles I've been around, I have found other developers who are a little too high on their own supply so to speak use "hurr durr web dev" as the "easy not real dev" insult. So I both wanted to shame that in case they were and also categorically show otherwise.
- paganholiday 10mo agoThe example wasn't even finding a right answer so I don't see where you got that.. Searching questions/answers on SO can surface correct paths on situations where the LLMs will keep giving you variants of a few wrong solutions, kind of like the toxic duplicate closers.. Ironically, if SO pruned the history to remove all failures to match its community standards then it would have the same problem.
- ianbutler 10mo ago"But losing SO means that we're getting an idiot friendly guy with a lot of credible but wrong answers in place of a grumpy and possibly toxic guy which, however, actually answered our questions." > "actually answered our questions." Read carefully.
- paganholiday 10mo ago>> Eventually I tried with something else, and found a question on stackoverflow, luckily with an answer. That was the game changer and eventually I was able to find the right doc Read carefully and paraphrase to the generous side. The metaphor that follows that is obviously trying to give an example of what might be somehow lost.
- ianbutler 10mo agoThis is a fair critique. I am often not generous enough with people.
- akoboldfrying 10mo agoInterpreting that claim as "SO users always, 100% of the time answer questions correctly" is uncharitable to the point of being unreasonable. Most people would interpret the claim as concisely expressing that you get better accuracy from grumpy SO users than friendly LLMs.
- ianbutler 10mo agoFor the record I was interpreting that as LLMs are useless (which may have been just as uncharitable), which I categorically deny. I would say they're about just as useful without wading through the mire that SO was.
- johnnyanmac 10mo agoYes, it does answer you question, when the site lets it go through. Note that "answers your question" does not mean "solving your problem". Sometimes the answer to a question is "this is infeasible because XYZ" and that's good feedback to get to help you re-evaluate a problem. Many LLMs still struggle with this and would rather give a wrong answer than a negative one. That said, the "why don't you use X" response is practically a stereotype for a reason. So it's certainly not always useful feedback. If people could introspect and think "can 'because my job doesn't allow me to install Z' be a valid response to this", we'd be in a true Utopia.
- jtrn 10mo agoNo point in arguing with people who bring a snowball into Congress to disprove global warming.
- vultour 10mo agoIt entirely depends on the language you were using. The quality of both questions and answers between e.g. Go and JavaScript is incredible. Even as a relative beginner in JS I could not believe the amount of garbage that I came across, something that rarely happened for Go.