6 ms·

I spent the last 14 days chasing an issue with a Spark transform. Gemini and Claude were exceptionally good at giving me answers that looked perfectly reasonabl

I spent the last 14 days chasing an issue with a Spark transform. Gemini and Claude were exceptionally good at giving me answers that looked perfectly reasonable: none of them worked, they were almost always completely off-road. Eventually I tried with something else, and found a question on stackoverflow, luckily with an answer. That was the game changer and eventually I was able to find the right doc in the Spark (actually Iceberg) website that gave me the final fix. This is to say that LLMs might be more friendly. But losing SO means that we're getting an idiot friendly guy with a lot of credible but wrong answers in place of a grumpy and possibly toxic guy which, however, actually answered our questions. Not sure why someone is thinking this is a good thing.

ianbutler 9mo ago
It's flat wrong to suggest SO had the right answer all the time, and in fact in my experience for trickier work it was often wrong or missing entirely. LLMs have a better hit rate with me.
- danver0 9mo ago
  [flagged]
  - johnsmith1840 9mo ago
    I mean when I was pure SO debugging I would say a solid 70% of the responses did not work. Different stack different bug ect. LLMs even if they can't guess the right answer can search the documentation really well.
  - ianbutler 9mo ago
    [flagged]
    - GaryBluto 9mo ago
      I'm not sure he intended it as an insult. Despite not using LLMs for HTML or JavaScript often (if at all), I was under the impression that it was one of their strong areas.
      - ianbutler 9mo ago
        In my experience, at least in circles I've been around, I have found other developers who are a little too high on their own supply so to speak use "hurr durr web dev" as the "easy not real dev" insult. So I both wanted to shame that in case they were and also categorically show otherwise.
- paganholiday 9mo ago
  The example wasn't even finding a right answer so I don't see where you got that.. Searching questions/answers on SO can surface correct paths on situations where the LLMs will keep giving you variants of a few wrong solutions, kind of like the toxic duplicate closers.. Ironically, if SO pruned the history to remove all failures to match its community standards then it would have the same problem.
  - ianbutler 9mo ago
    "But losing SO means that we're getting an idiot friendly guy with a lot of credible but wrong answers in place of a grumpy and possibly toxic guy which, however, actually answered our questions." > "actually answered our questions." Read carefully.
    - paganholiday 9mo ago
      >> Eventually I tried with something else, and found a question on stackoverflow, luckily with an answer. That was the game changer and eventually I was able to find the right doc Read carefully and paraphrase to the generous side. The metaphor that follows that is obviously trying to give an example of what might be somehow lost.
      - ianbutler 9mo ago
        This is a fair critique. I am often not generous enough with people.
    - akoboldfrying 9mo ago
      Interpreting that claim as "SO users always, 100% of the time answer questions correctly" is uncharitable to the point of being unreasonable. Most people would interpret the claim as concisely expressing that you get better accuracy from grumpy SO users than friendly LLMs.
      - ianbutler 9mo ago
        For the record I was interpreting that as LLMs are useless (which may have been just as uncharitable), which I categorically deny. I would say they're about just as useful without wading through the mire that SO was.
- jtrn 9mo ago
  No point in arguing with people who bring a snowball into Congress to disprove global warming.
- vultour 9mo ago
  It entirely depends on the language you were using. The quality of both questions and answers between e.g. Go and JavaScript is incredible. Even as a relative beginner in JS I could not believe the amount of garbage that I came across, something that rarely happened for Go.
specproc 9mo ago
What I always appreciate about SO is the dialogue between commenters. LLMs give one answer, or bullet points around a theme, or just dump a load of code in your IDE. SO gives a debate, in which the finer points of an issue are thrashed out, with the best answers (by and large) floating to the top. SO, at its best, is numerous highly-experienced and intelligent humans trying to demonstrate how clever they are. A bit like HN, you learn from watching the back and forth. I don't think this is something that LLMs can ever replicate. They don't have the egos and they certainly don't have the experience. Whatever people's gripes about the site, I learned a hell of a lot from it. I still find solutions there, and think a world without it would be worse.
- dpkirchner 9mo ago
  I don't know if this is still the case but back in the day people would often redirect comments to some stackoverflow chat feature, the links to which would always return 404 not found errors.
- NewJazz 9mo ago
  The fundamental difference between asking on SO and asking an LLM is that SO is a public forum, and an LLM will be communicated with in private. This has a lot of implications, most of which surround the ability for people to review and correct bad information.
  - nprateem 9mo ago
    Surely the fundamental difference is one asks actual humans who know what's right vs statistical models that are right by accident.
    - ijidak 9mo ago
      Providing context to ask a Stack Overflow question was time-consuming. In the time it takes to properly format and ask a question on Stack Overflow, an engineer can iterate through multiple bad LLM responses and eventually get to the right one. The stats tell the uncomfortable truth. LLMs are a better overall experience than Stack Overflow, even after accounting for inaccurate answers from the LLM. Don't forget, human answers on Stack Overflow were also often wrong or delayed by hours or days. I think we're romanticizing the quality of the average human response on Stack Overflow.
johnsmith1840 9mo ago
You still get the same thing though? That grumpy guy is using an LLM and debugging with it. Solves the problem. AI provider fine tunes their model with this. You now have his input baked into it's response. How you think these things work? It's either a human direct input it's remembering or a RL enviroment made by a human to solve the problem you are working on. Nothing in it is "made up" it's just a resolution problem which will only get better over time.
- nprateem 9mo ago
  How does that work if there's no new data for them to train on, only AI slurry?
bluedino 9mo ago
There are so many "great" answers on StackOverflow. Giving the why and not just the answer.
kurtis_reed 9mo ago
Q&A isn't going away. There's still GitHub Discussions.
znpy 9mo ago
> But losing SO means that we're getting an idiot friendly guy with a lot of credible but wrong answers in place of a grumpy and possibly toxic guy which, however, actually answered our questions. Which by the way is incredibly ironic to read on the internet after like fifteen years of annoying people left and right about toxic this and toxic that. Extreme example: Linus Torvalds used to be notoriously toxic. Would you still defend your position if the “grumpy” guy answered in Linus’ style?
- etamponi 9mo ago
  > Would you still defend your position if the “grumpy” guy answered in Linus’ style? If they answered correctly, yes. My point is that providing _actual knowledge_ is by itself so much more valuable compared to _simulated knowledge_, in particular when that simulated knowledge is hyper realistic and wrong.
- johnnyanmac 9mo ago
  Sadly, an accountable individual representing an organization is different from a community of semi-anonymous users with a bunch of bureaucracy that can't or doesn't care about every semis anonymous user
RobinL 9mo ago
I'm hoping increasing we'll see agents helping with this sort of issue. I would like an agent that would do things like pull the spark repo into the working area and consult the source code/cross reference against what you're trying to do. Once technique I've used successfully is to do this 'manually' to ensure codex/Claude code can grep around the libraries I'm using
solumunus 9mo ago
Because what you’re describing is the exception. Almost always with LLM’s I get a better solution, or helpful pointer in the direction of a solution, and I get it much faster. I honestly don’t understand anyone could prefer Google/SO, and in fact that the numbers show that they don’t. You’re in an extreme minority.

ianbutler 9mo ago

It's flat wrong to suggest SO had the right answer all the time, and in fact in my experience for trickier work it was often wrong or missing entirely. LLMs have a better hit rate with me.

danver0 9mo ago
[flagged]
- johnsmith1840 9mo ago
  I mean when I was pure SO debugging I would say a solid 70% of the responses did not work. Different stack different bug ect. LLMs even if they can't guess the right answer can search the documentation really well.
- ianbutler 9mo ago
  [flagged]
  - GaryBluto 9mo ago
    I'm not sure he intended it as an insult. Despite not using LLMs for HTML or JavaScript often (if at all), I was under the impression that it was one of their strong areas.
    - ianbutler 9mo ago
      In my experience, at least in circles I've been around, I have found other developers who are a little too high on their own supply so to speak use "hurr durr web dev" as the "easy not real dev" insult. So I both wanted to shame that in case they were and also categorically show otherwise.
paganholiday 9mo ago
The example wasn't even finding a right answer so I don't see where you got that.. Searching questions/answers on SO can surface correct paths on situations where the LLMs will keep giving you variants of a few wrong solutions, kind of like the toxic duplicate closers.. Ironically, if SO pruned the history to remove all failures to match its community standards then it would have the same problem.
- ianbutler 9mo ago
  "But losing SO means that we're getting an idiot friendly guy with a lot of credible but wrong answers in place of a grumpy and possibly toxic guy which, however, actually answered our questions." > "actually answered our questions." Read carefully.
  - paganholiday 9mo ago
    >> Eventually I tried with something else, and found a question on stackoverflow, luckily with an answer. That was the game changer and eventually I was able to find the right doc Read carefully and paraphrase to the generous side. The metaphor that follows that is obviously trying to give an example of what might be somehow lost.
    - ianbutler 9mo ago
      This is a fair critique. I am often not generous enough with people.
  - akoboldfrying 9mo ago
    Interpreting that claim as "SO users always, 100% of the time answer questions correctly" is uncharitable to the point of being unreasonable. Most people would interpret the claim as concisely expressing that you get better accuracy from grumpy SO users than friendly LLMs.
    - ianbutler 9mo ago
      For the record I was interpreting that as LLMs are useless (which may have been just as uncharitable), which I categorically deny. I would say they're about just as useful without wading through the mire that SO was.
jtrn 9mo ago
No point in arguing with people who bring a snowball into Congress to disprove global warming.
vultour 9mo ago
It entirely depends on the language you were using. The quality of both questions and answers between e.g. Go and JavaScript is incredible. Even as a relative beginner in JS I could not believe the amount of garbage that I came across, something that rarely happened for Go.

specproc 9mo ago

What I always appreciate about SO is the dialogue between commenters. LLMs give one answer, or bullet points around a theme, or just dump a load of code in your IDE. SO gives a debate, in which the finer points of an issue are thrashed out, with the best answers (by and large) floating to the top. SO, at its best, is numerous highly-experienced and intelligent humans trying to demonstrate how clever they are. A bit like HN, you learn from watching the back and forth. I don't think this is something that LLMs can ever replicate. They don't have the egos and they certainly don't have the experience. Whatever people's gripes about the site, I learned a hell of a lot from it. I still find solutions there, and think a world without it would be worse.

dpkirchner 9mo ago
I don't know if this is still the case but back in the day people would often redirect comments to some stackoverflow chat feature, the links to which would always return 404 not found errors.
NewJazz 9mo ago
The fundamental difference between asking on SO and asking an LLM is that SO is a public forum, and an LLM will be communicated with in private. This has a lot of implications, most of which surround the ability for people to review and correct bad information.
- nprateem 9mo ago
  Surely the fundamental difference is one asks actual humans who know what's right vs statistical models that are right by accident.
  - ijidak 9mo ago
    Providing context to ask a Stack Overflow question was time-consuming. In the time it takes to properly format and ask a question on Stack Overflow, an engineer can iterate through multiple bad LLM responses and eventually get to the right one. The stats tell the uncomfortable truth. LLMs are a better overall experience than Stack Overflow, even after accounting for inaccurate answers from the LLM. Don't forget, human answers on Stack Overflow were also often wrong or delayed by hours or days. I think we're romanticizing the quality of the average human response on Stack Overflow.

johnsmith1840 9mo ago

You still get the same thing though? That grumpy guy is using an LLM and debugging with it. Solves the problem. AI provider fine tunes their model with this. You now have his input baked into it's response. How you think these things work? It's either a human direct input it's remembering or a RL enviroment made by a human to solve the problem you are working on. Nothing in it is "made up" it's just a resolution problem which will only get better over time.

nprateem 9mo ago
How does that work if there's no new data for them to train on, only AI slurry?

bluedino 9mo ago

There are so many "great" answers on StackOverflow. Giving the why and not just the answer.

kurtis_reed 9mo ago

Q&A isn't going away. There's still GitHub Discussions.

znpy 9mo ago

> But losing SO means that we're getting an idiot friendly guy with a lot of credible but wrong answers in place of a grumpy and possibly toxic guy which, however, actually answered our questions. Which by the way is incredibly ironic to read on the internet after like fifteen years of annoying people left and right about toxic this and toxic that. Extreme example: Linus Torvalds used to be notoriously toxic. Would you still defend your position if the “grumpy” guy answered in Linus’ style?

etamponi 9mo ago
> Would you still defend your position if the “grumpy” guy answered in Linus’ style? If they answered correctly, yes. My point is that providing _actual knowledge_ is by itself so much more valuable compared to _simulated knowledge_, in particular when that simulated knowledge is hyper realistic and wrong.
johnnyanmac 9mo ago
Sadly, an accountable individual representing an organization is different from a community of semi-anonymous users with a bunch of bureaucracy that can't or doesn't care about every semis anonymous user

RobinL 9mo ago

I'm hoping increasing we'll see agents helping with this sort of issue. I would like an agent that would do things like pull the spark repo into the working area and consult the source code/cross reference against what you're trying to do. Once technique I've used successfully is to do this 'manually' to ensure codex/Claude code can grep around the libraries I'm using

solumunus 9mo ago

Because what you’re describing is the exception. Almost always with LLM’s I get a better solution, or helpful pointer in the direction of a solution, and I get it much faster. I honestly don’t understand anyone could prefer Google/SO, and in fact that the numbers show that they don’t. You’re in an extreme minority.