3 ms·
Call me crazy but sometimes I still find a better solution on StackOverflow than what Claude Code insists to do. I'm not sure we're better off without SO in th
by conradfr 5mo ago
Call me crazy but sometimes I still find a better solution on StackOverflow than what Claude Code insists to do.
I'm not sure we're better off without SO in the long run.
- ceejayoz 5mo agoMuch of what Claude insists you do probably came from SO.
- akkad33 5mo agoOr Reddit. I don't know about Claude but Gemini has given me answers that are verbatim comments from Reddit.
- arcanemachiner 5mo agoI've gotten my own answers given back to me for problems I forgot I already had.
- irishcoffee 5mo agoI had email correspondence once with a vendor about how to talk to their i2c bus. The documentation was all asm, and I wanted to at least “uplift” to C. They didn’t have any answers, so I sent them my solution which was was the asm calls that the c stdlib decompiled into. 4 years later my company had bought a different company, who happened to be using a newer model of the same board. They asked me how we could use the 12c bus. “Well before you bought us, we emailed the vendor and sent back this C snippet” It was my code, verbatim. I’ve always wondered how many times they passed that bit of code around.
- mcswell 5mo agoSomebody posted a similar comment above yours (somewhere...). I don't think your experience is unique!
- dd8601fn 5mo agoClaude does it quite a bit when you’re triggering the search tool functions. It’s fine, and what you would expect for certain prompts, except that the synthesized results often come back communicating more authority than they deserve.
- lukan 5mo agoIt was funny for me, when I asked it about something specific exotic - and it gave me a confident answer. But checking the sources I discovered it was from my own inquiries on a forum thread about it from the last time I unsuccesfully tried this (before the agents came) And so I knew, that any authorative tone was undeserved. On the other hand, Claude later nailed this project, where I as a human said before, no, too much extra work.
- Morromist 5mo agoI've gotten this too a lot. If you ask AI to cite where it got info you can lose a lot of confidence in it pretty quickly.
- worthless-trash 5mo agoI have seen it quote my own code back at me, including comments word for word.
- 20k 5mo agoI've seen chatgpt word for word plagiarise stack overflow answers
- andrekandre 5mo agoi've seen it plagiarize personal blog posts too, almost verbatim code line by line.... kind of shocking...
- tartoran 5mo agoWell, it's a plagiarizing machine after all and most of the time it remixes it well enough so most people can't tell.
- imtringued 4mo agoYeah, that's basically it. In robotics there is no free lunch dataset. You'll have to gather it yourself, but if you do that, you run into an obvious problem: labeling. With SO, you literally have the best possible scenario, because the data is clearly structured and separated into prompt and answer (aka label). Your objective function is literally "Say what he said".
- gpugreg 5mo agoSame here. LLMs are great at spitting out well-known solutions to problems instead of the best one. The "long tail" of solutions is usually lost due to how tokens are sampled from the LLM's probability distribution. What I found to help a lot is to ask for e.g. 10 different solutions to a problem and then choosing one of them. Sometimes, this even leads to borderline creative solutions if there aren't 10 different ones.
- exe34 5mo ago> LLMs are great at spitting out well-known solutions to problems instead of the best one. I remember how Stack Overflow would close questions as duplicates just because somebody suggested the wrong answer that is also the right answer to the existing question. The best way to get a correct answer on Stack Overflow (and forums before that) was to post the wrong answer as part of your question.
- btown 5mo agoIn theory reasoning tokens should do the equivalent of this - explicitly create options outside of the quick-response probability space, so those can guide future generation. In practice, models that do this won't be prioritized as much, because the economics of thinking tokens that stop by default at, say, one option plus a bit more planning (short of full alternatives) would be superior as long as billing is per-user instead of per-token. So we'll still need to play games with prompting!
- tliltocatl 5mo agoWithout continuous feedback from real world, lower-probability token (and soon high-probability ones as well) will be complete garbage.
- raffael_de 5mo agowell, SO is probably the highest quality data source for a language model and the rest of the internet is just diluting the final latent space limited by Jon Skeet.
- Morromist 5mo agoOne thing that SO had was you could see multiple solutions and implementations for something. Sometimes the "best" solution isn't very readable code, sometimes you are able to understand the problem better when you see a bunch of people solving it in different ways and arguing about it like angry monkeys. It really could be bad though.
- ssl-3 5mo agoThe bot can do that kind of thing, too. "Show me 6 very different solutions, and present arguments for/against each one as if a bunch of angry monkeys."
- allknowingfrog 5mo agoSO has always had a pretty strong stance against opinion-based questions, but this is maybe the niche they should be exploring now. Humans still have a lot to say about the "best" solution to a given problem. The whole idea of an "accepted" answer could be removed, for example, since that's what AI will already generate.
- deaton 5mo agoDefinitely not better off. SO was fairly mean spirited, but nowhere else has such a vast trove of high quality answers to common software problems been collected. SO likely trained many of these models with its answers, and I don't know what software development will look like when it dies.
- NoMoreNicksLeft 5mo agoNo where else has such a vast trove of high quality answers been hidden because the question was closed as duplicate when someone else asked that question later.
- FrustratedMonky 5mo agoAnd, the AI trained on Stack Overflow. So if no one is posting new questions, and new answers. What will AI train on next, for the next thing.
- bigfishrunning 5mo agoyour prompts, and the code you have it review.
- FrustratedMonky 5mo agoMaybe. I thought point was on Stack Overflow, there were community voting on 'best' answer. If it is just me and the AI. Then the AI training data, is just whatever I approved the AI to do. Just my opinion.
- ishurand4 5mo agoStack Overflow and Reddit are still getting threads. And as AI gets smarter, the questions will also expand.
- whateverboat 5mo agoWhat you are noticing in a long term is the "community" knowledge and communication which the chatGPT is now kind of destroying. In some sense, it is no different from the difference between studying along and studying with your peers at a university. You can definitely study alone and achieve perfect grades, but studying with your peers is how you build relationships for future life and take your community forward as a whole.
- jshen 5mo agoWe may need to create a community driven version of SO. Hard for it to be a successful business these days.
- sixtyj 5mo agoI am thinking to make canned encyclopaedia of stackoverflow answers. Claude/Grok/Gemini/Chatgpt answers are often so… how to say it… misleading? I have to stop the conversation as it leads nowhere (and it is not a skill issue :)
- deleted 5mo ago[deleted]
- andrekandre 5mo agowas just gonna post the same thing needed to implement a language feature that was a bit complicated and im not familiar with it so just planned with claude to do it, and after each write/fix cycle it just wouldn't work right.... gave up, went back to SO copy pasted the (not perfect but enough to start from) answer and worked up from there... at the same time my knowledge grew and im more confident to do this same capability myself whereas reiterating with claude it was just a slog and i didn't learn much... i think i may be starting to sour on these "do it all for me" usage scenarios for ai... especially for unfamiliar areas...
- ImageXav 5mo agoAgreed. Which is also odd, if you think about it. Surely with the amount of compute Anthropic and others have available, they could test each of the solutions in the SO data they surely have and rank them based on efficiency/elegance/other criteria and remove poor solutions from their training data.
- tayo42 5mo agoWhat are you looking for and finding on stack overflow that isn't begginer to intermediate level?
- Bender 5mo agoI'm not sure we're better off without SO in the long run. You're right but that site has been sputtering culturally for some time. I put a lot of effort into editing questions and answers on ServerFault (part of SO) but I feel that time was wasted. I think they knew for a while they just wanted to sell it and just stopped caring. A number of editors were allowed to be jerks for too long and it went to their heads. I wish I could take back all that effort.
- insane_dreamer 5mo agoagreed; sometimes the most useful info on SO wasn't in the replies, but the follow-up comments to the replies