3 ms·
> There is no question that Reddit is extremely valuable as training data. How often do you append “reddit” to your searches? Is it? When I worked in networki
by duxup 3y ago
> There is no question that Reddit is extremely valuable as training data. How often do you append “reddit” to your searches?
Is it?
When I worked in networking, and later web dev work I found Reddit to be a TERRIBLE place for Q & A type situations.
Answers on Reddit are often skewed by truthy answers from people with limited perspective in the industry who are surprisingly sort of militant about a given topic.
For example the networking topics often were “small shop” focused and users unaware of very common enterprise level practices, and when they saw someone mention it they would react really poorly. The result was advice tailored to small shops that have limited staff, so you got ideas that “work” but would perform poorly at scale (at best) and at times could present security risks, miss opportunities where more performance was needed, better solutions were available with a sizable staff and planning. They weren’t wrong, but the answers were skewed.
Subreddits are weird, they might be about a topic but the community often moves that topic/ has a specific pov within that topic… people without those views tend to abandon those subs and those who remain aren’t necessarily “right”. Reddit isn’t stack overflow where being correct is at least on the surface valued where we can run code and see if it works. Top answer on Reddit could be wrong and anyone taking issue with it may simply not answer. Heck there are whole subreddits dedicated to wallowing in misinformation…
Now I don’t know if Q&A is the start and end of LLM training (not my area of expertise at all, I am open to the possibility that what I’m talking about isn’t a problem at all) but Reddit as a source makes me wonder what the results would be.
- rjh29 3y agoYes. For anything from opinions of movies/games to DIY advice to nerdy stuff like the best watch to buy in a given price range or the best synthesiser. Or people's opinions on a particular episode of TV! Or basically any hobby. It's not great for business stuff because there are way more people using e.g. AWS for their hobby project than running multi-million $ businesses on it. What are the alternatives? Facebook and Discord are walled gardens. Quora is a shithole. The rest of the internet is blogspam and significantly more dubious than your average Reddit comment!
- flandre 3y agoMy favorite alternatives are email and going outside.
- Hunter_Barrett 3y ago[flagged]
- rjh29 3y agoGoing outside is a great alternative to mindlessly consuming Reddit, but we're talking about researching opinions here. That's something that Reddit is actually good for.
- mtlmtlmtlmtl 3y agoSo is going outside though. In fact, I think talking in person is usually a much better way of actually understanding someone else's opinion, because normative rules of social behaviour tend to stop the discussion from becoming a flamewar. I could spend hours just neutrally asking someone questions about a particular view they hold, why they hold it, etc. I think you gain a lot more insight into how people think this way than through an online forum.
- KptMarchewa 3y agoWho cares about "particular views they hold"? We're talking about fridge or tv brands, not some personal stuff.
- mtlmtlmtlmtl 3y ago>Who cares I do.
- Symmetry 3y agoPerhaps GP meant that since desire is the root of suffering, that rather than desire that your server work and suffer and strive to return it to functionality, you could go outside and mediate on the grass until you relinquished your desire for a working server. Or your desire for an umbrella that doesn't break in the wind, or the best public estimate of when Starship will launch, or any of the myriad other desires that cause people to append "reddit" to their searches.
- waihtis 3y agonot only that but 80% of conversations on reddit is idiotic jokes and puns
- john-radio 3y ago[flagged]
- Symmetry 3y agoThis is absolutely true for any subreddit that shows up on the front page but many subreddits with smaller user bases or draconian moderation policies (like /r/askhistorians) can have excellent quality.
- Arelius 3y agoYeah, interesting point. I'm curious to what your take is on Chat-GPT's response to the same sorts of questions? I also work in a specialist field, and I've come to understand that the instinct on of a particular question is going to give me a bad answer on Reddit or SO, is also finely tuned to about when GPT either starts hallucinating, or providing bad info.
- Lukeas14 3y agoThat's really only an issue for technical topics. Stack Overflow would obviously be a much better source of quality training data for the kind of questions you presented. But for most things outside of that Reddit is by far the best online source that actually represents answers you would get from real people. Questions like what is the nightlife like in x city don't have a single true answer and thrive due to Reddit's the diversity of thought.
- bdav2418 3y ago> Stack Overflow would obviously be a much better source of quality training data for the kind of questions you presented. I dont think we need a llm that only responds with "Your question was marked as a duplicate"
- avs733 3y ago> When I worked in networking, and later web dev work I found Reddit to be a TERRIBLE place for Q & A type situations. Training in this sense is about how people use language not what is the right answer. Among other sources - the exact thing you describe is a contributor to some of the things within the (in my opinion misnamed) internal taxonomy of 'hallucinations'. Right or wrong - reddit is human beings using (mostly english) language to speak to each other, and so if you want to train an ML model to do that - its a great source.
- chrismarlow9 3y agoMy experience in the past 5-10 years on Reddit is: - the voting system is about what people want the truth to be, not the truth. - the users can be very easily gamed in comments to vote one way or another just by the initial voting being negative or positive (aka most just vote with the trend) - the opinions all come from a bias of urban and major metro area people. It's painfully obvious they don't understand any mindset outside of this one. - it wasn't always this way. I agree with you, the quality went down as the quantity went up, and it may be possible to use the comments, but using the vote data as a signal of truth is going to produce a terribly narrow minded AI.
- duxup 3y ago> the voting system is about what people want the truth to be, not the truth Probably the best example is legal type advice. Personal example: The whole situation with internet archive and their e-book lending process. I was vocal that they didn’t have a leg to stand on and would lose. The comment was just about the legal situation. I was downvoted to hell for that opinion and identified as some sort of book publisher advocate/ internet archive hater. As was anyone else predicting a loss or just raising questions. Personally I love the internet archive, but I didn’t think they had a chance at winning. But go by the comments and it was a slam dunk win because of some idea of “freedom” the comments and voters had. And yet they lost the lawsuit… Interestingly, here on HN the response seem to be a generally more balanced. That’s not an endorsement of HN all the time, but it was an interesting contrast.
- dmonitor 3y agoI see the same thing with the situation surrounding the game "Dark and Darker". The TL;DR is that the game is likely stolen intellectual property and there is a lawsuit directed at the publisher and one of the developers. The lawsuit will likely prevent the game from being released, so the prevailing opinion on reddit is that the whole thing is an unjustified shakedown, despite the copious amount of evidence that the publisher stole intellectual property.
- BitwiseFool 3y ago
- tmpz22 3y ago> Answers on Reddit are often skewed by truthy answers from people with limited perspective in the industry who are surprisingly sort of militant about a given topic. Equally said of HN. Folks with a few years of dev or startup experience asserting this or that all over the place. Im sure less experienced folks read these "truthy" answers and take them as gospel in some cases.
- cooperadymas 3y ago> Answers on Reddit are often skewed by truthy answers from people with limited perspective in the industry who are surprisingly sort of militant about a given topic. That's kinda sorta exactly how ChatGPT answers stuff though, no?