9 ms·
My prediction is that pre-2022 data in machine learning will be like pre-WW2 steel in the medical industry (no radiation contamination from nuclear tests, canno
by juujian 3y ago
My prediction is that pre-2022 data in machine learning will be like pre-WW2 steel in the medical industry (no radiation contamination from nuclear tests, cannot be replicated). An irreplaceable resource with a limited supply. As much as I hate to say it, Reddit may be sitting in an actual gold mine.
- fnord77 3y agoare there any scraped copies of reddit data anywhere?
- birdyrooster 3y agoOne compensatory effect is that the noise will be more coherent than it used to be. Imagine how much entropy old SEO copy or spam could introduce to a model.
- jchw 3y agoOf course they're sitting on a goldmine. If they weren't, they'd be absolutely mortified by what they're doing to their own website. I think by June 30th, it's going to prove to be a solid win for Reddit: they still have all of the data they ever had, and they'll have better ad revenue than ever. Unless the community can get a lot more creative, Reddit is about to be absolutely fucking rich (even though the communities will all be hallowed out.)
- barelyauser 3y agoThe average Joe enriching the elites by voluntary unpaid work. What a time to be alive.
- jimbokun 3y agoIt's kind of funny to me, though, that the mods just recently realized that's what they've been doing all along.
- damnesian 3y agoCertainly a time to be a landowner. For the rest of us, serfdom beckons even more inescapably.
- retrocryptid 3y agoIt is sorta funny... I came to sili valley a moderate republican. I left it a marxist.
- mcmcmc 3y agoThis is why anyone serious about "protesting" the changes should be deleting their account and overwriting/deleting all their comments and posts. Does it suck for the future, and for anyone trying to use that wealth of collective knowledge? Yes, of course. But that's also the only way to really hurt them at this point. Subreddit blackouts are completely ineffective, and drops in traffic with resultant dips in ad revenue are momentary blips. It's scorched earth or nothing.
- jchw 3y agoHonestly, I think they're intentionally undeleting comments and un-editing them right now. Why wouldn't they? I'm pretty sure they legally can, and they clearly do not care that people hate them. I mean personally, I wouldn't feel safe if I worked for Reddit right now, but I am guessing the executives could absolutely not care less if anyone gets hurt over their amazing plan to get absolutely filthy rich.
- hutzlibu 3y ago"Reddit is about to be absolutely fucking rich (even though the communities will all be hallowed out.)" If the communities will be gone for good (not likely at this point) - who will they be serving ads to?
- jchw 3y agoThe users producing most of the value that made Reddit interesting when it was a small website that had a lot to prove have probably already left. The thing is though, it's now very well-established. Reddit shows up extremely often when you Google it, and undoubtedly there's a lot of people (possibly a majority) who will just kind of shrug this off after deciding they'd rather suck it up than be without something like Reddit for a while. But while the ad revenue is probably fine, the API revenue is definitely what they think the cash cow will be, and practically speaking they could lose 75% of their users and still probably not lose a whole lot of value just due to the sheer amount of historical data they have (especially if they start selling deleted content.)
- X0Refraction 3y agoThere are signs that the blackout will have an affect on their bottomline if it's kept up [1]. I'd imagine that's why they're making these moves to remove moderators participating in the blackout so quickly. I think it'll be interesting to see if they can find other moderators who participate for free and bring as much value to reddit as the current mods do. My guess is that the changes will still go ahead and the general quality of content on reddit will go down longer term although how much is an open question. [1] https://www.adweek.com/social-marketing/ripples-through-reddit-as-advertisers-weather-moderators-strike/# https://www.adweek.com/social-marketing/ripples-through-redd...
- edwiecron 3y agoIf reddit doesn't do this, the bots will take over anyways. They probably already have. AI is going to do a shitty job identifying AI, so all of your datasets are going to get worse and worse. I could imagine a bunch of generative AI bots just melting down every message board on the planet to the point where you have to stop training on recent data. I was hoping for the bots to export their existential crisis' to the rest of the web. Bots producing a loop, where they train on the data that they produce, would create some insane results.
- retrocryptid 3y agoI believe you mean "hollowed out" and not "hallowed out." There's an important distinction between the two words.
- jchw 3y agoGood catch, although now I'm outside of the edit window :)
- retrocryptid 3y agoGood enough just to verify your intent. I mean. It seemed obvious to me, but this subject matter seems to encourage people to wax poetic (and sometimes religious.)
- quikoa 3y agoIf you search around a bit the data can still be found. I don't know about the legal part of this so I'm not going to post any links.
- raverbashing 3y agoI'm very not sure about this Text quality in reddit is awful. Especially comments . Forget using it for any semblance of actual grammatic correctness. Probably can be used, after heavy cleaning and curation.
- Dr_Birdbrain 3y agoI think in this context, grammatical correctness is irrelevant, and what you actually want is just raw text as written by real humans from all walks of life.
- juujian 3y agoYup, variety is a big part of it. You don't just want a large language model that can produce meticulous text, you also need to be able to handle user input, which will frequently include faulty input itself. Otherwise you end up with something as mechanic and inflexible as google assist. Interactions with large language models can be much more organic if you have this kind of input data.
- ben_w 3y agoAgreed. We can make infinite grammatically correct text with perfect spelling, but if we limit the training set to that alone then ppl who wright liek this cant uze the ai lol & will be all lije "wtf m8 this aint skool i don care bout spekling rite u know wot i ment y cant ur machine?!?" … and while I enjoy my ivory tower as much as the rest of us, I do think it's important for the AI to understand everyone, not just me.
- MattGaiser 3y agoDepends on what you are looking to create. Good writing? Then it is an issue. Authentic seeming writing? Then it is exactly what you want.
- Balvarez 3y agoI might be wrong but I think they are sitting on the corpse of the golden goose.
- bonecrusher2102 3y agoWhat about it is valuable? Isn't all of that "data" just their content that is pretty easily scrape-able by anyone?
- Workaccount2 3y agoUnfortunately for reddit, there are many backups of that data that aren't held by reddit.
- lannisterstark 3y agoYep, a fuckton of it has been archived.
- jimbokun 3y agoBut does Reddit still control the legal rights? Who holds the copyright, and rights to use the data for various purposes? If some new AI was trained on those backups, could Reddit take the creators to court for using that data without a license?
- tyfon 3y agoThe users own their comments afaik, reddit only holds a license to them.
- mptest 3y agoAre closed source products forced to reveal their training materials? Would be interesting to see a new company come out be closed source have a killer product then get forced to reveal training data in court only to reveal they pretty much only used stuff they didn't have rights to. Is it technically illegal yet to "learn" from data you don't have ownership of yet?
- freedomben 3y agoIf somebody uses it and achieves some success, you can bet the truth will come out. Whether whistle blowers or crafty people on the internet piecing it together, people are pretty good at figuring that sort of stuff out.
- visarga 3y agoTake that one step further - what about learning on generated data from a model that used copyrighted data? It is exactly what MS "Orca" does. Does this step launder copyright?
- awb 3y agoPre-2022 Internet content definitely had spam/bot contamination though.
- thesuitonym 3y agoReddit has a lot of text on it, sure, but it's definitely not the best model out there, or should we expect language models that think it's appropriate to reply with "r/unexpectedoffice" for every single prompt?
- juujian 3y agoI mean, Wikipedia is certainly better, but Reddit covers some niches of language that Wikipedia will never cover. Wikipedia, Reddit, and Twitter are the three predominant sources of data for large language models, and Twitter has been such a pain for researchers to access for a while already.
- bonecrusher2102 3y agoWhat about like, blogs, for instance, as a source of training data? Is the differentiating factor here something like, it's lots of people interacting with other people?
- retrocryptid 3y agoBlogs are hard. You have to manage a large collection of shallow links. People like reddit and Twitter because there are very few deep links. And in the old days you just opened up a pipe and Twitter streamed new tweets directly at you.
- bagacrap 3y ago> pre-WW2 steel in the medical industry (no radiation contamination from nuclear tests, cannot be replicated) Interesting rabbit hole this one... but FWIW it's not as dramatic as this comment makes it sound. Newer steel is good enough because background radiation has dropped significantly since the 60s, plus there are non standard steel making techniques that don't lead to contamination.
- juujian 3y agoHaven't gone down that rabbit hole for a while, thanks for the update!
- NineStarPoint 3y agoDepending on the use case newer steel made in a standard fashion isn't always good enough. But indeed it's not that it cannot be replicated, but just that it's often more expensive to replicate than it is to just recycle old steel instead.
- samstave 3y agoI wonder if the seeming uptick in cancers for people born in the 50s is related to just how many test explosions were done? I cant pull up the stats via chatGPT, but I can pull up cancer stats from NIH - but the goal is to look at cancer death rates for the population of Las Vegas Nevada by each decade - as it states that radiation background has dropped since the 1960s -- It would be interesting noting births in Las Vegas starting in 1950, and comparing that with cancer rates btwn 1965-1975 compared with the number of nuke tests done in NV during the 1960s - My grandfather was a nuke eng for GE for 50+ years, he was on the design team of Hanford... He died of thyroid/throat cancer via exenguination and we won a lawsuit against GE for nukes that were exposed to high radiation levels unbeknownst to them for exposure and risks/threats...
- OscarCunningham 3y agoReddit is a mithril mine and r/counting is the Balrog.
- passion__desire 3y agoWhat if, future AIs gain capabilities to sort, categorize, discriminate, deduplicate and rank new post-2022 datasets? When we ask for a text to be human produced, what we are really asking for is novelty and groundedness to facts. Most of the opinions on reddit is regurgitated hive mind thoughts.
- ChatGTP 3y agoNot the valuable stuff, the the world evolves and there’s an almost infinite supply of new stuff to talk about.
- dylan604 3y agoand yet here we are like a broken record talking about chatbots
- retrocryptid 3y agoA Comment section post, like language, is a virus.
- ChatGTP 3y agoWhat should we talk about then ?
- juujian 3y agoOnce you have a reliable model to detect AI-content, you also have a new tool to train AI-models against. So ironically, any tool to detect AI content will just serve to make AI content more difficult to detect.
- PartiallyTyped 3y agoWhich you can use to train a better model to detect AI content. I wonder whether some models fit this - well, you could certainly call it generative, and obviously adversarial - framework of networks?
- 3y ago
- kneebonian 3y agoI had the realization this morning the people that really have a significant most are google because they have Gmail. Imagine how many people are sending messages every day via Gmail and how much of that text is human to human. And if you believe that Google wouldn't use Gmail in a heart beat to start training their models once they figure out a good way to do it I've got a bridge to sell ya.
- pyjarrett 3y agoNot only this, but also books. There's a big difference between ghostwriting (biographies and fiction) and then the now fully GPT-generated books (non-fiction, especially technical) for sale on Amazon.
- orbital-decay 3y agoIt's far more likely that training methods will improve. Humans don't have this issue.
- lachlan_gray 3y agoMaybe we will need “human farms” in response to bot farms. Maybe “professional human” will be a viable career option if human created data accrues extreme economic value.
- mahathu 3y agoThat prediction and analogy to low-background steel has been made 128+ times before on HN: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=low%20background%20steel&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
- juujian 3y agoNever realized I was so predictable
- simcop2387 3y agoAre you sure you aren't an ai already? That's also one reason I'm not so sure about the analogy because humans are already so gigantically redundant and we have lots of other bots already. I don't think there's ever been a low background steel on the internet in any real capacity. Just now it's more radioactive than it used to be
- juujian 3y agoI will check next time I have an opportunity. Ironically, the only way to prove I'm not an AI would be for me to post content that would be restricted in language models, so here comes my proof: shit©
- moffkalast 3y agoThat proves nothing, you could just be any of the uncensored LLama or Falcon models.
- ramesh31 3y ago>Never realized I was so predictable "The limits of my language mean the limits of my world" Our entire personality and sense of identity is nothing but the reflection of ourselves in relation to eachother. The summation of all cultural input. A man in a void is no more than a beast.
- interroboink 3y agoIf you haven't read it, check out the story "Lena" about MMAcevedo: https://qntm.org/mmacevedo https://qntm.org/mmacevedo
- nanidin 3y agoThe goldmine has been archived and democratized the whole time: https://academictorrents.com/details/7c0645c94321311bb05bd879ddee4d0eba08aaee https://academictorrents.com/details/7c0645c94321311bb05bd87...
- paddw 3y agoI think that as long as actual human beings continue to contribute content, AI would eventually "refine" that out and incorporate that into newer models. Even if say, 95% of the content in the dataset is AI generated, it will still capture the remainder. Of course, I don't know if scaling AI with more data will work to improve it or not, and I'm sure that anyone really knows right now.
- retrocryptid 3y agoWe can introduce healthy cows into the downer cow population. Even if say, 95% of the herd is exposed to BSE, it will still capture the healthy parts of cow carcasses we feed the downer cows. (Which is to say... I think there are more than homeopathic concentrations of questionable training data. Or at least I believe that was the assertion of the OP.)
- blarg1 3y ago> Reddit may be sitting in an actual gold mine A pretty depressing thought to imagine our societies run by a bunch of ai redditors.