9 ms·
Google cut a deal with Reddit for AI training data
- monkeydust 3y agoThis can't be exclusive for that amount or could it?
- dmix 3y agoNo way it’s exclusive. Reddit is making a push to find new $$ streams for their IPO and the API is a big part of that with AI being hot. They probably have lots of deals for their new API already and this one is just making headlines because it’s Google and it’s yearly/unlimited/relatively large.
- whoopdedo 3y ago> Google Search is currently expanding the test of a "forums" filter that lets you browse through results from sites with human discussion, like Reddit Yeah, about that whole "human discussion" thing...
- lcnPylGDnU4H9OF 3y agoI think Reddit is accurately described as a site "with human discussion". It's another matter whether such discussion is organic and actually shows up in search results.
- 0ckpuppet 3y agoIs that like the taco bell meat that "contains" some meat.
- salt-thrower 3y agoFair point that a lot of Reddit is already astroturfed, but this is definitely the final nail in the coffin. Now SEO is coming for the forums, too. Wonderful.
- jimbob45 3y agoIn fairness, Reddit's built-in search works well enough that I could still find stuff on Reddit if Google croaked tomorrow.
- Scoundreller 3y agoReddit search works effectively for you? I usually resort to using Google to search reddit.
- SV_BubbleTime 3y agoI always search: xyz site:Reddit.com And never use the internal search, I don’t know what gp is talking about. Reddit’s search is terrible.
- jprete 3y agoI think the idea is that if Google makes it easier to find human-authored content on Reddit, astroturfers and other spammers will have even more reason to post there.
- p1mrx 3y agoDidn't Google have a "forums" filter like 20 years ago? I can't seem to find results pages on archive.org.
- whycome 3y agoYou may be thinking of the Gmail default sorted inboxes
- quadrature 3y agonope, google had a discussions tab in search. Looked something like this https://www.ghacks.net/wp-content/uploads/2014/01/discussions.png https://www.ghacks.net/wp-content/uploads/2014/01/discussion...
- silisili 3y agoIt was called discussions search, and showed up in the suggestion bar with images and whatnot. I was beyond mad when they removed it. Everything's a circle.
- malfist 3y agoIf you want that now, kagi has a forums lens
- A_D_E_P_T 3y agoHah. In fairness, PR firms, digital marketers, "reputation management" experts, and SEO gurus are "human" too. These days at least 90% of product-related discussions on Reddit are astroturfed, though, and that's for sure. The main problem, among many others, is that it's far too easy to game the upvote/downvote system with bots, and most pros have bot networks of their own, or know how to hire them. The second problem is that Reddit "karma" is site-wide, which means that most "high karma" accounts that are sold on secondary markets are very useful for digital marketing purposes generally. The third problem... well... I could go on all day. Reddit's system is horrendous if you're looking for human discussion and unbiased advice.
- digging 3y agoKagi already has such a filter by default, although I've never tested it.
- LorenDB 3y agoHm, now I have to wonder if it'd be possible to create a tool that poisons your Reddit posts so they look fine to a human but completely trash LLM output. I'm not saying we should do it, but it's a fascinating thought.
- berbec 3y agoI'm saying we should do it for you.
- barryrandall 3y agodavidjl and several others did it.
- duffyjp 3y agohttps://www.smbc-comics.com/comic/got-you https://www.smbc-comics.com/comic/got-you
- mlekoszek 3y agoWhat about Markov chain babble? It would be interesting to see Bard learn to mimic people going far even as decided to use even go want to do look more like.
- purpleblue 3y agoSomeone should create a bot that responds to highly upvoted comments and remind the redditor that Reddit is making money off the comment and the information is being used by AI and they aren't seeing a dime of it.
- dageshi 3y agoIt was going to get scraped and used by AI anyway? Given that I'm not sure I care that much if reddit makes a bit of cash off it. Honestly, this is the future, if it's publically available on the internet it's going to be scraped and used by AI. The inevitable consequence of which is probably more conversation moves into private places like Discord. But that's the future we're in, it's not gonna get rolled back.
- belinder 3y agoI don't see why Discord would not follow suit if it has not already Everyone is privacy focused until they're not
- DiscourseFan 3y agoYeah Discord isn't E2EE, it could easily be transformed into a data mine, which is one of the reasons I stopped using it years ago.
- Imnimo 3y agoWith only $60M a year to go around, I'm not sure my share of comments would even add up to a full dime anyway. They can have it.
- hobs 3y agoThat's arguably more true in advertising driven reddit than AI reddit - no one comment is going to be worth the most in a data scraper but eyeballs win in advertising. I think the entire thing is scummy too, but its interesting to me how people think that advertisers were somehow less bad than this.
- mediumsmart 3y agooh cool so the contributors of the data are all getting paid?
- deleted 3y ago[deleted]
- MakeThemMoney 3y agoOr course! Same way we get paid by for the ideas we discuss here
- brookst 3y agoExactly. If you don’t get value from HN, don’t use HN. If you don’t get value from Reddit, don’t use Reddit.
- CSMastermind 3y agoPart of what makes hackernews worth participating in is that the owners are not profit-maximizing and trying to extract the maximum possible value from the site. It's a trade: y-combinator, by cultivating a group of talented technical contributors, gets to advertise hiring posts for free and, in some ways, autistically contributes to the community. In exchange, we all get a place for thoughtful discussion with reasonably fair moderation. It's a fair trade for those involved and closer to a non-profit model than Reddit which is clearly trying to extract as much value as possible from the content creators on their platform in exchange for giving them community/platform.
- paulddraper 3y ago> the owners are not profit-maximizing Should we tell him?
- relativ575 3y ago> Part of what makes hackernews worth participating in is that the owners are not profit-maximizing and trying to extract the maximum possible value from the site. How do you know? Just because there is no ads doesn't mean they don't have any benefits from keeping HN thriving. Also for me I like HN because of the content, not because I care about YC's motive. You are obviously different.
- steveBK123 3y ago$60M/year for GOOG to access all their data when they purport to be targeting a $5B valuation at IPO is really cheap. Arguably Reddit's value is it's data, and GOOG is renting it for 1.2%/year?
- taion 3y agoGoogle are already indexing Reddit, so this might reflect the incremental value of direct API access plus keeping everyone happy. I would be surprised if Reddit’s threats to block Google’s crawler were all that credible.
- hobs 3y agoIt's 100% this - Google is going out and shopping so they can get easy access and avoid any future lawsuits if things change in the AI training space. I imagine this wont be the last announcement in this space.
- leros 3y agoIt wouldn't surprise me either if something like half of Reddit's traffic was Google constantly crawling them.
- declaredapple 3y ago> Arguably Reddit's value is it's data Reddit has a chicken and egg problem - nobody knows what the value of it's data even is. They've never sold it before. You can sell pet rocks for 1 billion dollars, but nobody will value them if there's been 0 sales. (ask vs bid vs last sale price). I'm guessing reddit is rushing to sell their data so they can show that their data is worth anything at all.
- Scoundreller 3y ago> They've never sold it before. Do we know this?
- declaredapple 3y ago
- DueDilligence 3y ago[dead]
- jiveturkey42 3y agoAI is about to become really sarcastic, pedantic, and absurdly moralistic
- Almondsetat 3y agoThat's already what googlers developing it are
- deleted 3y ago[deleted]
- mvdtnz 3y agoAnd even more confidently wrong.
- Scoundreller 3y agoCan’t wait for heavily downvoted unfortunate truths to be eliminated from AI
- mike_hearn 3y agoThat already happens. Standard academic datasets that use Reddit start by eliminating any comments that are under +3, for example.
- Freedom2 3y agoConfidently wrong? I thought it was getting access to Reddit, not HN.
- bbor 3y agoHate on HN all you want, I’ve been without my ADHD meds (warning: the company “Done” is not technically a scam) and spending way too much time on each for the past few days, and I can say this for sure: at least people on HN pay heed to the concept of premises and careful, non-combative argument. Most responses on Reddit are “no, that’s dumb” or “yes, that reminds me of my metaphysical takes”…
- we_love_idf 3y agoThis is dystopian. A company like Google will be able to surveil more and the only outcome is more invasion of privacy and more targeted ads. Why doesn't FTC stand up for this?
- paulddraper 3y agoFor what exactly? Doing something you don't like?
- jpalawaga 3y agoI know reddit has hired a lot. This sort of feel like a fundraising round to keep operations afloat. This must be the real reason for destroying public API access.
- SeanAnderson 3y agoSource for Reddit hiring more than average tech company? I feel like every tech company scaled up in the bull market and then trimmed back down when hype subsided. For example, Google grew by 72,000 between 2019 and 2022 and has since shed about 20,000. Reddit is IPOing next month. IMHO they cut public API access as a way to simplify and pump bookkeeping numbers to get a better listing price.
- bbor 3y agoTo be fair, “they were only as insane as everyone else” doesn’t necessarily mean that they didn’t vastly over-hire. I mean Jesus I got hired in Silicon Valley in 2022, it was crazy out there
- shmatt 3y agoIn the long term no company offers free unlimited APIs like that. It was a dumb decision to offer it in the last 5 years
- 7373737373 3y agoCopying this right from the Reddit TOS for all those who consider posting anything of any value there, present or future: > You retain any ownership rights you have in Your Content, but you grant Reddit the following license to use that Content: > When Your Content is created with or submitted to the Services, you grant us a worldwide, royalty-free, perpetual, irrevocable, non-exclusive, transferable, and sublicensable license to use, copy, modify, adapt, prepare derivative works of, distribute, store, perform, and display Your Content and any name, username, voice, or likeness provided in connection with Your Content in all media formats and channels now known or later developed anywhere in the world. This license includes the right for us to make Your Content available for syndication, broadcast, distribution, or publication by other companies, organizations, or individuals who partner with Reddit. You also agree that we may remove metadata associated with Your Content, and you irrevocably waive any claims and assertions of moral rights or attribution with respect to Your Content. (https://www.redditinc.com/policies/user-agreement-february-15-2024 https://www.redditinc.com/policies/user-agreement-february-1...) tl;dr: "We will be able to do anything we fucking want with anything you contribute to our site"
- Scoundreller 3y agoReddit should be paying Google. Some of Reddit’s success can be attributed to the difficulty of getting your personal blog indexed about your ‘87 Nissan Sentra anymore on Google.
- CamelCaseName 3y agoMaybe. 75% of their incremental users are logged out and likely coming from Google. Still, maybe Google also derives a lot of value from Reddit. Would either of the two risk their position to extort some money from the other?
- silisili 3y agoI hope they're filtering it heavily and specifically. While Reddit does still have some valuable discussion in the more niche subs, I've noticed the main subs moving further left, hatefully and almost violently so. I can't wait to search for who the Republican nominee is, and Google tell me to kill myself.
- khazhoux 3y ago> I've noticed the main subs moving further left, hatefully and almost violently so. I’ve not seen this. What do you mean?
- deleted 3y ago[deleted]
- silisili 3y agoSo, I'm only replying to give some color into that thought, not to start a flame war or debate, because it is a legitimate concern I have with using the site as data. I'm not necessarily complaining that it's left leaning, it always has been in US parlance at least. My worry is that the tone as changed from disagreement to some form of disgust and hate, and yes even at times calls for murders. Maybe that's just the landscape now, broadly? Just look at some of the 'mainstream' comments linked below. This is probably a 'both sides' thing, but I don't see the other side represented here and I would have equal concerns if it were in similar manner. That said, I don't have a Reddit account, so just opened the site to see what it showed me. I'm not sure of its algo, as a lot of it is video games, but here are a few that showed up in the top 15 or so - https://www.reddit.com/r/nottheonion/comments/1axe9rk/darryl_george_texas_judge_rules_school_district/ https://www.reddit.com/r/nottheonion/comments/1axe9rk/darryl... https://www.reddit.com/r/LeopardsAteMyFace/comments/1axas1q/conservatives_who_insist_people_should_be_able_to/ https://www.reddit.com/r/LeopardsAteMyFace/comments/1axas1q/... https://www.reddit.com/r/AnythingGoesNews/comments/1axad5x/trump_left_stunned_as_his_potential_running_mate/ https://www.reddit.com/r/AnythingGoesNews/comments/1axad5x/t... head over to /news, and you see - https://www.reddit.com/r/news/comments/1ax8mnl/tax_evasion_by_millionaires_and_billionaires_tops/ https://www.reddit.com/r/news/comments/1ax8mnl/tax_evasion_b... https://www.reddit.com/r/news/comments/1axe1ac/darryl_george_texas_judge_rules_school_district/ https://www.reddit.com/r/news/comments/1axe1ac/darryl_george... /politics, you get https://www.reddit.com/r/politics/comments/1axa6m9/donald_trump_accused_of_not_having_the_cash_to/ https://www.reddit.com/r/politics/comments/1axa6m9/donald_tr... https://www.reddit.com/r/politics/comments/1ax49rq/alabamas_unhinged_embryo_ruling_shows_where_the/ https://www.reddit.com/r/politics/comments/1ax49rq/alabamas_...
- Fervicus 3y agoAn AI company with an agenda getting their data from a social platform with an agenda. What could go wrong?
- khazhoux 3y agoWhat would you say is the agenda of Reddit (the company, not the users)?
- add-sub-mul-div 3y agoAny ideas that make you uncomfortable are an "agenda" now.
- Fervicus 3y agoAttempting to rewrite history and racism regardless of the race involved makes me uncomfortable yes. Do you disagree that Google had an agenda when they programmed their AI to do the things it does?
- asadotzler 3y ago[flagged]
- razodactyl 3y agoThis is an interesting comment because we readily anthropomorphise every entity and fail to see Google as an umbrella for a group of people that may make decisions in terms of a democratic or autocratic means depending on context. In this instance "The Google" has expressed "an agenda". I disagree that individuals might have been trying to wrong society but unfortunately it's the measurable results and direction of the company as a whole that exhibits the agenda and in this instance it's quite a blow to the ability for society to record a factual account of history so it can learn from its mistakes. This seems to be the case for the latest tech. in general, suddenly upending the taxi industry, forcing small businesses to pay the gig-culture delivery tax, AI mass generating content and creating an imbalance in creative output. We have a fun decade ahead...
- deleted 3y ago[deleted]
- lijok 3y agoOn one hand a measly 60 mill for Reddits data On the other hand they bought a poison pill Is whoever behind the decision making here blindsided by Reddit being a massive trove of data, and are not realizing most of that data is shit?
- licheness 3y ago[dead]
- xyst 3y agoReddit is really pumping their P&L sheet in preparation for IPO. The parasites at the top have $ in their eyes rn
- ChrisArchitect 3y ago[dupe] Discussion on official post here: https://news.ycombinator.com/item?id=39471317 https://news.ycombinator.com/item?id=39471317
- rebelde 3y agoGoogle is paying Reddit instead of just taking it for free like they do from all other websites?
- BadHumans 3y agoSecond line in the article: "In an update on Thursday, Reddit announced it will start providing Google “more efficient ways to train models.” They will helping Google create a real-time pipeline to Reddit's data.
- huitzitziltzin 3y ago$60 M for absolutely cutting edge dick joke generation in next gen LLMs. I question whether Reddit data is worth anything at all.
- shmatt 3y agoReddit and Facebook Groups (the latter being much better) are the only 2 places I can search for reviews or recommendations to pretty much anything these days. Google & co. are impossible to use for "best" anything
- bpanon 3y agoMost of my google searches include "reddit" now. From what I've read from others, this is pretty common.
- crazysim 3y agoAmazing. It used to be a free BigQuery dataset. Apparently it was worth $60M.
- izyda 3y agoThe lack of revocability, marginal temporal value, and downstream governance I think makes the prospect of more such data deals happening slim -- or at least, slim without regret. I wrote an essay on this here: https://magis.substack.com/p/llm-data-sales-a-market-for-lemons https://magis.substack.com/p/llm-data-sales-a-market-for-lem...
- PedroBatista 3y agoOnly $60M? Damn.. Reddit is for the streets. And that's a sweet deal for Google, paying pennies to partially patch their deteriorating reputation about their search engine becoming useless.
- Mo3 3y agoLol, it's essentially just advertisement and psyops bots, echo chambers and specific discussions in specific subreddits at this point.
- lolpanda 3y agoReddit blocks all search engines from crawling their comments, essentially blocking all user generated contents. See this directive in https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt `Disallow: /r//comments////*` But I can still search Reddit on Google. How does Google manage to get the data?
- gruez 3y agoI think you're misinterpreting the rule. The relevant robots.txt rules are: User-Agent: * Disallow: */comment/* Disallow: /r/*/comments/*/*/*/* The url for the comment sections looks like: https://www.reddit.com/r/[subreddit]/comments/[id]/[slug]/ https://www.reddit.com/r/[subreddit]/comments/[id]/[slug]/. This doesn't match the above rules because there's only 2 parts after "comments", not 4 parts as specified by the rules.
- hollerith 3y agoNow it says "Disallow: /"
- Ajay-p 3y agoDoes Reddit really block all search engines, or do "all" search engines abide by No Spiders? If it's freely available it's ripe for scraping, no matter what Reddit may say.
- Mo3 3y agoJust a reminder this exists: https://github.com/j0be/PowerDeleteSuite https://github.com/j0be/PowerDeleteSuite https://github.com/andrewbanchich/shreddit https://github.com/andrewbanchich/shreddit
- latentcall 3y agoThis worked amazingly for me when I stopped using Reddit.
- klipklop 3y agoWho knows if the "deleted" data can still be sold off. Sure you can't view it on the website, but nobody knows what Reddit will hand to Google. Then again certain laws passed recently might make such a thing illegal...
- Mo3 3y agoThe second option I linked also replaces all comments and posts with random scrambled gibberish, not only "deleting" it. I would agree that deleted != actually deleted, but scrambling all content might actually work to prevent further siphoning
- deleted 3y ago[deleted]
- klipklop 3y agoWhat incentive is there for posters on Reddit to continue to produce new content that is sold off for a profit? Sure they agreed to it in the TOS, but I suspect people might leave the platform. Also how many of the posts are already generated by AI? Seems like a raw deal to pay for data that could be a large percentage of bot activity. Why visit Reddit at all for information when you can just ask Gemini.
- RecycledEle 3y agoI hope they get all versions of edited and moderated comments. An AI might be able to determine when the moderation was valid and when it was playing games. I would love to see a leak of that dataset, sorted my username.
- iteratethis 3y agoWhen I first learned that AI companies are vacuuming up all internet content without any regard for permission, attribution or compensation for the content creator, I found that deeply immoral. I figured they should pay for it. But now that they do (in this instance), I'm realizing this might be even worse. They can just buy the entire market in the same way they buy Google search users by paying Apple billions a year. And still the actual content creator, a Reddit user in this case, is not compensated. It's truly wild how lax regulation is. This is probably the most important technology ever created and we just let 2 companies have it all: the data and the compute.
- AH4oFVbPT4f8 3y agoThis was one of the reasons I no longer contribute to reddit. I don't post, comment, upvote nor downvote. I'll still consume it from time to time, something I try to stop doing. All I can do is vote with my money and my time and I try to spend it else where.
- asadotzler 3y agoSection 230 lets them pretend they don't own the content, shielding them from liability, yet they can simultaneously claim ownership enough to sell it. That's effed up. Either it's not theirs, it's mine or ours, in which case I get to sell it or at least get a cut, or it's theirs in which case they're liable for any illicit content and should face those suits. Which is it, Silicon Valley? Is my post yours or is it mine? You shouldn't get to have it both ways.
- jfghi 3y agoI wonder how well they identify posts written via AI (and what proportion of posts that is). Also, there is a ton of misleading astroturfing for some types of businesses.
- gremlinsinc 3y agoDo people still use reddit? I haven't since RelayForReddit went subscription only thanks to the new Reddit API rules.
- rldjbpin 3y agoi like how one half of the discussion is about getting your share of pay from the deal, while the other disregards whatever is posted on the site.