7 ms·

Related to https://news.ycombinator.com/item?id=35617763 https://news.ycombinator.com/item?id=35617763 ("Reddit Wants to Get Paid for Helping to Teach Big A.I.

Related to https://news.ycombinator.com/item?id=35617763 https://news.ycombinator.com/item?id=35617763 ("Reddit Wants to Get Paid for Helping to Teach Big A.I. Systems"; an aside, but I much prefer the title of this post I'm commenting on as it describes the actual change) and it's hard to find this particularly disagreeable. Especially considering: > Reddit’s API will remain free to developers who want to build apps and bots that help people to use Reddit, as well as to researchers who wish to study Reddit for strictly academic or noncommercial purposes. > But companies that “crawl” Reddit for data and “don’t return any of that value” to users will have to pay up,” Reddit co-founder and CEO Steve Huffman told The Times.

MuffinFlavored 4y ago
> But companies that “crawl” Reddit for data It's not possible to hide crawling at a large enough scale, right? At some point, certain IPs/user agents will (should be?) hit with CAPTCHAs to be able to have access to content and no amount of user agent/cookie/session/whatever spoofing will get around that, yeah?
- wswope 4y ago
  Cat & mouse game. If you’re defending against a whitehat business scraping with curl from data center IPs, sure. Against a less-savory actor using hundreds of IPs from residential proxies/compromised hosts, you’re gonna have a rough time, especially if you’re unwilling or unable to use aggresive fingerprinting or (vomit) CloudFlare. Not to mention CAPTCHAs are generally already a solved problem for scrapers.
  - AlphaSite 4y ago
    Require auth and it’ll help a ton.
    - Zak 4y ago
      Reddit wants people to visit the site, become interested in the content they see, and start participating regularly. That's not compatible with hiding enough content behind a registration wall to thwart sufficiently sophisticated scraping.
    - thomastjeffery 4y ago
      One of the core features of Reddit is that any person may create as many accounts as they like for free. Changing that would be incredibly disruptive. You could set a minimum karma threshold, but that would only promote karma farming; which is already widespread.
      - MuffinFlavored 4y ago
        > One of the core features of Reddit is that any person may create as many accounts as they like for free. Changing that would be incredibly disruptive. I wonder what their monthly active users look like if you filter out 1 person switching through 3 usernames/accounts for example.
        deleted 4y ago
        [deleted]
      - asdadsdad 4y ago
        reddit might be one of the few last places on the internet that hold the old times of pseudo-names and mindless anonymity. I don't see how changing that would benefit the company. see twitter
  - kayson 4y ago
    Why is cloudflare vomit-inducing?
    - wswope 4y ago
      While I appreciate it as an irreplaceable tool for countering DDoS, its premise is antithetical to a reliable and open web IMO, and it suffers from the same lack of accessible, customer-facing support as other big tech players. Lazy examples from HN algolia search: https://news.ycombinator.com/item?id=32912075 https://news.ycombinator.com/item?id=32912075 https://news.ycombinator.com/item?id=17750801 https://news.ycombinator.com/item?id=17750801 https://news.ycombinator.com/item?id=22109969 https://news.ycombinator.com/item?id=22109969 https://news.ycombinator.com/item?id=30764757 https://news.ycombinator.com/item?id=30764757 https://news.ycombinator.com/item?id=29839960 https://news.ycombinator.com/item?id=29839960 https://news.ycombinator.com/item?id=22406277 https://news.ycombinator.com/item?id=22406277 https://news.ycombinator.com/item?id=23897705 https://news.ycombinator.com/item?id=23897705 https://news.ycombinator.com/item?id=34639212 https://news.ycombinator.com/item?id=34639212 (Hypocrisy disclaimer: I have sites behind CloudFlare.)
      - gnicholas 4y ago
        Upvoted for “hypocrisy disclaimer”
    - tomwheeler 4y ago
      I intensely dislike them taking over as gatekeepers of the web. Perhaps because my browser is configured to resist fingerprinting and to avoid running arbitrary scripts from random websites, it is very frequently blocked by Cloudflare. As one example, I can no longer browse the site for Lowe's (big box home improvement chain). Consequently, I now buy everything from Home Depot (their competitor). It's astonishing how Cloudflare can do such a poor job of determining the difference between a potential customer and an attacker. Life's too short to solve captchas for an intermediary, so I don't bother, I just find a competitor who wants my business.
      - travisjungroth 4y ago
  - SheinhardtWigCo 4y ago
    Residential proxies are a completely solved problem, for companies that actually lose money to them (e.g. Ticketmaster, whose profit is maximized by blocking third-party scalpers so they can do the scalping themselves) For companies that make money by having more MAUs, well, yeah, they're going to have a real "rough time" detecting inauthentic traffic
- zer0tonin 4y ago
  It's possible but might end up more expensive (and definitely less reliable) than just paying whatever reddit asks for.
- stingraycharles 4y ago
  There are residential proxies that allow you bypass most of these things. I’ve been using them to crawl e.g. Amazon or Instagram without any issues, but they’re expensive. IIRC something like $10/GB
  - sonofhans 4y ago
    Serious question — is “residential proxies” a euphemism for “botnet?”
    - lobsterthief 4y ago
      I’m curious about this too.
      - jamesfinlayson 4y ago
        Last week someone here said that some of the big VPN players use botnets to residential IP addresses. I assumed they got residential IP addresses from ISPs but maybe not all ISPs in all parts of the world offer that.
    - bikingbismuth 4y ago
      Essentially yes. Sometimes it also includes people who have installed “free” VPNs.
    - aeyes 4y ago
      No because they are only HTTP proxies. But you don’t actually know how these companies get them, rumor is that they are part of browser extensions or free VPNs which users might install on their devices. The most “reputable” company in this space is Bright Data (formerly Luminati).
    - sleepybrett 4y ago
      Yes and also people who install certain shady "VPN" software.
    - celestialcheese 4y ago
- henryfjordan 4y ago
  There are services out there that have a large pool of consumer IPs that are marketed at crawlers for exactly this reason. A lot of them are either using hacked hardware or one of those free VPN browser plugins so it would be very hard to distinguish the traffic from a legit user.
- pocket_cheese 4y ago
  IP restrictions are easy to overcome using mobile networks. Basically, mobile networks assigns your device an internal ip and NATs out to a very small pool of ip public addresses. If they block you, they also block a very large chunk of legitimate mobile users. I'm a big ol' dummy when it comes to networking, so I imagine I explained something poorly... so any mobile network nerds feel free to pile on! Captchas are super easy! There's a gagillion captcha bypass services for every type of captcha. Just snag the captcha token, send it in an API call, and then you get a verified captcha token. See CGNAT for more details about mobile networks. https://en.wikipedia.org/wiki/Carrier-grade_NAT#cite_note-ofcom-cgnat-7 https://en.wikipedia.org/wiki/Carrier-grade_NAT#cite_note-of... It's pretty much impossible to stop the top 1% of the most dedicated scrapers without affecting end user experience.
  - tjohns 4y ago
    > IP restrictions are easy to overcome using mobile networks. Only if the connection is over IPv4. The mobile networks were among the first major adopters of IPv6, and most now give each device a unique IPv6 address.
    - abhibeckert 4y ago
      My mobile device (iPhone) relays most traffic through the nearest Akamai datacenter. So they don't get my IP address. And that datacenter has a massive number of IP addresses, which are rotated.
      - denkmoon 4y ago
        Out of interest how do you know it's being relayed through an Akamai DC? I assume you're talking about private relay which I also use, but I thought cloudflare was the 2nd hop for that?
        lathiat 4y ago
        They're using multiple providers including Fastly, Akamai and Cloudflare: https://www.streamingmediablog.com/2021/06/apple-private-relay.html https://www.streamingmediablog.com/2021/06/apple-private-rel...
- oceanplexian 4y ago
  CAPTCHAs have been broken by primitive AI for a long time (Long before GPT4-like tools). Their only purpose is to deter the lazy bots. User agents, and any other arbitrary HTTP headers, cookies, etc. have been easy to circumvent as long as the internet has existed. The only thing that sort of works is IP reputation but with IPV6 you can have as many legitimate IPs as you want. tl;dr Dedicated crawlers built by sophisticated actors are more or less impossible to defeat.
- wlesieutre 4y ago
  It's hard to imagine captchas being a workable solution as better AI models get cheaper. At this point computers are probably already better at solving them than humans are.
- gloosx 4y ago
  It is very easy and cheap to scale. 1$ for 1000 captchas solved, 10$ for 1000 proxies. Then you have 1000 users, and these are kinda impossible to distinguish from your typical common users if you cared to randomize the digital fingerprints for each client to some extent. Paid APIs for publicly accessible data are not something that makes sense or works well in this world.
  - asdadsdad 4y ago
    why would paid scraping services work then?
    - gloosx 4y ago
      You're right... I went off-lane there. It makes total sense of course since there is a demand for data, and clearly, just a minority of the people can just scrape everything at will, even if it sounds like kids play to me. And actually, it all makes sense now, since pay walling your own API is just throwing some competition to scrapers, which is totally legit. Sometimes a simple question can do a good deed, thank you:)
      - asdadsdad 4y ago
        Ha, no worries. I ask because I've also contemplated that kind of thing and realized that I was chasing my own tail perhaps.
ChuckNorris89 4y ago
>> But companies that “crawl” Reddit for data and “don’t return any of that value” to users will have to pay up,” Reddit co-founder and CEO Steve Huffman told The Times. But they do return value to users. I'd much rather get my answer from a Chat-GPT query than scouring through Reddit. Maybe he meant that they're not returning value to Reddit in which case he'd be right, but I hate him trying to spin this for the users.
- lcnPylGDnU4H9OF 4y ago
  E.g., the "remind me" bot uses Reddit's API and genuinely returns value to at least some of Reddit's users (unless it just plainly never reminds people). Comparing ChatGPT to things of that nature makes the difference more apparent to me.
- kbelder 4y ago
  >“don’t return any of that value” to users Notice that 'to users' was outside of the quote. That was an editorial addition.
  - majormajor 4y ago
    In the original NY Times article there's this line: > “Crawling Reddit, generating value and not returning any of that value to our users is something we have a problem with,” Mr. Huffman said. “It’s a good time for us to tighten things up.” So it is "to users" but more specifically it's to our users. I would agree with Huffman here: crawling the data to build ChatGPT gives the value to ChatGPT users who aren't necessarily Reddit users, and by short-circuiting queries and processes that otherwise may have led to new Reddit users, it's taking value from all Reddit users.
- kulahan 4y ago
  ChatGPT having info from reddit does not help redditors, it helps ChatGPTers. You can't really say they're providing a service for reddit users.
  - thomastjeffery 4y ago
candiddevmike 4y ago
I thought the supreme court found you can't stop folks from scraping data in the LinkedIn case? I think that applies here in some way.
- lcnPylGDnU4H9OF 4y ago
  If one scraped the content that's served to a browser when it's navigated to www.reddit.com I would expect that ruling to apply. If the API is considered a separate service, then I would imagine they could restrict access under separate terms.
  - shagie 4y ago
    With the ongoing "slowly breaking old reddit" and the move to SPA and mobile app, the data for comments and posts will be via OAuth API access rather than a server rendered html page.
- jonny_eh 4y ago
  I think you can physically block them, you can't sue them though.
- gdulli 4y ago
  As is usually the case, what they decided was something much narrower than the general case. LinkedIn still can and does make efforts to prevent/restrict automated browsing at scale. What they can't do is selectively block traffic from the plaintiff company altogether, when the content is otherwise publicly available.
  - zamnos 4y ago
    > In a November 2022 ruling the Ninth Circuit ruled that hiQ had breached LinkedIn's User Agreement and a settlement agreement was reached between the two parties. https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
  - iudqnolq 4y ago
dang 4y ago
Ok - I'm going to merge the threads but will use the more limited title on the merged post. (Edit: merging https://news.ycombinator.com/item?id=35618695 https://news.ycombinator.com/item?id=35618695 hither now)
BeFlatXIII 4y ago
Reddit is upset at all the Reddit mirrors that preserve deleted + removed comments, simple as.
- dmix 4y ago
  Oh no, will they go offline because of this? Unddit is extremely useful for seeing how mods manipulate subreddit and just general curiosity for what sort things are no longer in the overtone window this year.
  - tric 4y ago
    They rely on the pushshift api to get comments that were deleted. The other comments are pulled by the browser at the time the page is accessed. Depends on if the changes affect pushshift's crawling.
deleted 4y ago
[deleted]
boredtofears 4y ago
> Reddit’s API will remain free to developers who want to build apps and bots that help people to use Reddit I'd be fine if they didn't. The ratio of useful bots to annoying ones is very low.
- no_carrier 4y ago
  My thoughts exactly.
- faeranne 4y ago
  Bots? eh... probably gonna agree. Apps, hard no. API access to third-party apps is the only way to make competition work out in the end. IRQ, AIM, and MSN Messenger all existed around the same time, and thanks to XMPP, worked equally well on an XMPP client. This made it reasonable to use all 3 if a user wanted to, plus they could use their own server too, or a friends, or a company, or whatever. Thanks to SMTP, email is (mostly) the same way right now. On the other hand, the perfect example of how shuttering API access to apps can completely kill any competition exists right now in the form of Discord. Sure, Guilded exists, and one could argue Slack is a competitor, but tell me, do you genuinely use all three? Would they be interchangeable? Or do you split personal and professional between Discord and Slack? If all 3 had a common standard, or at least had open client api's, we'd already have a unified client, making all 3 easy to access at the same time, and we'd have good competition. Reddit has competition, and there are many third party apps that allow using all of them under one roof. Killing that off would not be a welcome change. So yeah, Reddit may not need bots, but refusing to allow apps is just pushing another nail in the coffin of competition.
  - sebazzz 4y ago
    Unfortunately the developer of the Apollo app already got a call, and apps will need to pay. That's then the end of reddit on mobile for me. The official app is unusable and had annoying behaviour in desperate attempts to boost engagement. https://old.reddit.com/r/apolloapp/comments/12ram0f/had_a_few_calls_with_reddit_today_about_the https://old.reddit.com/r/apolloapp/comments/12ram0f/had_a_fe... > There was a quote in an article about how these changes would not affect Reddit apps, that was meant in reference to “apps on the Reddit platform”, as in embedded into the Reddit service itself, not mobile apps > > tl;dr: Paid API coming.
alfiedotwtf 4y ago
> But companies that “crawl” Reddit for data and “don’t return any of that value” to users will have to pay up,” Reddit co-founder and CEO Steve Huffman told The Times Pot-Kettle! The elephant in the room but everyone is forgetting is - how much does Reddit pay its users for content? Reddit's value comes from its users, which is completely voluntarily contributed lol.
- Espressosaurus 4y ago
  Except the users are contributing, in the form of upvotes, downvotes, comments, and moderation. Reddit wouldn't exist without the work of volunteer moderators, as ripe for abuse as the positions are. Even search engines provide value in that they provide alternative search functionality.
- rafark 4y ago
  “ The elephant in the room but everyone is forgetting is - how much does Reddit pay its users for content? ” I have to say I’m not a fan of reddit, but you could also ask the question: how much do users pay to access reddit? The web has made a lot of people feel entitled to free (high quality) services. But as developers we know building and maintaining services like reddit is not cheap (let alone free).
  - ohgodplsno 4y ago
    Reddit users pay lot, considering just how many ads Reddit has on every page, as well as the arranged content promotions that regularly pop up "naturally".
prepend 4y ago
It’s funny because posts on Reddit don’t belong to Reddit, they belong to the users who created them. Why would I, as someone who’s made tens of thousands of comments, care if someone scrapes and reuses my comments. I don’t want them to pay up. This is a really rich comment from a company that relies entirely on user submitted content and has never “paid up.”
- markdown 4y ago
  > Why would I, as someone who’s made tens of thousands of comments, care if someone scrapes and reuses my comments. I don’t want them to pay up Why would I, as someone who's made tens of thousands of comments, be happy with a corporation scraping my content to create a service that they'll turn around and charge me for? I want them to pay up, so that Reddit, this wonderful service that has given me thousands of hours of entertainment and education, can be sustainable and grow. > This is a really rich comment from a company that relies entirely on user submitted content and has never “paid up.” Most redditors will agree that they get much more from Reddit than they give. I for one am very happy with the arrangement I have with Reddit.
  - mxkopy 4y ago
    Because reddit is a terrible company and you don't want to subsidize their transition into a shitty ad service.
    - lillecarl 4y ago
      If companies pay reddit for "public" data the incentive to poision the platform with too much ads decreases, and there's always adblock.
      - mxkopy 4y ago
        Having more ads and monetizing the API aren't mutually exclusive. Look at the avatar/award system for example, which evolved in tandem with dark patterns that push the user to the app where they can serve unblockable ads.

MuffinFlavored 4y ago

> But companies that “crawl” Reddit for data It's not possible to hide crawling at a large enough scale, right? At some point, certain IPs/user agents will (should be?) hit with CAPTCHAs to be able to have access to content and no amount of user agent/cookie/session/whatever spoofing will get around that, yeah?

wswope 4y ago
Cat & mouse game. If you’re defending against a whitehat business scraping with curl from data center IPs, sure. Against a less-savory actor using hundreds of IPs from residential proxies/compromised hosts, you’re gonna have a rough time, especially if you’re unwilling or unable to use aggresive fingerprinting or (vomit) CloudFlare. Not to mention CAPTCHAs are generally already a solved problem for scrapers.
- AlphaSite 4y ago
  Require auth and it’ll help a ton.
  - Zak 4y ago
    Reddit wants people to visit the site, become interested in the content they see, and start participating regularly. That's not compatible with hiding enough content behind a registration wall to thwart sufficiently sophisticated scraping.
  - thomastjeffery 4y ago
    One of the core features of Reddit is that any person may create as many accounts as they like for free. Changing that would be incredibly disruptive. You could set a minimum karma threshold, but that would only promote karma farming; which is already widespread.
    - MuffinFlavored 4y ago
      > One of the core features of Reddit is that any person may create as many accounts as they like for free. Changing that would be incredibly disruptive. I wonder what their monthly active users look like if you filter out 1 person switching through 3 usernames/accounts for example.
      - deleted 4y ago
        [deleted]
    - asdadsdad 4y ago
      reddit might be one of the few last places on the internet that hold the old times of pseudo-names and mindless anonymity. I don't see how changing that would benefit the company. see twitter
- kayson 4y ago
  Why is cloudflare vomit-inducing?
  - wswope 4y ago
    While I appreciate it as an irreplaceable tool for countering DDoS, its premise is antithetical to a reliable and open web IMO, and it suffers from the same lack of accessible, customer-facing support as other big tech players. Lazy examples from HN algolia search: https://news.ycombinator.com/item?id=32912075 https://news.ycombinator.com/item?id=32912075 https://news.ycombinator.com/item?id=17750801 https://news.ycombinator.com/item?id=17750801 https://news.ycombinator.com/item?id=22109969 https://news.ycombinator.com/item?id=22109969 https://news.ycombinator.com/item?id=30764757 https://news.ycombinator.com/item?id=30764757 https://news.ycombinator.com/item?id=29839960 https://news.ycombinator.com/item?id=29839960 https://news.ycombinator.com/item?id=22406277 https://news.ycombinator.com/item?id=22406277 https://news.ycombinator.com/item?id=23897705 https://news.ycombinator.com/item?id=23897705 https://news.ycombinator.com/item?id=34639212 https://news.ycombinator.com/item?id=34639212 (Hypocrisy disclaimer: I have sites behind CloudFlare.)
    - gnicholas 4y ago
      Upvoted for “hypocrisy disclaimer”
  - tomwheeler 4y ago
    I intensely dislike them taking over as gatekeepers of the web. Perhaps because my browser is configured to resist fingerprinting and to avoid running arbitrary scripts from random websites, it is very frequently blocked by Cloudflare. As one example, I can no longer browse the site for Lowe's (big box home improvement chain). Consequently, I now buy everything from Home Depot (their competitor). It's astonishing how Cloudflare can do such a poor job of determining the difference between a potential customer and an attacker. Life's too short to solve captchas for an intermediary, so I don't bother, I just find a competitor who wants my business.
    - travisjungroth 4y ago
- SheinhardtWigCo 4y ago
  Residential proxies are a completely solved problem, for companies that actually lose money to them (e.g. Ticketmaster, whose profit is maximized by blocking third-party scalpers so they can do the scalping themselves) For companies that make money by having more MAUs, well, yeah, they're going to have a real "rough time" detecting inauthentic traffic
zer0tonin 4y ago
It's possible but might end up more expensive (and definitely less reliable) than just paying whatever reddit asks for.
stingraycharles 4y ago
There are residential proxies that allow you bypass most of these things. I’ve been using them to crawl e.g. Amazon or Instagram without any issues, but they’re expensive. IIRC something like $10/GB
- sonofhans 4y ago
  Serious question — is “residential proxies” a euphemism for “botnet?”
  - lobsterthief 4y ago
    I’m curious about this too.
    - jamesfinlayson 4y ago
      Last week someone here said that some of the big VPN players use botnets to residential IP addresses. I assumed they got residential IP addresses from ISPs but maybe not all ISPs in all parts of the world offer that.
  - bikingbismuth 4y ago
    Essentially yes. Sometimes it also includes people who have installed “free” VPNs.
  - aeyes 4y ago
    No because they are only HTTP proxies. But you don’t actually know how these companies get them, rumor is that they are part of browser extensions or free VPNs which users might install on their devices. The most “reputable” company in this space is Bright Data (formerly Luminati).
  - sleepybrett 4y ago
    Yes and also people who install certain shady "VPN" software.
  - celestialcheese 4y ago
henryfjordan 4y ago
There are services out there that have a large pool of consumer IPs that are marketed at crawlers for exactly this reason. A lot of them are either using hacked hardware or one of those free VPN browser plugins so it would be very hard to distinguish the traffic from a legit user.
pocket_cheese 4y ago
IP restrictions are easy to overcome using mobile networks. Basically, mobile networks assigns your device an internal ip and NATs out to a very small pool of ip public addresses. If they block you, they also block a very large chunk of legitimate mobile users. I'm a big ol' dummy when it comes to networking, so I imagine I explained something poorly... so any mobile network nerds feel free to pile on! Captchas are super easy! There's a gagillion captcha bypass services for every type of captcha. Just snag the captcha token, send it in an API call, and then you get a verified captcha token. See CGNAT for more details about mobile networks. https://en.wikipedia.org/wiki/Carrier-grade_NAT#cite_note-ofcom-cgnat-7 https://en.wikipedia.org/wiki/Carrier-grade_NAT#cite_note-of... It's pretty much impossible to stop the top 1% of the most dedicated scrapers without affecting end user experience.
- tjohns 4y ago
  > IP restrictions are easy to overcome using mobile networks. Only if the connection is over IPv4. The mobile networks were among the first major adopters of IPv6, and most now give each device a unique IPv6 address.
  - abhibeckert 4y ago
    My mobile device (iPhone) relays most traffic through the nearest Akamai datacenter. So they don't get my IP address. And that datacenter has a massive number of IP addresses, which are rotated.
    - denkmoon 4y ago
      Out of interest how do you know it's being relayed through an Akamai DC? I assume you're talking about private relay which I also use, but I thought cloudflare was the 2nd hop for that?
      - lathiat 4y ago
        They're using multiple providers including Fastly, Akamai and Cloudflare: https://www.streamingmediablog.com/2021/06/apple-private-relay.html https://www.streamingmediablog.com/2021/06/apple-private-rel...
oceanplexian 4y ago
CAPTCHAs have been broken by primitive AI for a long time (Long before GPT4-like tools). Their only purpose is to deter the lazy bots. User agents, and any other arbitrary HTTP headers, cookies, etc. have been easy to circumvent as long as the internet has existed. The only thing that sort of works is IP reputation but with IPV6 you can have as many legitimate IPs as you want. tl;dr Dedicated crawlers built by sophisticated actors are more or less impossible to defeat.
wlesieutre 4y ago
It's hard to imagine captchas being a workable solution as better AI models get cheaper. At this point computers are probably already better at solving them than humans are.
gloosx 4y ago
It is very easy and cheap to scale. 1$ for 1000 captchas solved, 10$ for 1000 proxies. Then you have 1000 users, and these are kinda impossible to distinguish from your typical common users if you cared to randomize the digital fingerprints for each client to some extent. Paid APIs for publicly accessible data are not something that makes sense or works well in this world.
- asdadsdad 4y ago
  why would paid scraping services work then?
  - gloosx 4y ago
    You're right... I went off-lane there. It makes total sense of course since there is a demand for data, and clearly, just a minority of the people can just scrape everything at will, even if it sounds like kids play to me. And actually, it all makes sense now, since pay walling your own API is just throwing some competition to scrapers, which is totally legit. Sometimes a simple question can do a good deed, thank you:)
    - asdadsdad 4y ago
      Ha, no worries. I ask because I've also contemplated that kind of thing and realized that I was chasing my own tail perhaps.

ChuckNorris89 4y ago

>> But companies that “crawl” Reddit for data and “don’t return any of that value” to users will have to pay up,” Reddit co-founder and CEO Steve Huffman told The Times. But they do return value to users. I'd much rather get my answer from a Chat-GPT query than scouring through Reddit. Maybe he meant that they're not returning value to Reddit in which case he'd be right, but I hate him trying to spin this for the users.

lcnPylGDnU4H9OF 4y ago
E.g., the "remind me" bot uses Reddit's API and genuinely returns value to at least some of Reddit's users (unless it just plainly never reminds people). Comparing ChatGPT to things of that nature makes the difference more apparent to me.
kbelder 4y ago
>“don’t return any of that value” to users Notice that 'to users' was outside of the quote. That was an editorial addition.
- majormajor 4y ago
  In the original NY Times article there's this line: > “Crawling Reddit, generating value and not returning any of that value to our users is something we have a problem with,” Mr. Huffman said. “It’s a good time for us to tighten things up.” So it is "to users" but more specifically it's to our users. I would agree with Huffman here: crawling the data to build ChatGPT gives the value to ChatGPT users who aren't necessarily Reddit users, and by short-circuiting queries and processes that otherwise may have led to new Reddit users, it's taking value from all Reddit users.
kulahan 4y ago
ChatGPT having info from reddit does not help redditors, it helps ChatGPTers. You can't really say they're providing a service for reddit users.
- thomastjeffery 4y ago

candiddevmike 4y ago

I thought the supreme court found you can't stop folks from scraping data in the LinkedIn case? I think that applies here in some way.

lcnPylGDnU4H9OF 4y ago
If one scraped the content that's served to a browser when it's navigated to www.reddit.com I would expect that ruling to apply. If the API is considered a separate service, then I would imagine they could restrict access under separate terms.
- shagie 4y ago
  With the ongoing "slowly breaking old reddit" and the move to SPA and mobile app, the data for comments and posts will be via OAuth API access rather than a server rendered html page.
jonny_eh 4y ago
I think you can physically block them, you can't sue them though.
gdulli 4y ago
As is usually the case, what they decided was something much narrower than the general case. LinkedIn still can and does make efforts to prevent/restrict automated browsing at scale. What they can't do is selectively block traffic from the plaintiff company altogether, when the content is otherwise publicly available.
- zamnos 4y ago
  > In a November 2022 ruling the Ninth Circuit ruled that hiQ had breached LinkedIn's User Agreement and a settlement agreement was reached between the two parties. https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
- iudqnolq 4y ago

dang 4y ago

Ok - I'm going to merge the threads but will use the more limited title on the merged post. (Edit: merging https://news.ycombinator.com/item?id=35618695 https://news.ycombinator.com/item?id=35618695 hither now)

BeFlatXIII 4y ago

Reddit is upset at all the Reddit mirrors that preserve deleted + removed comments, simple as.

dmix 4y ago
Oh no, will they go offline because of this? Unddit is extremely useful for seeing how mods manipulate subreddit and just general curiosity for what sort things are no longer in the overtone window this year.
- tric 4y ago
  They rely on the pushshift api to get comments that were deleted. The other comments are pulled by the browser at the time the page is accessed. Depends on if the changes affect pushshift's crawling.

deleted 4y ago

[deleted]

boredtofears 4y ago

> Reddit’s API will remain free to developers who want to build apps and bots that help people to use Reddit I'd be fine if they didn't. The ratio of useful bots to annoying ones is very low.

no_carrier 4y ago
My thoughts exactly.
faeranne 4y ago
Bots? eh... probably gonna agree. Apps, hard no. API access to third-party apps is the only way to make competition work out in the end. IRQ, AIM, and MSN Messenger all existed around the same time, and thanks to XMPP, worked equally well on an XMPP client. This made it reasonable to use all 3 if a user wanted to, plus they could use their own server too, or a friends, or a company, or whatever. Thanks to SMTP, email is (mostly) the same way right now. On the other hand, the perfect example of how shuttering API access to apps can completely kill any competition exists right now in the form of Discord. Sure, Guilded exists, and one could argue Slack is a competitor, but tell me, do you genuinely use all three? Would they be interchangeable? Or do you split personal and professional between Discord and Slack? If all 3 had a common standard, or at least had open client api's, we'd already have a unified client, making all 3 easy to access at the same time, and we'd have good competition. Reddit has competition, and there are many third party apps that allow using all of them under one roof. Killing that off would not be a welcome change. So yeah, Reddit may not need bots, but refusing to allow apps is just pushing another nail in the coffin of competition.
- sebazzz 4y ago
  Unfortunately the developer of the Apollo app already got a call, and apps will need to pay. That's then the end of reddit on mobile for me. The official app is unusable and had annoying behaviour in desperate attempts to boost engagement. https://old.reddit.com/r/apolloapp/comments/12ram0f/had_a_few_calls_with_reddit_today_about_the https://old.reddit.com/r/apolloapp/comments/12ram0f/had_a_fe... > There was a quote in an article about how these changes would not affect Reddit apps, that was meant in reference to “apps on the Reddit platform”, as in embedded into the Reddit service itself, not mobile apps > > tl;dr: Paid API coming.

alfiedotwtf 4y ago

> But companies that “crawl” Reddit for data and “don’t return any of that value” to users will have to pay up,” Reddit co-founder and CEO Steve Huffman told The Times Pot-Kettle! The elephant in the room but everyone is forgetting is - how much does Reddit pay its users for content? Reddit's value comes from its users, which is completely voluntarily contributed lol.

Espressosaurus 4y ago
Except the users are contributing, in the form of upvotes, downvotes, comments, and moderation. Reddit wouldn't exist without the work of volunteer moderators, as ripe for abuse as the positions are. Even search engines provide value in that they provide alternative search functionality.
rafark 4y ago
“ The elephant in the room but everyone is forgetting is - how much does Reddit pay its users for content? ” I have to say I’m not a fan of reddit, but you could also ask the question: how much do users pay to access reddit? The web has made a lot of people feel entitled to free (high quality) services. But as developers we know building and maintaining services like reddit is not cheap (let alone free).
- ohgodplsno 4y ago
  Reddit users pay lot, considering just how many ads Reddit has on every page, as well as the arranged content promotions that regularly pop up "naturally".

prepend 4y ago

It’s funny because posts on Reddit don’t belong to Reddit, they belong to the users who created them. Why would I, as someone who’s made tens of thousands of comments, care if someone scrapes and reuses my comments. I don’t want them to pay up. This is a really rich comment from a company that relies entirely on user submitted content and has never “paid up.”

markdown 4y ago
> Why would I, as someone who’s made tens of thousands of comments, care if someone scrapes and reuses my comments. I don’t want them to pay up Why would I, as someone who's made tens of thousands of comments, be happy with a corporation scraping my content to create a service that they'll turn around and charge me for? I want them to pay up, so that Reddit, this wonderful service that has given me thousands of hours of entertainment and education, can be sustainable and grow. > This is a really rich comment from a company that relies entirely on user submitted content and has never “paid up.” Most redditors will agree that they get much more from Reddit than they give. I for one am very happy with the arrangement I have with Reddit.
- mxkopy 4y ago
  Because reddit is a terrible company and you don't want to subsidize their transition into a shitty ad service.
  - lillecarl 4y ago
    If companies pay reddit for "public" data the incentive to poision the platform with too much ads decreases, and there's always adblock.
    - mxkopy 4y ago
      Having more ads and monetizing the API aren't mutually exclusive. Look at the avatar/award system for example, which evolved in tandem with dark patterns that push the user to the app where they can serve unblockable ads.