4 ms·

They changed robots.txt a month or so ago. For the first 19 years of life, reddit had a very permissive robots.txt. We allowed all by default and then only re

They changed robots.txt a month or so ago. For the first 19 years of life, reddit had a very permissive robots.txt. We allowed all by default and then only restricted certain poorly behaved agents (and Bender's Shiny Metal Ass(tm)) But I can understand why they made the change they did. The data was being abused. My guess is that this was an oversight -- that they will do an audit and reopen it for search engines after those engines agree not to use the data for training, because let's face it, reddit is a for profit business and they have to protect their income streams.

JohnMakin 2y ago
One (in this case, 2) company's incentive for profit should not take priority over the usability/well being of the internet as a whole, ever, and is exactly why we are where we are now. This is an absolutely terrible precedent.
- jedberg 2y ago
  I agree with you in theory, but in practice someone has to pay for all this magic.
  - JohnMakin 2y ago
    This is a false dichotomy. You can have services, and not have them devolve into complete unusability in the name of profit. This isn’t sustainable either. The myopic pursuit of short term gains at the expense of the product will collapse at some point in the future, no matter how much you believe in this weird frog-boil internet we’ve inherited now.
    - talldayo 2y ago
      > The myopic pursuit of short term gains at the expense of the product will collapse at some point in the future, The myopic pursuit of short-term gains is the only playbook that works. Long-term business strategy is a gamble, and today's businesses have all learned that they'd rather make hay when the sun is shining than be remembered as a good business. Twitter tried a long-term playbook to reverse their unprofitable sinkhole of a website. That ended up with them being undervalued and sold to the highest bidder.
      - latexr 2y ago
        > Twitter tried a long-term playbook to reverse their unprofitable sinkhole of a website. From what I recall reading at the time, Twitter was finally becoming profitable before the sale (last two years? It’s hard to find a source now since every story since is about some shit show or other post sale). > That ended up with them being undervalued and sold to the highest bidder. You make it seem they were in dire straits and had to be sold for scraps, but that’s far from the case. They sold for more than their valuation to the only bidder because they understood what a good deal it was for them. They forced the buyer to not back out, after all.
    - twelve40 2y ago
      Complete unusability is when ai tools clone the content and people stop visiting the original service and participating. I'll leave it up to them to defend blocking duck duck go for example, but blocking "AI" bots for an online community is a matter of survival at this point.
      - talldayo 2y ago
        Alternatively, it's because the base platform has also devolved into unusability. Both Reddit and Twitter are in a position where their info is easily scraped, and their community is barely worth the advertising/paid-premium experience they demand from you. As both platforms continue to decline in quality, you might not even need to replace the original service. Both businesses appear intent on getting replaced.
  - ToucanLoucan 2y ago
    We did. As in we, the Internet, existed for a long time without anyone making money and we paid for the privilege. Websites were built and hosted at owner's expense, for years, with no expectation that they be financially rewarded. Sure some would run donation drives, or work with sponsors relevant to the community in question, but a whole ton, mine included, just cost me a lot of money over many years. Those websites were definitely technically inferior, as the march of progress is unavoidable, but web hosting is cheaper than it's ever been. A VPS that utterly blows away what mine was capable of in 2007 for nearly a hundred a month can now be had for about $10 per month. Yet everyone wants these monolith platforms, but even that wouldn't be the worst thing ever, except that every one of these platforms has a backend to support that we in the Old Internet never did: a C-suite's worth of executives and millions of shareholders, who for some reason have decided that reddit can't exist unless reddit makes them reams and reams of money. I'd be very, very interested to see how much of, even what's probably the most massive one of all, Facebook, is non-essential busywork that could easily be shut down tomorrow with no adverse effects to the platform. Firstly the entire executive class, just, they don't do shit to make Facebook the product. In fact I'd argue their decisions almost universally have made it worse as a product very consistently for it's entire lifetime. Then, all the marketing people. There's just no goddamn reason to advertise Facebook (or reddit for that matter) the brand is so ubiquitous, if you actually found someone who'd never heard of it, I'd give you a large chunk of money. Add to that, if Facebook was doing a good job of being what it ostensibly is, then people immediately become the best advertising, because people want to hang with people in these digital spaces. Then get rid of the people working to make Facebook addictive with dark patterns. Then get rid of the entire targeted ad division, because it's gross and inhumane. Pare the company down to engineers who build the product, and if anything, expand the moderation team so they can actually ensure the safety of the platform, and of course the IT staff to back them. Now what does Facebook cost to operate? As far as I'm concerned, this pearl-clutching about "well websites have to make money" is grossly, grossly overstated. Websites don't cost that much to run. A ton of money is being siphoned off by the MBA parasites playing games in Excel all day. A ton more is being wasted developing features that advertisers want and users hate. A ton more is being funneled into making products artificially addictive to vulnerable people, to exploit them, so let's just not do that. And of course, leadership, rewarding themselves with generous compensation packages they aren't even remotely able to justify. Now what does your website cost to maintain? Surely not nothing, and for websites of substantial size, it will still be high, but I'm willing to bet it's a hell, hell, hell of a lot less than it was before.
  - meiraleal 2y ago
    how can we keep paying the ever-growing profits of multi-trillion dollar companies? This is insane.
    - jsnell 2y ago
      Reddit is 100x from being a trillion-dollar company, and is not profitable.
      - meiraleal 2y ago
        Reddit offers no magic is just a forum. Google used to do some magic decades ago and still profit from it.
  - account42 2y ago
    People were paying for forums before Reddit came along.
- BeetleB 2y ago
  I know people will hate to hear this, but Reddit it's not important to the A well being of the Internet.
  - TeaBrain 2y ago
    I think it's the other way around, in that people don't like to hear how Reddit has become important due to the death of independent forums and the degree to which information has become concentrated on the site.
    - BeetleB 2y ago
      The death of independent forums has been greatly exaggerated. Of all the forums I used to be active in, many are still active. The ones that died did so because the community died (i.e. they did not shift to Reddit and the like). Reddit is great simply because it allowed anyone to create a community. No need to get a LAMP stack and deal with security vulnerabilities in your forum SW. These days you have Lemmy and its ilk. Much higher barrier than the old LAMP stack, but also much superior to it. I do hope it takes off.
    - latexr 2y ago
      Independent forums, like RSS, are not dead. I use both every day.
  - account42 2y ago
    GP was not claiming that it is.
ColinHayhurst 2y ago
Person extensively quoted in the article here. They are welcome to reach out. But not a single person from any level did that, nor replied to my polite requests to explain and engage. We first contacted them in early June and by 13th June, I had escalated to Steve Huffman @spez.
- toomuchtodo 2y ago
  An acquaintance investigating Reddit's moderation mechanization inquired how a major subreddit was moderated after an Associated Press post was auto removed by automod. They were banned from said sub. They inquired why they were banned, and they shared they would share any responses with a journalism org (to be transparent where any replies would be going, because they are going to a journalism org). They were muted by mods for 28 days and were "told off" in a very poor manner (per the screenshots I've seen) by the anonymous mod who replied to them. They were then banned from Reddit for 3 days after an appeal for "harassment"; when they requested more info about what was considered harassment, they were ignored. Ergo, inquiring as to how the mods of a major sub are automodding non-biased journalism sources (the AP, in this case) without any transparency appears to be considered harassment by Reddit. The interaction was submitted to the FTC through their complaint system to contribute towards their existing antitrust investigation of Reddit. Shared because it is unlikely Reddit responds except when required by law, so I recommend engaging regulators (FTC, and DOJ at the bare minimum) and legislators (primarily those focused on Section 230 reforms) whenever possible with regards to this entity. They're the only folks worth escalating to, as Reddit's incentives are to gate content, keep ad buyers happy, and keep the user base in check while they struggle to break even, sharing as little information publicly as possible along the way [1] [2]. [1] https://www.bloomberg.com/news/articles/2024-05-09/reddit-lays-out-content-policy-while-seeking-more-licensing-deals https://www.bloomberg.com/news/articles/2024-05-09/reddit-la... | https://archive.today/wQuKM https://archive.today/wQuKM [2] https://www.sec.gov/edgar/browse/?CIK=1713445 https://www.sec.gov/edgar/browse/?CIK=1713445
  - account42 2y ago
    > non-biased journalism sources No such thing, and definitely not the AP.
fredgrott 2y ago
the article quotes reddit policy change: Reddit considers search and ads commercial activities and thus subject to robot.txt block and exclusion.
ColinHayhurst 2y ago
The blocks for MojeekBot, as Cloudflare verified and respectful bot for 20 years, started before the robots.txt file changes. We first noticed in early June. We thought it was an oversight too at first. It usually is. Large publishers have blocked us when they have not considered the details, but then reinstated us when we got in touch and explained.
Closi 2y ago
> But I can understand why they made the change they did. The data was being abused. Depends how you see it - if you see it as 'their' data (legally true) or if you see it as user content (how their users would likely see it). If you see it as 'user content', they are actually selling the data to be abused by one company, rather than stopping it being abused at all. From a commercial 'lets sell user data and make a profit' perspective I get it, although does seem short-sighted to decide to effectively de-list yourself from alternative search engines (guess they just got enough cash to make it worth their while).
- Ajedi32 2y ago
  > if you see it as 'their' data (legally true) Is that actually true? Reddit may indeed have a license to use that data (derived from their ToS), but I very much doubt they actually own the copyright to it. If I write a comment on Reddit, then copy-paste it somewhere else, can Reddit sue me for copyright infringement?
  - jedberg 2y ago
    They own a non-exclusive worldwide right to it. You own the copyright, they have a license to use it however they see fit.
    - account42 2y ago
      > They own a non-exclusive worldwide right to it. They demand that right. That doesn't mean they actually have a right to use the content in ways that are not directly required for the operation of the website or that are otherwise surprising to the average user. Putting something in the TOS doesn't always make it a valid contract.
- passwordoops 2y ago
  Enough cash or enough data on hand to show the majority of traffic comes from the search monopoly
ekidd 2y ago
I personally feel that this kind of "exclusive search only by Google deal" should result in an anti-trust case against Google. This is the kind of abuse of monopoly power that caused anti-trust laws to be passed in the 1890s.
- eddd-ddde 2y ago
  if i create a vacuum cleaner and decide to only sell it at Walmart you can't get mad at me for not wanting to sell it at costco you can always buy a competitor's or make your own vacuum cleaner if you hate buying at Walmart maybe what you are really mad about is Reddit monopolising content
  - ekidd 2y ago
    Usually, to trigger any kind of anti-trust law, you need to have massive market share. In this case, for example, Reddit almost certainly hasn't committed any antitrust violations, because they're a relatively minor player in their market. Similarly, if you start a vacuum cleaner company, you can make whatever exclusive deals you want. But if you control 80% of the market for vacuum cleaners, then you might need to be more careful about leveraging your market share in unfair ways. If a company is part of a robust, competitive market (like Reddit), it's usually wiser to let customers vote with their wallets, and leave the government out of it. If a company becomes massively dominant (like Google or TicketMaster), and if it starts pushing exclusive contracts, it's much harder for customers to switch away.
  - ffgjgf1 2y ago
    Unless you’re deemed to be an unfair monopoly and abuse your position to harm consumers interests. You don’t even need >90% market share for that to be the case. e.g. Standard Oil only controlled 64% of the US market at most, it was still broken upz
EasyMark 2y ago
how was it being abused. You still clicked on the information and saw the reddit ads? Now they won't get any of that from "rival" sites to google. I guess they figured the 60 million was more than that ad revenue. Seems greedy but I don't think it's illegal like others are suggesting.
account42 2y ago
Ah so when reddit uses user content for monetization it's ok but when others do it then it isn't? Reddit may want that double standard but I think the only thing they are going to achieve with this stunt is more people ignoring robots.txt.

JohnMakin 2y ago

One (in this case, 2) company's incentive for profit should not take priority over the usability/well being of the internet as a whole, ever, and is exactly why we are where we are now. This is an absolutely terrible precedent.

jedberg 2y ago
I agree with you in theory, but in practice someone has to pay for all this magic.
- JohnMakin 2y ago
  This is a false dichotomy. You can have services, and not have them devolve into complete unusability in the name of profit. This isn’t sustainable either. The myopic pursuit of short term gains at the expense of the product will collapse at some point in the future, no matter how much you believe in this weird frog-boil internet we’ve inherited now.
  - talldayo 2y ago
    > The myopic pursuit of short term gains at the expense of the product will collapse at some point in the future, The myopic pursuit of short-term gains is the only playbook that works. Long-term business strategy is a gamble, and today's businesses have all learned that they'd rather make hay when the sun is shining than be remembered as a good business. Twitter tried a long-term playbook to reverse their unprofitable sinkhole of a website. That ended up with them being undervalued and sold to the highest bidder.
    - latexr 2y ago
      > Twitter tried a long-term playbook to reverse their unprofitable sinkhole of a website. From what I recall reading at the time, Twitter was finally becoming profitable before the sale (last two years? It’s hard to find a source now since every story since is about some shit show or other post sale). > That ended up with them being undervalued and sold to the highest bidder. You make it seem they were in dire straits and had to be sold for scraps, but that’s far from the case. They sold for more than their valuation to the only bidder because they understood what a good deal it was for them. They forced the buyer to not back out, after all.
  - twelve40 2y ago
    Complete unusability is when ai tools clone the content and people stop visiting the original service and participating. I'll leave it up to them to defend blocking duck duck go for example, but blocking "AI" bots for an online community is a matter of survival at this point.
    - talldayo 2y ago
      Alternatively, it's because the base platform has also devolved into unusability. Both Reddit and Twitter are in a position where their info is easily scraped, and their community is barely worth the advertising/paid-premium experience they demand from you. As both platforms continue to decline in quality, you might not even need to replace the original service. Both businesses appear intent on getting replaced.
- ToucanLoucan 2y ago
  We did. As in we, the Internet, existed for a long time without anyone making money and we paid for the privilege. Websites were built and hosted at owner's expense, for years, with no expectation that they be financially rewarded. Sure some would run donation drives, or work with sponsors relevant to the community in question, but a whole ton, mine included, just cost me a lot of money over many years. Those websites were definitely technically inferior, as the march of progress is unavoidable, but web hosting is cheaper than it's ever been. A VPS that utterly blows away what mine was capable of in 2007 for nearly a hundred a month can now be had for about $10 per month. Yet everyone wants these monolith platforms, but even that wouldn't be the worst thing ever, except that every one of these platforms has a backend to support that we in the Old Internet never did: a C-suite's worth of executives and millions of shareholders, who for some reason have decided that reddit can't exist unless reddit makes them reams and reams of money. I'd be very, very interested to see how much of, even what's probably the most massive one of all, Facebook, is non-essential busywork that could easily be shut down tomorrow with no adverse effects to the platform. Firstly the entire executive class, just, they don't do shit to make Facebook the product. In fact I'd argue their decisions almost universally have made it worse as a product very consistently for it's entire lifetime. Then, all the marketing people. There's just no goddamn reason to advertise Facebook (or reddit for that matter) the brand is so ubiquitous, if you actually found someone who'd never heard of it, I'd give you a large chunk of money. Add to that, if Facebook was doing a good job of being what it ostensibly is, then people immediately become the best advertising, because people want to hang with people in these digital spaces. Then get rid of the people working to make Facebook addictive with dark patterns. Then get rid of the entire targeted ad division, because it's gross and inhumane. Pare the company down to engineers who build the product, and if anything, expand the moderation team so they can actually ensure the safety of the platform, and of course the IT staff to back them. Now what does Facebook cost to operate? As far as I'm concerned, this pearl-clutching about "well websites have to make money" is grossly, grossly overstated. Websites don't cost that much to run. A ton of money is being siphoned off by the MBA parasites playing games in Excel all day. A ton more is being wasted developing features that advertisers want and users hate. A ton more is being funneled into making products artificially addictive to vulnerable people, to exploit them, so let's just not do that. And of course, leadership, rewarding themselves with generous compensation packages they aren't even remotely able to justify. Now what does your website cost to maintain? Surely not nothing, and for websites of substantial size, it will still be high, but I'm willing to bet it's a hell, hell, hell of a lot less than it was before.
- meiraleal 2y ago
  how can we keep paying the ever-growing profits of multi-trillion dollar companies? This is insane.
  - jsnell 2y ago
    Reddit is 100x from being a trillion-dollar company, and is not profitable.
    - meiraleal 2y ago
      Reddit offers no magic is just a forum. Google used to do some magic decades ago and still profit from it.
- account42 2y ago
  People were paying for forums before Reddit came along.
BeetleB 2y ago
I know people will hate to hear this, but Reddit it's not important to the A well being of the Internet.
- TeaBrain 2y ago
  I think it's the other way around, in that people don't like to hear how Reddit has become important due to the death of independent forums and the degree to which information has become concentrated on the site.
  - BeetleB 2y ago
    The death of independent forums has been greatly exaggerated. Of all the forums I used to be active in, many are still active. The ones that died did so because the community died (i.e. they did not shift to Reddit and the like). Reddit is great simply because it allowed anyone to create a community. No need to get a LAMP stack and deal with security vulnerabilities in your forum SW. These days you have Lemmy and its ilk. Much higher barrier than the old LAMP stack, but also much superior to it. I do hope it takes off.
  - latexr 2y ago
    Independent forums, like RSS, are not dead. I use both every day.
- account42 2y ago
  GP was not claiming that it is.

ColinHayhurst 2y ago

Person extensively quoted in the article here. They are welcome to reach out. But not a single person from any level did that, nor replied to my polite requests to explain and engage. We first contacted them in early June and by 13th June, I had escalated to Steve Huffman @spez.

toomuchtodo 2y ago
An acquaintance investigating Reddit's moderation mechanization inquired how a major subreddit was moderated after an Associated Press post was auto removed by automod. They were banned from said sub. They inquired why they were banned, and they shared they would share any responses with a journalism org (to be transparent where any replies would be going, because they are going to a journalism org). They were muted by mods for 28 days and were "told off" in a very poor manner (per the screenshots I've seen) by the anonymous mod who replied to them. They were then banned from Reddit for 3 days after an appeal for "harassment"; when they requested more info about what was considered harassment, they were ignored. Ergo, inquiring as to how the mods of a major sub are automodding non-biased journalism sources (the AP, in this case) without any transparency appears to be considered harassment by Reddit. The interaction was submitted to the FTC through their complaint system to contribute towards their existing antitrust investigation of Reddit. Shared because it is unlikely Reddit responds except when required by law, so I recommend engaging regulators (FTC, and DOJ at the bare minimum) and legislators (primarily those focused on Section 230 reforms) whenever possible with regards to this entity. They're the only folks worth escalating to, as Reddit's incentives are to gate content, keep ad buyers happy, and keep the user base in check while they struggle to break even, sharing as little information publicly as possible along the way [1] [2]. [1] https://www.bloomberg.com/news/articles/2024-05-09/reddit-lays-out-content-policy-while-seeking-more-licensing-deals https://www.bloomberg.com/news/articles/2024-05-09/reddit-la... | https://archive.today/wQuKM https://archive.today/wQuKM [2] https://www.sec.gov/edgar/browse/?CIK=1713445 https://www.sec.gov/edgar/browse/?CIK=1713445
- account42 2y ago
  > non-biased journalism sources No such thing, and definitely not the AP.

fredgrott 2y ago

the article quotes reddit policy change: Reddit considers search and ads commercial activities and thus subject to robot.txt block and exclusion.

ColinHayhurst 2y ago

The blocks for MojeekBot, as Cloudflare verified and respectful bot for 20 years, started before the robots.txt file changes. We first noticed in early June. We thought it was an oversight too at first. It usually is. Large publishers have blocked us when they have not considered the details, but then reinstated us when we got in touch and explained.

Closi 2y ago

> But I can understand why they made the change they did. The data was being abused. Depends how you see it - if you see it as 'their' data (legally true) or if you see it as user content (how their users would likely see it). If you see it as 'user content', they are actually selling the data to be abused by one company, rather than stopping it being abused at all. From a commercial 'lets sell user data and make a profit' perspective I get it, although does seem short-sighted to decide to effectively de-list yourself from alternative search engines (guess they just got enough cash to make it worth their while).

Ajedi32 2y ago
> if you see it as 'their' data (legally true) Is that actually true? Reddit may indeed have a license to use that data (derived from their ToS), but I very much doubt they actually own the copyright to it. If I write a comment on Reddit, then copy-paste it somewhere else, can Reddit sue me for copyright infringement?
- jedberg 2y ago
  They own a non-exclusive worldwide right to it. You own the copyright, they have a license to use it however they see fit.
  - account42 2y ago
    > They own a non-exclusive worldwide right to it. They demand that right. That doesn't mean they actually have a right to use the content in ways that are not directly required for the operation of the website or that are otherwise surprising to the average user. Putting something in the TOS doesn't always make it a valid contract.
passwordoops 2y ago
Enough cash or enough data on hand to show the majority of traffic comes from the search monopoly

ekidd 2y ago

I personally feel that this kind of "exclusive search only by Google deal" should result in an anti-trust case against Google. This is the kind of abuse of monopoly power that caused anti-trust laws to be passed in the 1890s.

eddd-ddde 2y ago
if i create a vacuum cleaner and decide to only sell it at Walmart you can't get mad at me for not wanting to sell it at costco you can always buy a competitor's or make your own vacuum cleaner if you hate buying at Walmart maybe what you are really mad about is Reddit monopolising content
- ekidd 2y ago
  Usually, to trigger any kind of anti-trust law, you need to have massive market share. In this case, for example, Reddit almost certainly hasn't committed any antitrust violations, because they're a relatively minor player in their market. Similarly, if you start a vacuum cleaner company, you can make whatever exclusive deals you want. But if you control 80% of the market for vacuum cleaners, then you might need to be more careful about leveraging your market share in unfair ways. If a company is part of a robust, competitive market (like Reddit), it's usually wiser to let customers vote with their wallets, and leave the government out of it. If a company becomes massively dominant (like Google or TicketMaster), and if it starts pushing exclusive contracts, it's much harder for customers to switch away.
- ffgjgf1 2y ago
  Unless you’re deemed to be an unfair monopoly and abuse your position to harm consumers interests. You don’t even need >90% market share for that to be the case. e.g. Standard Oil only controlled 64% of the US market at most, it was still broken upz

EasyMark 2y ago

how was it being abused. You still clicked on the information and saw the reddit ads? Now they won't get any of that from "rival" sites to google. I guess they figured the 60 million was more than that ad revenue. Seems greedy but I don't think it's illegal like others are suggesting.

account42 2y ago

Ah so when reddit uses user content for monetization it's ok but when others do it then it isn't? Reddit may want that double standard but I think the only thing they are going to achieve with this stunt is more people ignoring robots.txt.