They can and will probably respect it and still be able to have reddit results. “User-agent: * Disallow: /“ in the `robots.txt` simply instructs all web crawle

They can and will probably respect it and still be able to have reddit results. “User-agent: * Disallow: /“ in the `robots.txt` simply instructs all web crawlers not to crawl any pages on the website, not whether to show pages in search results. Of course new pages wont be shown if they cant be crawled if theres no other way you can get it.. But that line simply prevents crawlers that abide by robots.txt from retrieving the content of any pages. But Google doesnt need to crawl Reddit anymore. Reddit is directly giving the data to them to serve to users! Bing, DDG and any other search engines now are basically forced to start paying Reddit. And presumably, the Reddit execs have calculated that future revenue will be more because of this. Don’t be fooled: This isnt a “good for internet, morality” decision. Reddit is a public company now and has shareholders. They are doing this because it will equate to more $$$$