3 ms·
Suppose a crawler or rival search engine doesn’t respect robots.txt, reddit can’t stop them. Make it a bit trickier, yes, but not stop them.
by nomilk 2y ago
Suppose a crawler or rival search engine doesn’t respect robots.txt, reddit can’t stop them. Make it a bit trickier, yes, but not stop them.
- eschneider 2y agoIt is evidence that they didn't have permission if you sue them.
- kingnothing 2y agoThere's no grounds on which to file suit. The 9th circuit court found web scraping is legal. https://techcrunch.com/2022/04/18/web-scraping-legal-court/ https://techcrunch.com/2022/04/18/web-scraping-legal-court/
- tagawa 2y agoThis is not even scraping - it’s just crawling and indexing.
- miyuru 2y agoreddit blocked datacenter IPs even before this change.
- nomilk 2y agoCould a motivated scraper not buy IPs/proxies that aren’t in those ranges, i.e. to blend in with general users?
- xeromal 2y agoJust like every security feature in the physical and digital worlds, security just inconveniences honest people and the cost to bypass reduces the amount of people who try. Eventually it becomes expensive to scrape reddit's data and most people will stop.
- Manuel_D 2y agoProxy IPs are also known and typically blocked. In fact, you can't even browse reddit without logging in when connected to most proxies. Many web scraping companies have loads of phones hooked up in a rack in order to use mobile IPs. Companies can't just block mobile IPs because their site would become unusable for several city blocks (mobile IPs often correspond to a specific cell tower). This is the face of modern web scraping: https://i.imgur.com/U2RXi5G.jpeg https://i.imgur.com/U2RXi5G.jpeg
- BeFlatXIII 2y agoOr, if you're on the shadier parts of the web, hacked IoT devices put to good use instead of DDoS zombies.