3 ms·

> For decades, robots.txt governed the behavior of web crawlers. It never governed anything, web crawlers were never under any obligation to follow robots.txt.

• fallingknife 3y ago
  > they took an existing controversy, rebranded it as something new, then blamed in on <new thing> Basically describes most of what passes as journalism these days.
• franze 3y ago
  > web crawlers were never under any obligation to follow robots.txt. other than the fact that you could get successful sued if you do not follow them i.e.: eBay v. Bidder's Edge in 2000
  • nadermx 3y ago
    That was overturned, https://en.wikipedia.org/wiki/Intel_Corp._v._Hamidi https://en.wikipedia.org/wiki/Intel_Corp._v._Hamidi and White Buffalo Ventures LLC v. University of Texas at Austin
  • geor9e 3y ago
    The Internet Archive never follows robots.txt and they're still around
• janalsncm 3y ago
  No, I think this is something new. It’s an existential problem for newspapers. Most newspapers have never thrived online but they got a stream of clicks and a few 100x stories every once in a while. But with language models and RAG, we may create a world where no one has to visit their site at all. This really hit home for me in Hard Fork’s interview with the CEO of Perplexity AI [1]. The crux of the problem is, if Perplexity does what it strives to do, provide answers rather than links, no one will be clicking on links and generating revenue for newspapers anymore. I encourage you to watch him struggle with this question because this is really an irreconcilable problem. [1] https://m.youtube.com/watch?v=AJuv_UwxELA https://m.youtube.com/watch?v=AJuv_UwxELA