3 ms·

But of course search engines are fine

• wraptile 3y ago
  Having your cake and eating it too is a natural goal of every business and honestly it was just a matter of time till web pages figured out they can have the benefits of public data and avoid the costs. Web scraping and botting is basically a solved problem too - just put a login gate for the data which allows you to legally litigate against scrapers and bots. Done. However, nobody wants to lose the benefits of public data so here we are.
  • CalRobert 3y ago
    I used to care about respecting robots.txt until it was clear that established search engines are fine but any newcomers can go right to hell.