3 ms·

robots.txt isn't legally binding. I am interested to know if and how services even interact with it. It's more like a clue on when the interesting content for s

robots.txt isn't legally binding. I am interested to know if and how services even interact with it. It's more like a clue on when the interesting content for scrapers is on your site. This is how I imagine it goes: "Hey, don't scrape the data here." "You know what? I'm scrape it even harder!"

bbor 2y ago
Soooo nonconsensual. Maybe bluesky is right… are we the baddies?
tonyhart7 2y ago
it is legally binding if your company based on SV (only California implement this law) and they can prove it