12 ms·

Do these services respect norobot manifests? Isn't this all kinda... illegal...? Or at least non-consensual?

• basilgohar 2y ago
  robots.txt isn't legally binding. I am interested to know if and how services even interact with it. It's more like a clue on when the interesting content for scrapers is on your site. This is how I imagine it goes: "Hey, don't scrape the data here." "You know what? I'm scrape it even harder!"
  • bbor 2y ago
    Soooo nonconsensual. Maybe bluesky is right… are we the baddies?
  • tonyhart7 2y ago
    it is legally binding if your company based on SV (only California implement this law) and they can prove it
• fc417fc802 2y ago
  [dead]