3 ms·

everyone with limited bandwidth has been trying to limit site access to robots. the latest generation of AI web scrapers are brutal and do not respect robots.tx

• varispeed 2y ago
  There are websites where you can only register to in person and have two existing members vouch for you. Probably still can be gamed, but sounds like a great barrier to entry for robots (for now).
  • tmpz22 2y ago
    What prevents someone from getting access and then running an authenticated headless browser to scoop the data?
    • varispeed 2y ago
      Admins will see unusual traffic from that account and then take action. Of course it will not be perfect as there could be a way to mimic human traffic and slowly scrape the data anyway, that's why there is element of trust (two existing members to vouch).
      • tmpz22 2y ago
        Yeah don’t get me wrong I believe raising the burden of extraction is an effective strategy I just think it’s been solved at scale ie voting rings and astro turfing operations on Reddit - and at the nation state level I’d just bribe or extort the mods and admins directly (or the IT person to dump the database).
        • varispeed 2y ago
          That's entirely possible, especially if the site is small and not run by people with access to resources like physical security, legal etc.