3 ms·
> If you put your content on the web, and don't require authentication to access it, it's going to be crawled and scraped. Most of the time you're happy about t
by fzeroracer 2y ago
> If you put your content on the web, and don't require authentication to access it, it's going to be crawled and scraped. Most of the time you're happy about this — you want search providers to index your content
> The default-public accessibility of information on the internet is a net-good for the technology ecosystem. Want to host things online? Learn how.
These two statements are at odds, I hope you realize. You say public accessibility of information is a good thing, while blaming someone for being effectively DDOS'd as a result of having said information public.
- hd4 2y agoThey're not at odds. "default-public accessibility of information" doesn't necessarily translate into "default-public accessibility of content" ie. media. Content should be served behind an authentication layer. The clickbaity hysteria here is missing out how this sort of scraping has been possible long before AI agents showed up a couple of years back.
- macintux 2y agoOf course it was possible, but the incentives have changed. Now anyone can use the accumulated knowledge of the world to build something new, so more independent actors are doing so, often very badly.