4 ms·
Scraping itself isn't a problem. Scraping content where people are actively trying to stop it from being scraped is where things get questionable. both Ope
by mmcclure 3y ago
Scraping itself isn't a problem. Scraping content where people are actively trying to stop it from being scraped is where things get questionable.
both OpenAI and Google rely on scraping as their foundation for…everything.
OpenAI and Google also (claim to) respect the most basic bot deterrent, robots.txt, with their crawlers. If someone doesn't want their content scraped to be included by search indexes or LLMs/models, then that certainly includes "smaller indie hackers."