3 ms·
> The AI companies won't just scrape IA once, they're keeping come back to the same pages and scraping them over and over. Even if nothing has changed. Maybe t
by iririririr 8mo ago
> The AI companies won't just scrape IA once, they're keeping come back to the same pages and scraping them over and over. Even if nothing has changed.
Maybe they vibecoded the crawlers. I wish I were joking.
- terminalshort 8mo agoIsn't this just how crawlers work? How do you know if a page has changed if you don't keep visiting it?
- heavyset_go 8mo agoHEAD requests
- Findecanor 8mo agoScrapers could also use the "Cache-Control", "Expires" and "If-Modified-Since" headers properly, to reduce their traffic a little, but do they?