3 ms·
Isn't this just how crawlers work? How do you know if a page has changed if you don't keep visiting it?
by terminalshort 8mo ago
Isn't this just how crawlers work? How do you know if a page has changed if you don't keep visiting it?
- heavyset_go 8mo agoHEAD requests
- Findecanor 8mo agoScrapers could also use the "Cache-Control", "Expires" and "If-Modified-Since" headers properly, to reduce their traffic a little, but do they?