4 ms·
If the content is literally the same, the crawler should be able to use If-Modified-Since, right? It still has to make a HTTP request, but not parse or index an
by strken 3y ago
If the content is literally the same, the crawler should be able to use If-Modified-Since, right? It still has to make a HTTP request, but not parse or index anything.
- codedokode 3y agoIf the content is dynamic (e.g. a list of popular articles in a sidebar has changed), then the page will be considered "updated".
- wise_young_man 3y agoThis is not correct. It’s up to the server, controlled by the application to send that or other headers. Similar to sending a <title> tag. The headers take priority and similar to what another person said they will do a HEAD request first and not bother with a GET request for the content.