4 ms·

Sitemaps?

• mryall 16y ago
  Yeah, Google supports the sitemaps standard but that doesn't really cater for content as dynamic as Stack Overflow's. The last-updated format is a day rather than a timestamp, for example, making it useless for very-frequently updated content. http://www.sitemaps.org/protocol.php http://www.sitemaps.org/protocol.php
  • gregable 16y ago
    This is true, but the big win is being able to cull out a list of URLs that have not been updated and thus do not need recrawling.