5 ms·
> How can one automatically determine whether a page is unique or static? They crawled it for years and it never changed? It is a blog post.
by JetSpiegel 8y ago
> How can one automatically determine whether a page is unique or static?
They crawled it for years and it never changed? It is a blog post.
- zamadatix 8y agoThe person you are replying to said "unique immutable pages", by definition you would be able to crawl these for years and they would never change. [1] is a site that contains all possible 3200 page books with the ability to consistently index content as an example. [1] http://libraryofbabel.info/About.html http://libraryofbabel.info/About.html