3 ms·

Yup. It's very hard to avoid getting stuck in accidental tar pits. One page with inexhaustible URL variations. http://example.com/article/story?storyID=19039

by staunch 14y ago

Yup. It's very hard to avoid getting stuck in accidental tar pits. One page with inexhaustible URL variations. http://example.com/article/story?storyID=19039&ref=329932&sessionID=9043275 http://example.com/article/story?storyID=19039&ref=902932&sessionID=9023409 http://example.com/article/story?storyID=19039&ref=904354&sessionID=8230235 ... infinite ... Canonical URL meta tags can help. Filtering out certain query parameters (JSESSIONID, PHPSESSIONID, etc) can help. Sitemaps help. I'm rather impressed that search engines do it so well. I imagine the right approach involves examining the contents of the pages and doing checksums, but I'd love to know what the real search engines do. I do know that a naive crawler will completely fail to crawl any significant portion of the real web without solving this. It's quite possible those 250 million "pages" were actually something like 1 million distinct pages.

TazeTSchnitzel 14y ago
Also, some sites have a LOT of generated pages for a tiny section. For instance, crawling my GitHub account with my homemade crawler was impossible. There is an amazing level of depth there.
hntester123 14y ago
>I'm rather impressed that search engines do it so well. I imagine the right approach involves examining the contents of the pages and doing checksums Interesting point. Q: Is it possible for two different pages to give the same checksum? (Asking for my own info; I do know what checksums are, in overview, from way back, but haven't checked :) them out much.
- josephlord 14y ago
  Given two different pages getting the same checksum should be VERY unlikely. Getting a collision among billions of pages is fairly or very likely depending on the type of checksum. So it is likely to be important to only compare within domains or divide the space by some method. Also it doesn't help you at all because pages returned that you want to treat as the same aren't actually the same because they include the request time or other unique element. Edit: Added sentence about comparing with domains.
  - mrkrwtsn 14y ago
    You could use an edit distance algorithm instead of using checksums. Although that would be really time intensive.

by staunch 14y ago

TazeTSchnitzel 14y ago

Also, some sites have a LOT of generated pages for a tiny section. For instance, crawling my GitHub account with my homemade crawler was impossible. There is an amazing level of depth there.

hntester123 14y ago

>I'm rather impressed that search engines do it so well. I imagine the right approach involves examining the contents of the pages and doing checksums Interesting point. Q: Is it possible for two different pages to give the same checksum? (Asking for my own info; I do know what checksums are, in overview, from way back, but haven't checked :) them out much.

josephlord 14y ago
Given two different pages getting the same checksum should be VERY unlikely. Getting a collision among billions of pages is fairly or very likely depending on the type of checksum. So it is likely to be important to only compare within domains or divide the space by some method. Also it doesn't help you at all because pages returned that you want to treat as the same aren't actually the same because they include the request time or other unique element. Edit: Added sentence about comparing with domains.
- mrkrwtsn 14y ago
  You could use an edit distance algorithm instead of using checksums. Although that would be really time intensive.