Great document! I architected and implemented a crawler, reading this documents really confirmed problems encountered and taught me a thing or two on how to im

Great document! I architected and implemented a crawler, reading this documents really confirmed problems encountered and taught me a thing or two on how to improve. This document goes the same scope as: Why Writing Your Own Search Engine is Hard - by Anna Patterson, Stanford University (http://www.acmqueue.org/modules.php?name=Content&pa=showpage&pid=143 http://www.acmqueue.org/modules.php?name=Content&pa=show...) for those interested in search engines.