MOMSpider was the one I remember. What's interesting about web crawling is that it has always been controversial. Many website operators back then did not lik

MOMSpider was the one I remember. What's interesting about web crawling is that it has always been controversial. Many website operators back then did not like crawlers. And that sort of sentiment continues to this day. The web has never fully consented to being indexed. This has always struck me as odd. Perhaps there should have been more thought about a bifurcation of "private" web and "public" web. Of course in retrospect the value of the public one seems far greater; Google's corporate wealth points to that conclusion.