I think your recollection of history is completely off. By the time Google came on the scene, web crawling was well established. AltaVista and Lycos, among oth

I think your recollection of history is completely off. By the time Google came on the scene, web crawling was well established. AltaVista and Lycos, among others, were already entrenched search engines. robots.txt was created by the guy that created the first search engine.