I used to be on the Google indexing team. Disregarding limits on the length of URLs, the size of the visible web is already infinite. For instance, there are

I used to be on the Google indexing team. Disregarding limits on the length of URLs, the size of the visible web is already infinite. For instance, there are many calendar pages out there that will happily give you month after month ad infinitum if you keep following the "next" link. Now, depending on how you prune your crawl to get rid of "uninteresting" content (such as infinite calendars) and how you deduplicate the pages you find, you'll come up with vastly varying estimates of how big the visible web is. Edit: on a side note, don't crawl the web using a naive depth-first search. You'll get stuck in some uninteresting infinitely deep branch of the web.