My bet is that it is for technical reasons. Whenever you see a reply link (among other things) the server has to persist a continuation for it. Getting all of t

My bet is that it is for technical reasons. Whenever you see a reply link (among other things) the server has to persist a continuation for it. Getting all of those pages crawled must generate a massive number of continuations to persist. I know that this is one of the reasons that vote-links look the way they do. If that's not the reason, then my #2 guess is that getting the whole site crawled is moving too many comments into the cache. If I remember correctly, whenever an uncached comment gets accessed, it gets cached (basically adding it to a hash), and thus added to the in-memory ram. That gradually raises the memory requirements for running the app, until pg does a restart, which resets the cache as well.