Crawlers are pretty hard to build, they have an insane number of corner cases they need to deal with if you want them to perform well AND be perceived as respec

Crawlers are pretty hard to build, they have an insane number of corner cases they need to deal with if you want them to perform well AND be perceived as respectful, and crawlers (if you go that route) find themselves among the harder problems in distributed computing, with a huge shared mutable state and some very complex shared timers. If you're in a hurry to race to the market, it's very likely you'll run into these issues and find yourself tempted to cut corners, and unfortunately, with nearly unbounded cloud spend, cutting corners in a large scale crawler operation can very believably cause major disruption all over the web.