4 ms·It's also because it saves a ton of resources not having to serve crawler requests.by timcederman 13y agoIt's also because it saves a ton of resources not having to serve crawler requests.