4 ms·It would help if big sites like Bytedance's Bytespider actually observed robots.txtby bediger4000 2y agoIt would help if big sites like Bytedance's Bytespider actually observed robots.txt