3 ms·
Great list. Would be nice if they had a fully filled robots.txt for download (I could only find the example file).
by ano-ther 3y ago
Great list.
Would be nice if they had a fully filled robots.txt for download (I could only find the example file).
- gavinhking 3y agoRight here: https://darkvisitors.com/robots-txt-builder https://darkvisitors.com/robots-txt-builder This example blocks everything, which is probably not what you want, but it's meant to be a starting point.