3 ms·
robots.txt is not a sitemap. If it worked that way you could just make a 5TB file linking to a billion pages that look like static links but are dynamically gen
by brookst 2y ago
robots.txt is not a sitemap. If it worked that way you could just make a 5TB file linking to a billion pages that look like static links but are dynamically generated.
- ccgreg 2y agorobots.txt has a maximum relevant size of 500 kib.