3 ms·
We already have sitemap.xml that could be used for this purpose
by exocet_uk 8y ago
We already have sitemap.xml that could be used for this purpose
- albertgoeswoof 8y agoYou're right actually. Building a web extension for this would be pretty cool. I'll take that on when I get some free time.
- matt4077 8y agositemaps easily run into the gigabytes for larger sites. I just checked https://www.nytimes.com/sitemaps/www.nytimes.com/sitemap.xml.gz https://www.nytimes.com/sitemaps/www.nytimes.com/sitemap.xml... and it contains 2000 urls of individual weekly sitemaps of 1 or 2MB each. And that's just part of their sitemap structure.