4 ms·
On my server with 70 sites with about 100.000.000 unique pages I saw 60 urls per second regularly. Not even Googlebot hammers my server at that rate.
by spiderfarmer 2y ago
On my server with 70 sites with about 100.000.000 unique pages I saw 60 urls per second regularly. Not even Googlebot hammers my server at that rate.
- deleted 2y ago[deleted]
- iLoveOncall 2y agoThat's 0.00006% of all your pages per second. Even if it's sustained it really isn't much at all. At this rate sustained it would take more than 20 days to crawl all the pages. Again, this is really not painting the picture you want it to paint.