2 ms·
What is Nuuton? Why invent a proprietary robots.txt feature instead of the (common but non-standard) "Crawl-delay:"?
by cpeterso 14y ago
What is Nuuton? Why invent a proprietary robots.txt feature instead of the (common but non-standard) "Crawl-delay:"?
- orangethirty 14y ago1. Nuuton is a search engine. 2. Because this feature ties in with others that I cannot talk about. Its optional, and meant for optimizing for Nuuton.