3 ms·
Based on logs it would seem that the crawler, or at least parts of it are written in golang. When following some redirects the useragent would be cleared and wo
by abritishguy 11y ago
Based on logs it would seem that the crawler, or at least parts of it are written in golang. When following some redirects the useragent would be cleared and would identify itself as golang.