3 ms·
Are you sure it's a crawler and not a proxy of some sort? Eg. one of your links is on something high traffic in facebook, and all requests are human, running th
by pmlnr 6y ago
Are you sure it's a crawler and not a proxy of some sort? Eg. one of your links is on something high traffic in facebook, and all requests are human, running through fb machines.
- napolux 6y agoNah, it's a specific user-agent for their crawler