3 ms·
It might be present in your web server logs, with the caveat that you may not have them or have access to them and the bots may not pass the HTTP_REFERER header
by benoau 1y ago
It might be present in your web server logs, with the caveat that you may not have them or have access to them and the bots may not pass the HTTP_REFERER header and the logs may not record it.
It might be present in client-side logs, with the caveat the crawler might block tracker URLs or not open a page in a way that executes their scripts.
Either of these may reveal what the first-page was a crawler reached, or the pages they directly-access without navigating your site.
You might be able to correlate timing for when a "user query" occurs and subsequent scraping.