3 ms·
I have observed this bot requesting URLs that haven’t been live for over a decade, and to which no reference can now be found in search engines. I imagine there
by jl6 2y ago
I have observed this bot requesting URLs that haven’t been live for over a decade, and to which no reference can now be found in search engines. I imagine there must be a private trade in URL lists.
- is_true 2y agomaybe they are using commoncrawl, webarchive, yandex as indexes?
- jefozabuss 2y agoIn addition to those it's also possible they just found a website that published a scraped list back then and got de-indexed for obvious spammy content. I would not be surprised if there are still some auto generated link directories left from the "golden ages" of blackhat.