5 ms·
The buried lede here is what's going on with Baidu. Presumably the initial erroneous request was typed into a browser, does Baidu operate a browser in China? Do
by vermilingua 6y ago
The buried lede here is what's going on with Baidu. Presumably the initial erroneous request was typed into a browser, does Baidu operate a browser in China? Do they operate a portion of the backbone network infrastructure, and are inspecting traffic?
If not, how could they possibly have identified this as a target for crawling, unless perhaps they are being fed traffic by the Great Firewall?
- erk__ 6y agoI don't think they operate a browser anymore. My guess at how this happend was that some user tried to open the site, it failed and the copied and pasted the url into Baidu search which then crawled it, and probably also corrected the misspelling.
- _trampeltier 6y agoI did hear in a talk, where they did this with (as far I remember) google.com, a lot of such wrong traffic came from china too.
- kalleboo 6y agoEven if they don't have a browser they could have a browser extension edit: it looks like they also have antivirus software which could be scanning urls
- deleted 6y ago[deleted]
- JdeBP 6y agoThat's easy enough. There are WWW pages that contain "time.wiodows.com". I turned up a couple with Bing, in examples of how to use the ntpdate program, and there are no doubt more, including ones where this is hyperlinked by some "smart" program. No need for a nefarious explanation when a WWW spider found this on a WWW page and decided to crawl it will do just as well. (-:
- banana_giraffe 6y agoEven if they're not hyperlinked, at this point, I wouldn't be surprised to learn of a crawler looking for FQDNs and things that just generally look like URLs in text to crawl into.
- vermilingua 6y agoI don’t buy it. The odds that Baidu only crawls that page immediately after it is connected to by a chinese IP are astronomical. EDIT: Ah, unless the page containing the link was accessed via a Baidu search. That makes sense.
- JdeBP 6y agoThey could both have been the WWW spider. Search engines have been known to not announce themselves, to check what they are seeing against what everyone else sees.