3 ms·
And? Their robots.txt allows it [1] -- it only disallows URL's with a question mark in them (i.e. GET parameters) and none of the indexed pages have that: D
by crazygringo 7y ago
And?
Their robots.txt allows it [1] -- it only disallows URL's with a question mark in them (i.e. GET parameters) and none of the indexed pages have that:
Disallow: /*?
And Google is indexing pages like:
https://duckduckgo.com/Boeing
So not sure what the point of this submission is?
[1] https://duckduckgo.com/robots.txt https://duckduckgo.com/robots.txt
- saalweachter 7y agoYeah, the https://en.m.wikipedia.org/wiki/Robots_exclusion_standard https://en.m.wikipedia.org/wiki/Robots_exclusion_standard dates from a more civilized time, when URLs were more, erm, uniformly structured.
- t0astbread 7y agoSomeone found it mildly interesting. And it shows how you can mess up your robots.txt.
- jrwr 7y agoInteresting, They block the Wayback Machine's Scraper