5 ms·
Google indexing DDG search results
- zamalek 7y agoIt would probably be more conspicuous if they didn't.
- deleted 7y ago[deleted]
- InsomniacL 7y agoIs their robots.txt valid? https://duckduckgo.com/robots.txt https://duckduckgo.com/robots.txt # No search result pages Disallow: /*?
- rbinv 7y agoIt is valid, and so are Google's results as the URLs don't contain a question mark.
- deleted 7y ago[deleted]
- crazygringo 7y agoAnd? Their robots.txt allows it [1] -- it only disallows URL's with a question mark in them (i.e. GET parameters) and none of the indexed pages have that: Disallow: /*? And Google is indexing pages like: https://duckduckgo.com/Boeing So not sure what the point of this submission is? [1] https://duckduckgo.com/robots.txt https://duckduckgo.com/robots.txt
- saalweachter 7y agoYeah, the https://en.m.wikipedia.org/wiki/Robots_exclusion_standard https://en.m.wikipedia.org/wiki/Robots_exclusion_standard dates from a more civilized time, when URLs were more, erm, uniformly structured.
- t0astbread 7y agoSomeone found it mildly interesting. And it shows how you can mess up your robots.txt.
- jrwr 7y agoInteresting, They block the Wayback Machine's Scraper
- NikolaeVarius 7y agoAre we outraging yet?
- wmf 7y agoGoogle has a policy of not indexing search results so these will probably be removed although technically there's nothing wrong here.
- utopcell 7y agoIt almost looks staged.
- raslah 7y agoIt's weird that there some subtle defensiveness here. To heck with robots.txt and what not, when a site touts it's privacy, even this, however irrelevant, is slightly eyebrow-raising.
- kchamplewski 7y agoHow does this in any way reflect on either the privacy of DuckDuckGo (I don't think they ever claimed or even vaguely implied that if I search for, say, Banana, no one else will be able to see the results page for Banana, as that would be absurd), or Google, who are simply indexing publicly accessible pages, which is what they've always done since coming into existence. What violation of privacy or hint thereof is happening here?
- raslah 7y agoNone. Thus my question of why the Curt responses and passive aggression. DDG has positioned itself as the anti Google and lo and behold Google still finds out what's being searched there, though not necessarily who searched it. If you're a bit cynical, it's vaguely interesting.
- theboulevardier 7y agoIf you look at the cached Google page, you will see that Google is only indexing the knowledge panel, not the results themselves. E.g https://webcache.googleusercontent.com/search?q=cache:t8Z5T9lBLDQJ:https://duckduckgo.com/Database+&cd=10&hl=en&ct=clnk&gl=uk https://webcache.googleusercontent.com/search?q=cache:t8Z5T9...
- jtbayly 7y agoI'd love to see a feedback loop of Google and DDG both looking at each other for factual information to deliver alongside search results.
- lgats 7y agoLooks like the query results come from a url that includes a ? and therefore they are blocked by the robots.txt "/*?" rule. https://duckduckgo.com/d.js?q=%C5%8Ctsuchi&t=A&l=wt-wt&s=0&ct=US&ss_mkt=us&vqd=3-98318517210371365151044395550013262794-162317560025750407458371726853926388732&p_ent=&ex=-1&sp=0&v7exp=a https://duckduckgo.com/d.js?q=%C5%8Ctsuchi&t=A&l=wt-wt&s=0&c...
- cocktailpeanuts 7y agoHave you ever thought, maybe that's how DDG intended it? small time search engines like DDG have no luxury to block any potential routes of people discovering them, especially through Google. I would even bet they are doing "SEO" to optimize their search results for Google.