10 ms·

Wikivoyage has disappeared from DuckDuckGo

240 points by gsa 5y ago

encryptluks2 5y ago
What efforts has DDG actually taken to build a decentralized search index or anything to that degree? Do they make profit, and is their business model literally just acting as a proxy for Bing and what not?
bunnyfoofoo 5y ago
Yeah, it's horrible. If anyone has a way to get in touch with DDG, that'd be great. I've been using !sp to get startpage results with wikivoyage queries.
- sct202 5y ago
  I'm not sure if you can do it on behalf of a website, but Bing Webmaster Support has been pretty responsive in my experience with my own domains, like an actual person emailed me back after a couple days. And since Bing seems to feed a lot of DDG's index, if it gets back in Bing it will probably also go back to normal in DDG.
- FabHK 5y ago
  Just use the !wv bang?
  - bunnyfoofoo 5y ago
    Thanks, I wasn't aware that this one existed.
- karlzt 5y ago
  !s is enough.
  - bunnyfoofoo 5y ago
    Thanks, didn't know this.
- dredmorbius 5y ago
  There is a "share feedback" link on every DDG search page. Select "Other" and you'll have an opportuity to add a comment. I've linked TFA from this post. DDG also monitor a subreddit: https://old.reddit.com/r/duckduckgo/ https://old.reddit.com/r/duckduckgo/ A post has been submitted there: https://old.reddit.com/r/duckduckgo/comments/qgkx46/wikivoyage_has_disappeared_from_duckduckgo/ https://old.reddit.com/r/duckduckgo/comments/qgkx46/wikivoya...
ChuckMcM 5y ago
From the article -- What concerns me is how obscurely the results have disappeared from three search engines, which also happen to command a fair share of the market after Google. What the author really means is "they have disappeared from the only other English language index in the US." Yandex, an English language index hosted in Russia returns the wikivoyage queries as expected. If I were to hazard a guess (and I am guessing here) the answer is here (also in the article): The first result links directly to Wikitravel. Wikitravel is the original project and a schism in the project resulted in the creation of Wikivoyage, ... Based on this piece of information I would guess that Wikitravel sent Bing a DMCA take down notice telling them that Wikivoyage infringed on their copyrighted material. And as a result Wikivoyage was de-indexed from general results.
- cjensen 5y ago
  Wikitravel does not have any copyrights to anything, so I would hope not. It's all contributor-created content licensed under CC-by-SA which is free to copy under certain terms. More importantly, Wikitravel is basically a dead man walking, and Wikivoyage is a living project.
  - tw04 5y ago
    > Wikitravel does not have any copyrights to anything, so I would hope not. I guess I’d ask what authority you’re speaking from, because a quick perusal of their terms of use directly contradicts your claim. When you post content you sign over all ownership to them, and they do not allow scraping it. https://www.internetbrands.com/ibterms https://www.internetbrands.com/ibterms > You automatically grant and assign to us, and you represent and warrant that you have the right to grant and assign to us, a perpetual, irrevocable, unlimited, fully paid, fully sub-licensable (through multiple tiers), worldwide license to copy, perform, display, distribute, prepare derivative works from (including, without limitation, incorporating into other works) and otherwise use any content that you post. You also expressly grant and assign to us all rights and causes of action to prohibit and enforce against any unauthorized copying, performance, display, distribution, use or exploitation of, or creation of derivative works from, any content that you post (including but not limited to any unauthorized downloading, extraction, harvesting, collection or aggregation of content that you post). >Any copying, aggregation, display, distribution, performance or derivative use of our sites and services or any content posted on our sites and services whether done directly or through intermediaries (including but not limited to by means of spiders, robots, crawlers, scrapers, framing, iframes or RSS feeds) is prohibited.
mivade 5y ago
The bang syntax works though: `!wv malta` sends me to the Wikivoyage page on Malta.
- deleted 5y ago
  [deleted]
mellosouls 5y ago
Edit: (apologies for the misinformation). I misread the Wikipedia and related infoboxes which have wikivoyage as an immediate outlink. --- (Original comment) I just tried it and it's the first result fwiw. So whatever the issue is, it hasn't disappeared from DDG entirely, or has been fixed since the post went up.
- nonameiguess 5y ago
  It doesn't show up for me on the first page of results in the main page, but the info box to the right has a direct link to it. Not sure what that means, but it's kind of weird and I've never seen anything like that before. Of course, I'd never heard of WikiVoyage before this, so nice publicity either way, I guess.
- kemayo 5y ago
  I searched for it right now -- DDG for "Wikivoyage" doesn't turn up the site Wikivoyage on the first page of results. Amusingly, the linked article is on the first page.
  - mellosouls 5y ago
    Interesting. This query works for me, so something under the hood (locale?) may be skewing things. https://duckduckgo.com/?q=wikivoyage https://duckduckgo.com/?q=wikivoyage
    - kemayo 5y ago
      Yeah, for me it's the Wikipedia article about Wikivoyage first. The infobox to the right is still there, but I assume that's separate from the index (and searching e.g. "Wikivoyage London" gets no useful results). https://imgur.com/a/JwK6Few https://imgur.com/a/JwK6Few
      - mellosouls 5y ago
        Yes. You're correct, I just went back and realised what I'd seen. I'll update my original comment.
dylan604 5y ago
I just tried going to wikivoyage.org directly and did a search for Malta there. Guess what. It works. You already know what website you want, go there. Do your business there. Cut out the middle man hoovering up your data (even though DDG isn't supposed to do that. still wasted navigation)
- kemayo 5y ago
  Although it's true that people who know they want to use Wikivoyage can get around this, people who're searching for something like "travel to Malta" are the ones really affected by having the Wikivoyage results elided.
  - dylan604 5y ago
    this is true, but if this was the case, you wouldn't be typing "wikivoyage malta" in the first place.
    - kemayo 5y ago
      ...yes, but again, you might be typing "travel to Malta". Wikivoyage being completely scrubbed from the index means it has no chance of helpfully showing up in those results.
      - dylan604 5y ago
        but again, you are missing my point focused on making your point. I understand that this site is suddenly missing some traffic due to whatever reason nefarious or not. That's a given. Only the people using the search phrase "wikivoyage malta" would realize this. The people that search for "travel to Malta" would just get whatever results that are available for whatever reasons those are what is available. Sure, these peple might miss out on some valuable information, or they might not and nobody cares except tech nerds. The world did not stop spining on its axis because of this.
        kymaz 5y ago
tagawa 5y ago
DuckDuckGo gets the bulk of its organic results from Bing[1], and Bing doesn’t seem to show pages from Wikivoyage[2], so contacting Bing to address that issue should fix it in DuckDuckGo, as well as any other search engines that use Bing’s index. [1] https://help.duckduckgo.com/duckduckgo-help-pages/results/sources/ https://help.duckduckgo.com/duckduckgo-help-pages/results/so... [2] https://www.bing.com/search?q=wikivoyage%20malta https://www.bing.com/search?q=wikivoyage%20malta
- deleted 5y ago
  [deleted]
engineeringwoke 5y ago
Congrats on the post Gagan :)
EastOfTruth 5y ago
I switched from Google to DDG about a year ago but I just switched to Startpage about a week ago... lets see how it goes (I was getting bored of Bing's results). But I wonder how long Wikivoyage has been blocked from DDG and/or Bing...
- julienpalard 5y ago
  Isn't startpage owned by an advertisment company? https://restoreprivacy.com/startpage-system1-privacy-one-group/ https://restoreprivacy.com/startpage-system1-privacy-one-gro...
- FabHK 5y ago
  I like Startpage, but was a bit miffed when it blocked me when I was using a VPN.
- ronyclau 5y ago
  Startpage was my preferred search engine for the past two years, however in the past half year, I get many of my searches blocked when they contain "site:" param. The situation got worse in thev part few months when some searches with double-quotes get blocked too, which is a deal breaker for me. For blocked search, I meant Startpage showed a page claiming unusual network activity, and presented a report form. There was no captcha, and there's no way to continue the search from there. I wasn't using VPN when this happened, and I encountered this on my home WiFi, office WiFi, and mobile network, MacBook and Android.
deelowe 5y ago
Seems to work for me.
- FabHK 5y ago
  Ha, when I put "Malta Wikivoyage" into DDG, I get this thread now. That helps... :-)
deleted 5y ago
[deleted]
mgdlbp 5y ago
I've gotten into a habit of instinctively searching for queries in multiple major search engines as I've often found that many useful results are found by one but not another. Most of the time, notable differences seem to be in NLP or 'Oh, I know what you're looking for!' algorithms or are the result of DMCA or other requests, but I've noticed a few situations like this where results differ or seem special-cased, sometimes amusingly, for mysterious reasons (edit: another comment points out possible copyright issues with Wikivoyage). Here are a couple cases where I archived results pages from Bing, Google, and Yandex with the Wayback Machine and archive.is. A particularly puzzling case as to why the result might be filtered: one particular Wikipedia page. Searching by its title, using the queries: victorian erotica wikipedia or "victorian erotica" site:en.wikipedia.org Bing refuses to present the page, while Google and Yandex offer it with no qualms. Interestingly, when I discovered this, the article's talk page was the first result in Bing, but now even that no longer appears. Another query, where the issue is likely that it borders on one of a diverse variety of sensitive and/or politically controversial topics where results sometimes seem deliberately tuned in one index or another (in my anecdata: mostly by Google), is the title of one of the hoax papers submitted in the grievance studies affair, which appears verbatim in many pages about it: my struggle to dismantle my whiteness a critical race examination of whiteness from within whiteness Bing and Yandex exhibit expected behavior. Google, however (which is usually the best at determining the subject of a query consisting of an out-of-context quote) seems to only focus on a small fraction of keywords and finds no results that are remotely related to the grievance studies affair itself. I've seen discussions about the decreasing relevancy of search results in general where some have suggested that some sort of central repository be started for archiving interesting search results. I wonder if there have been any efforts toward such a project? Note: when archiving search pages, I stripped all unnecessary parameters from the URLs. For example, https://web.archive.org/web/0/https://www.google.com/search?q=my+struggle+to+dismantle+my+whiteness+a+critical+race+examination+of+whiteness+from+within+whiteness https://web.archive.org/web/0/https://www.google.com/search?...
- Igelau 5y ago
  > central repository be started for archiving interesting search results. I wonder if there have been any efforts toward such a project? What you want is a "directory". Like Yahoo! of old. Search killed these during the window where search was useful.
Jerry2 5y ago
I gave Brave search a try and it did return lots of results for Wikivoyage [1] but the first result was a page in German and yet I'm not in Germany and don't even speak German. English page was 4th down. Still, I prefer Brave's search results to those of DuckDuckGo. I just don't like how much stuff is missing from DDG and Bing (DDG's actual search provider). DDG is feels like a proxy for Bing so if I was to choose a different search engine, I'd just go straight to Bing and bypass DDG altogether. Anyway, give Brave Search a chance! [2] [1] https://i.imgur.com/5f4RHBh.jpg https://i.imgur.com/5f4RHBh.jpg [2] https://search.brave.com/search?q=Wikivoyage+malta&source=web https://search.brave.com/search?q=Wikivoyage+malta&source=we...
- wly_cdgr 5y ago
  Most likely you got a German result cos the engine was built by a German team at Cliqz, originally for the German market
c80e74f077 5y ago
What I realized some times ago, but can't explain (I didn't investigate though), it's that different versions of DDG have different indexes or at least return different results. This is once again confirmed with the example in the article: - DDG regular, no expected result: https://duckduckgo.com/?q=wikivoyage+malta https://duckduckgo.com/?q=wikivoyage+malta - DDG Lite, the expected result is returned: https://lite.duckduckgo.com/lite/?q=wikivoyage+malta https://lite.duckduckgo.com/lite/?q=wikivoyage+malta
- diggernet 5y ago
  When I tried just now, both links gave me the expected result.
Markoff 5y ago
DDG no thanks, as much as I dislike Google, their search engine is pretty much the only service I use, because there is no viable competition my 2nd option would be yandex.ru guess what, both these search engines return results for wikivoyage malta without any issues

240 points by gsa 5y ago

What efforts has DDG actually taken to build a decentralized search index or anything to that degree? Do they make profit, and is their business model literally just acting as a proxy for Bing and what not?

bunnyfoofoo 5y ago

Yeah, it's horrible. If anyone has a way to get in touch with DDG, that'd be great. I've been using !sp to get startpage results with wikivoyage queries.

sct202 5y ago
I'm not sure if you can do it on behalf of a website, but Bing Webmaster Support has been pretty responsive in my experience with my own domains, like an actual person emailed me back after a couple days. And since Bing seems to feed a lot of DDG's index, if it gets back in Bing it will probably also go back to normal in DDG.
FabHK 5y ago
Just use the !wv bang?
- bunnyfoofoo 5y ago
  Thanks, I wasn't aware that this one existed.
karlzt 5y ago
!s is enough.
- bunnyfoofoo 5y ago
  Thanks, didn't know this.
dredmorbius 5y ago
There is a "share feedback" link on every DDG search page. Select "Other" and you'll have an opportuity to add a comment. I've linked TFA from this post. DDG also monitor a subreddit: https://old.reddit.com/r/duckduckgo/ https://old.reddit.com/r/duckduckgo/ A post has been submitted there: https://old.reddit.com/r/duckduckgo/comments/qgkx46/wikivoyage_has_disappeared_from_duckduckgo/ https://old.reddit.com/r/duckduckgo/comments/qgkx46/wikivoya...

ChuckMcM 5y ago

From the article -- What concerns me is how obscurely the results have disappeared from three search engines, which also happen to command a fair share of the market after Google. What the author really means is "they have disappeared from the only other English language index in the US." Yandex, an English language index hosted in Russia returns the wikivoyage queries as expected. If I were to hazard a guess (and I am guessing here) the answer is here (also in the article): The first result links directly to Wikitravel. Wikitravel is the original project and a schism in the project resulted in the creation of Wikivoyage, ... Based on this piece of information I would guess that Wikitravel sent Bing a DMCA take down notice telling them that Wikivoyage infringed on their copyrighted material. And as a result Wikivoyage was de-indexed from general results.

cjensen 5y ago
Wikitravel does not have any copyrights to anything, so I would hope not. It's all contributor-created content licensed under CC-by-SA which is free to copy under certain terms. More importantly, Wikitravel is basically a dead man walking, and Wikivoyage is a living project.
- tw04 5y ago
  > Wikitravel does not have any copyrights to anything, so I would hope not. I guess I’d ask what authority you’re speaking from, because a quick perusal of their terms of use directly contradicts your claim. When you post content you sign over all ownership to them, and they do not allow scraping it. https://www.internetbrands.com/ibterms https://www.internetbrands.com/ibterms > You automatically grant and assign to us, and you represent and warrant that you have the right to grant and assign to us, a perpetual, irrevocable, unlimited, fully paid, fully sub-licensable (through multiple tiers), worldwide license to copy, perform, display, distribute, prepare derivative works from (including, without limitation, incorporating into other works) and otherwise use any content that you post. You also expressly grant and assign to us all rights and causes of action to prohibit and enforce against any unauthorized copying, performance, display, distribution, use or exploitation of, or creation of derivative works from, any content that you post (including but not limited to any unauthorized downloading, extraction, harvesting, collection or aggregation of content that you post). >Any copying, aggregation, display, distribution, performance or derivative use of our sites and services or any content posted on our sites and services whether done directly or through intermediaries (including but not limited to by means of spiders, robots, crawlers, scrapers, framing, iframes or RSS feeds) is prohibited.

mivade 5y ago

The bang syntax works though: `!wv malta` sends me to the Wikivoyage page on Malta.

deleted 5y ago
[deleted]

mellosouls 5y ago

Edit: (apologies for the misinformation). I misread the Wikipedia and related infoboxes which have wikivoyage as an immediate outlink. --- (Original comment) I just tried it and it's the first result fwiw. So whatever the issue is, it hasn't disappeared from DDG entirely, or has been fixed since the post went up.

nonameiguess 5y ago
It doesn't show up for me on the first page of results in the main page, but the info box to the right has a direct link to it. Not sure what that means, but it's kind of weird and I've never seen anything like that before. Of course, I'd never heard of WikiVoyage before this, so nice publicity either way, I guess.
kemayo 5y ago
I searched for it right now -- DDG for "Wikivoyage" doesn't turn up the site Wikivoyage on the first page of results. Amusingly, the linked article is on the first page.
- mellosouls 5y ago
  Interesting. This query works for me, so something under the hood (locale?) may be skewing things. https://duckduckgo.com/?q=wikivoyage https://duckduckgo.com/?q=wikivoyage
  - kemayo 5y ago
    Yeah, for me it's the Wikipedia article about Wikivoyage first. The infobox to the right is still there, but I assume that's separate from the index (and searching e.g. "Wikivoyage London" gets no useful results). https://imgur.com/a/JwK6Few https://imgur.com/a/JwK6Few
    - mellosouls 5y ago
      Yes. You're correct, I just went back and realised what I'd seen. I'll update my original comment.

dylan604 5y ago

I just tried going to wikivoyage.org directly and did a search for Malta there. Guess what. It works. You already know what website you want, go there. Do your business there. Cut out the middle man hoovering up your data (even though DDG isn't supposed to do that. still wasted navigation)

kemayo 5y ago
Although it's true that people who know they want to use Wikivoyage can get around this, people who're searching for something like "travel to Malta" are the ones really affected by having the Wikivoyage results elided.
- dylan604 5y ago
  this is true, but if this was the case, you wouldn't be typing "wikivoyage malta" in the first place.
  - kemayo 5y ago
    ...yes, but again, you might be typing "travel to Malta". Wikivoyage being completely scrubbed from the index means it has no chance of helpfully showing up in those results.
    - dylan604 5y ago
      but again, you are missing my point focused on making your point. I understand that this site is suddenly missing some traffic due to whatever reason nefarious or not. That's a given. Only the people using the search phrase "wikivoyage malta" would realize this. The people that search for "travel to Malta" would just get whatever results that are available for whatever reasons those are what is available. Sure, these peple might miss out on some valuable information, or they might not and nobody cares except tech nerds. The world did not stop spining on its axis because of this.
      - kymaz 5y ago

tagawa 5y ago

DuckDuckGo gets the bulk of its organic results from Bing[1], and Bing doesn’t seem to show pages from Wikivoyage[2], so contacting Bing to address that issue should fix it in DuckDuckGo, as well as any other search engines that use Bing’s index. [1] https://help.duckduckgo.com/duckduckgo-help-pages/results/sources/ https://help.duckduckgo.com/duckduckgo-help-pages/results/so... [2] https://www.bing.com/search?q=wikivoyage%20malta https://www.bing.com/search?q=wikivoyage%20malta

deleted 5y ago
[deleted]

engineeringwoke 5y ago

Congrats on the post Gagan :)

EastOfTruth 5y ago

I switched from Google to DDG about a year ago but I just switched to Startpage about a week ago... lets see how it goes (I was getting bored of Bing's results). But I wonder how long Wikivoyage has been blocked from DDG and/or Bing...

julienpalard 5y ago
Isn't startpage owned by an advertisment company? https://restoreprivacy.com/startpage-system1-privacy-one-group/ https://restoreprivacy.com/startpage-system1-privacy-one-gro...
FabHK 5y ago
I like Startpage, but was a bit miffed when it blocked me when I was using a VPN.
ronyclau 5y ago
Startpage was my preferred search engine for the past two years, however in the past half year, I get many of my searches blocked when they contain "site:" param. The situation got worse in thev part few months when some searches with double-quotes get blocked too, which is a deal breaker for me. For blocked search, I meant Startpage showed a page claiming unusual network activity, and presented a report form. There was no captcha, and there's no way to continue the search from there. I wasn't using VPN when this happened, and I encountered this on my home WiFi, office WiFi, and mobile network, MacBook and Android.

deelowe 5y ago

Seems to work for me.

FabHK 5y ago
Ha, when I put "Malta Wikivoyage" into DDG, I get this thread now. That helps... :-)

deleted 5y ago

[deleted]

mgdlbp 5y ago

I've gotten into a habit of instinctively searching for queries in multiple major search engines as I've often found that many useful results are found by one but not another. Most of the time, notable differences seem to be in NLP or 'Oh, I know what you're looking for!' algorithms or are the result of DMCA or other requests, but I've noticed a few situations like this where results differ or seem special-cased, sometimes amusingly, for mysterious reasons (edit: another comment points out possible copyright issues with Wikivoyage). Here are a couple cases where I archived results pages from Bing, Google, and Yandex with the Wayback Machine and archive.is. A particularly puzzling case as to why the result might be filtered: one particular Wikipedia page. Searching by its title, using the queries: victorian erotica wikipedia or "victorian erotica" site:en.wikipedia.org Bing refuses to present the page, while Google and Yandex offer it with no qualms. Interestingly, when I discovered this, the article's talk page was the first result in Bing, but now even that no longer appears. Another query, where the issue is likely that it borders on one of a diverse variety of sensitive and/or politically controversial topics where results sometimes seem deliberately tuned in one index or another (in my anecdata: mostly by Google), is the title of one of the hoax papers submitted in the grievance studies affair, which appears verbatim in many pages about it: my struggle to dismantle my whiteness a critical race examination of whiteness from within whiteness Bing and Yandex exhibit expected behavior. Google, however (which is usually the best at determining the subject of a query consisting of an out-of-context quote) seems to only focus on a small fraction of keywords and finds no results that are remotely related to the grievance studies affair itself. I've seen discussions about the decreasing relevancy of search results in general where some have suggested that some sort of central repository be started for archiving interesting search results. I wonder if there have been any efforts toward such a project? Note: when archiving search pages, I stripped all unnecessary parameters from the URLs. For example, https://web.archive.org/web/0/https://www.google.com/search?q=my+struggle+to+dismantle+my+whiteness+a+critical+race+examination+of+whiteness+from+within+whiteness https://web.archive.org/web/0/https://www.google.com/search?...

Igelau 5y ago
> central repository be started for archiving interesting search results. I wonder if there have been any efforts toward such a project? What you want is a "directory". Like Yahoo! of old. Search killed these during the window where search was useful.

Jerry2 5y ago

I gave Brave search a try and it did return lots of results for Wikivoyage [1] but the first result was a page in German and yet I'm not in Germany and don't even speak German. English page was 4th down. Still, I prefer Brave's search results to those of DuckDuckGo. I just don't like how much stuff is missing from DDG and Bing (DDG's actual search provider). DDG is feels like a proxy for Bing so if I was to choose a different search engine, I'd just go straight to Bing and bypass DDG altogether. Anyway, give Brave Search a chance! [2] [1] https://i.imgur.com/5f4RHBh.jpg https://i.imgur.com/5f4RHBh.jpg [2] https://search.brave.com/search?q=Wikivoyage+malta&source=web https://search.brave.com/search?q=Wikivoyage+malta&source=we...

wly_cdgr 5y ago
Most likely you got a German result cos the engine was built by a German team at Cliqz, originally for the German market

c80e74f077 5y ago

What I realized some times ago, but can't explain (I didn't investigate though), it's that different versions of DDG have different indexes or at least return different results. This is once again confirmed with the example in the article: - DDG regular, no expected result: https://duckduckgo.com/?q=wikivoyage+malta https://duckduckgo.com/?q=wikivoyage+malta - DDG Lite, the expected result is returned: https://lite.duckduckgo.com/lite/?q=wikivoyage+malta https://lite.duckduckgo.com/lite/?q=wikivoyage+malta

diggernet 5y ago
When I tried just now, both links gave me the expected result.

Markoff 5y ago

DDG no thanks, as much as I dislike Google, their search engine is pretty much the only service I use, because there is no viable competition my 2nd option would be yandex.ru guess what, both these search engines return results for wikivoyage malta without any issues