5 ms·

From the article -- What concerns me is how obscurely the results have disappeared from three search engines, which also happen to command a fair share of the m

From the article -- What concerns me is how obscurely the results have disappeared from three search engines, which also happen to command a fair share of the market after Google. What the author really means is "they have disappeared from the only other English language index in the US." Yandex, an English language index hosted in Russia returns the wikivoyage queries as expected. If I were to hazard a guess (and I am guessing here) the answer is here (also in the article): The first result links directly to Wikitravel. Wikitravel is the original project and a schism in the project resulted in the creation of Wikivoyage, ... Based on this piece of information I would guess that Wikitravel sent Bing a DMCA take down notice telling them that Wikivoyage infringed on their copyrighted material. And as a result Wikivoyage was de-indexed from general results.

cjensen 5y ago
Wikitravel does not have any copyrights to anything, so I would hope not. It's all contributor-created content licensed under CC-by-SA which is free to copy under certain terms. More importantly, Wikitravel is basically a dead man walking, and Wikivoyage is a living project.
- tw04 5y ago
  > Wikitravel does not have any copyrights to anything, so I would hope not. I guess I’d ask what authority you’re speaking from, because a quick perusal of their terms of use directly contradicts your claim. When you post content you sign over all ownership to them, and they do not allow scraping it. https://www.internetbrands.com/ibterms https://www.internetbrands.com/ibterms > You automatically grant and assign to us, and you represent and warrant that you have the right to grant and assign to us, a perpetual, irrevocable, unlimited, fully paid, fully sub-licensable (through multiple tiers), worldwide license to copy, perform, display, distribute, prepare derivative works from (including, without limitation, incorporating into other works) and otherwise use any content that you post. You also expressly grant and assign to us all rights and causes of action to prohibit and enforce against any unauthorized copying, performance, display, distribution, use or exploitation of, or creation of derivative works from, any content that you post (including but not limited to any unauthorized downloading, extraction, harvesting, collection or aggregation of content that you post). >Any copying, aggregation, display, distribution, performance or derivative use of our sites and services or any content posted on our sites and services whether done directly or through intermediaries (including but not limited to by means of spiders, robots, crawlers, scrapers, framing, iframes or RSS feeds) is prohibited.
  - easton 5y ago
    https://wikitravel.org/shared/Copyleft https://wikitravel.org/shared/Copyleft seems to imply that you're allowed to share the content as long as you attribute it, which I'm almost certain wikivoyage is.
    - tw04 5y ago
      Wikitravel shared is a different site and different license than the main wikitravel site.
      - easton 5y ago
        From the bottom of https://wikitravel.org/en/Nepal https://wikitravel.org/en/Nepal: "Text is available under [http://wikitravel.org/shared/Copyleft http://wikitravel.org/shared/Copyleft Creative Commons Attribution-ShareAlike 3.0] images are available under [http://wikitravel.org/shared/How_to_re-use_Wikitravel_guides http://wikitravel.org/shared/How_to_re-use_Wikitravel_guides various licenses], see each image for details. This site is owned, operated, and maintained by MH Sub I, LLC dba Internet Brands. Copyright © 2018 MH Sub I, LLC dba Internet Brands. " which implies the licensing details are hosted on the shared site, but they apply to the content on wikitravel.org.
  - dustintrex 5y ago
    The footer on every page of the site states: Content is available under Creative Commons Attribution-ShareAlike 3.0 unless otherwise noted.
  - cjensen 5y ago
    For authority: I hold copyright which I've never assigned to a "few" pages. Wikitravel is using my content under the CC-bySA license. The license they are using my content under does not permit them to forbid someone from copying the content.
    - nwiswell 5y ago
      Is this the case for ALL of the content which was copied by Wikivoyage? It doesn't matter if just 1% of the content was copied unlawfully - that's infringement.
- ghostly_s 5y ago
  Did you bother inquiring with the maintainers of this site, or just go straight to blogging about it?
  - cjensen 5y ago
    As a former moderator at both sites who frequently explained this policy to contributors, no I didn't ask the maintainers, who once sued a moderator over the schism, any questions.
- 8kundv3 5y ago
  Legalities come later. If a valid complaint is made, a platform must take action on the infringing content. Valid complaint does not mean honest or accurate complaint. Only that all information legally required to file a complaint is supplied.
  - dang 5y ago
    Could you please stop creating accounts for every few comments you post? We ban accounts that do that. This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix&page=0&query=community%20identity%20by:dang https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
- shp0ngle 5y ago
  Wikitravel and Wikivoyage articles are usually functionally identical. There are sometimes information that are in one and not the other, or vice versa. However, when I look at actual edit history, wikivoyage seems indeed more active. So you got a point there. I usually check both.
oh_sigh 5y ago
Or, search engines see duplicate content on wikitravel/wikivoyage and consider wikitravel the real source and wikivoyage as spammers or SEO tricksters
stonogo 5y ago
That lawsuit was settled years ago. https://www.theverge.com/2013/2/16/3995358/wikimedia-internet-brands-lawsuit-settled https://www.theverge.com/2013/2/16/3995358/wikimedia-interne...
- ChuckMcM 5y ago
  Given that they have already been in court, I'm even more convinced that the DMCA is involved. Here is my amended idea, Wikitravel added some sort of "gotcha" asset to their site, and Wikivoyage scraped their site and sucked it up, and when it appeared on Wikivoyage, Wikitravel sent the DMCA. I could not find WikiVoyage's registered DMCA agent so it is entirely possible that Bing didn't know who to send notice of the takedown too. Basically there seems to be a lot of bad blood between the two sites and as silly as it is, DMCA is an excellent "sniping" tool for taking cheap shots at someone you don't like.
  - deleted 5y ago
    [deleted]
  - bawolff 5y ago
    > I could not find WikiVoyage's registered DMCA agent so it is entirely possible that Bing didn't know who to send notice of the takedown too. Its not hard to find: https://dmca.copyright.gov/osp/search.html;jsessionid=F17F14D2705A1189ABF36A8F802A4FDD?key=Wikivoyage&action=search https://dmca.copyright.gov/osp/search.html;jsessionid=F17F14... Besides, there's also the emails in whois if all else fails.
    - ChuckMcM 5y ago
      Nice catch. I wonder if they have received any email from Bing.
laurent123456 5y ago
Or maybe wikivoyage was incorrectly flagged as a spam website that duplicates content from wikitravel?
- arp242 5y ago
  Yes, that would be my guess. I talked to some people from Wikivoyage years ago, and one of the problems they talked about is that they were getting penalized for "duplicate content" a lot even though Wikivoyage was clearly the better alternative in every way (according to them anyway, I can't attest if that's true or not, or if it's still true since this was years ago). It's not hard to imagine the "duplicate score ranking" got one-upped to "spam domain ranking".
  - Scoundreller 5y ago
    Wikivoyage, by being updated (even subtly) probably gets flagged as spun content.
  - splonk 5y ago
    "better alternative in every way" is overstating things a bit, but my experience when I've sporadically done side-by-side comparisons is that Wikivoyage tends to be more up to date and complete. However, there have definitely been a few articles where Wikitravel was considerably better. Neither is very good for business listings (out of date, spammy, badly sampled). WV tends to be better with practical information, like transportation options from the airport, and how to use public transit. I think the general idea is that power users on travel community sites mostly choose to use WV at this point, but WT tends to rank higher in searches, so it may get more edits from other people.
vzaliva 5y ago
Yandex does a plenty their own filtering. For example they filter all the sites in this list https://eais.rkn.gov.ru/en/ https://eais.rkn.gov.ru/en/ which is used for political censorship in Russia.
tomrod 5y ago
Is it not the case that almost every non-Google, US-based search engine is basically Bing under the hood? It's a market I am not too familiar with, but seems I've heard that repeatedly over the years.
- burnte 5y ago
  No, that's not true. Yahoo's search was for a while, but DDG isn't just Bing, it's an aggregator.
  - EVa5I7bHFq9mnYK 5y ago
    Interesting, what percentage of DDG search results comes from Bing and what percentage comes from Yandex.
- wly_cdgr 5y ago
  The Brave search, based on work done by Cliqz, is independent
cheald 5y ago
Brave has their own index and Wikivoyage is happily found there: https://search.brave.com/search?q=wikivoyage https://search.brave.com/search?q=wikivoyage

cjensen 5y ago

Wikitravel does not have any copyrights to anything, so I would hope not. It's all contributor-created content licensed under CC-by-SA which is free to copy under certain terms. More importantly, Wikitravel is basically a dead man walking, and Wikivoyage is a living project.

tw04 5y ago
> Wikitravel does not have any copyrights to anything, so I would hope not. I guess I’d ask what authority you’re speaking from, because a quick perusal of their terms of use directly contradicts your claim. When you post content you sign over all ownership to them, and they do not allow scraping it. https://www.internetbrands.com/ibterms https://www.internetbrands.com/ibterms > You automatically grant and assign to us, and you represent and warrant that you have the right to grant and assign to us, a perpetual, irrevocable, unlimited, fully paid, fully sub-licensable (through multiple tiers), worldwide license to copy, perform, display, distribute, prepare derivative works from (including, without limitation, incorporating into other works) and otherwise use any content that you post. You also expressly grant and assign to us all rights and causes of action to prohibit and enforce against any unauthorized copying, performance, display, distribution, use or exploitation of, or creation of derivative works from, any content that you post (including but not limited to any unauthorized downloading, extraction, harvesting, collection or aggregation of content that you post). >Any copying, aggregation, display, distribution, performance or derivative use of our sites and services or any content posted on our sites and services whether done directly or through intermediaries (including but not limited to by means of spiders, robots, crawlers, scrapers, framing, iframes or RSS feeds) is prohibited.
- easton 5y ago
  https://wikitravel.org/shared/Copyleft https://wikitravel.org/shared/Copyleft seems to imply that you're allowed to share the content as long as you attribute it, which I'm almost certain wikivoyage is.
  - tw04 5y ago
    Wikitravel shared is a different site and different license than the main wikitravel site.
    - easton 5y ago
      From the bottom of https://wikitravel.org/en/Nepal https://wikitravel.org/en/Nepal: "Text is available under [http://wikitravel.org/shared/Copyleft http://wikitravel.org/shared/Copyleft Creative Commons Attribution-ShareAlike 3.0] images are available under [http://wikitravel.org/shared/How_to_re-use_Wikitravel_guides http://wikitravel.org/shared/How_to_re-use_Wikitravel_guides various licenses], see each image for details. This site is owned, operated, and maintained by MH Sub I, LLC dba Internet Brands. Copyright © 2018 MH Sub I, LLC dba Internet Brands. " which implies the licensing details are hosted on the shared site, but they apply to the content on wikitravel.org.
- dustintrex 5y ago
  The footer on every page of the site states: Content is available under Creative Commons Attribution-ShareAlike 3.0 unless otherwise noted.
- cjensen 5y ago
  For authority: I hold copyright which I've never assigned to a "few" pages. Wikitravel is using my content under the CC-bySA license. The license they are using my content under does not permit them to forbid someone from copying the content.
  - nwiswell 5y ago
    Is this the case for ALL of the content which was copied by Wikivoyage? It doesn't matter if just 1% of the content was copied unlawfully - that's infringement.
ghostly_s 5y ago
Did you bother inquiring with the maintainers of this site, or just go straight to blogging about it?
- cjensen 5y ago
  As a former moderator at both sites who frequently explained this policy to contributors, no I didn't ask the maintainers, who once sued a moderator over the schism, any questions.
8kundv3 5y ago
Legalities come later. If a valid complaint is made, a platform must take action on the infringing content. Valid complaint does not mean honest or accurate complaint. Only that all information legally required to file a complaint is supplied.
- dang 5y ago
  Could you please stop creating accounts for every few comments you post? We ban accounts that do that. This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix&page=0&query=community%20identity%20by:dang https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
shp0ngle 5y ago
Wikitravel and Wikivoyage articles are usually functionally identical. There are sometimes information that are in one and not the other, or vice versa. However, when I look at actual edit history, wikivoyage seems indeed more active. So you got a point there. I usually check both.

oh_sigh 5y ago

Or, search engines see duplicate content on wikitravel/wikivoyage and consider wikitravel the real source and wikivoyage as spammers or SEO tricksters

stonogo 5y ago

That lawsuit was settled years ago. https://www.theverge.com/2013/2/16/3995358/wikimedia-internet-brands-lawsuit-settled https://www.theverge.com/2013/2/16/3995358/wikimedia-interne...

ChuckMcM 5y ago
Given that they have already been in court, I'm even more convinced that the DMCA is involved. Here is my amended idea, Wikitravel added some sort of "gotcha" asset to their site, and Wikivoyage scraped their site and sucked it up, and when it appeared on Wikivoyage, Wikitravel sent the DMCA. I could not find WikiVoyage's registered DMCA agent so it is entirely possible that Bing didn't know who to send notice of the takedown too. Basically there seems to be a lot of bad blood between the two sites and as silly as it is, DMCA is an excellent "sniping" tool for taking cheap shots at someone you don't like.
- deleted 5y ago
  [deleted]
- bawolff 5y ago
  > I could not find WikiVoyage's registered DMCA agent so it is entirely possible that Bing didn't know who to send notice of the takedown too. Its not hard to find: https://dmca.copyright.gov/osp/search.html;jsessionid=F17F14D2705A1189ABF36A8F802A4FDD?key=Wikivoyage&action=search https://dmca.copyright.gov/osp/search.html;jsessionid=F17F14... Besides, there's also the emails in whois if all else fails.
  - ChuckMcM 5y ago
    Nice catch. I wonder if they have received any email from Bing.

laurent123456 5y ago

Or maybe wikivoyage was incorrectly flagged as a spam website that duplicates content from wikitravel?

arp242 5y ago
Yes, that would be my guess. I talked to some people from Wikivoyage years ago, and one of the problems they talked about is that they were getting penalized for "duplicate content" a lot even though Wikivoyage was clearly the better alternative in every way (according to them anyway, I can't attest if that's true or not, or if it's still true since this was years ago). It's not hard to imagine the "duplicate score ranking" got one-upped to "spam domain ranking".
- Scoundreller 5y ago
  Wikivoyage, by being updated (even subtly) probably gets flagged as spun content.
- splonk 5y ago
  "better alternative in every way" is overstating things a bit, but my experience when I've sporadically done side-by-side comparisons is that Wikivoyage tends to be more up to date and complete. However, there have definitely been a few articles where Wikitravel was considerably better. Neither is very good for business listings (out of date, spammy, badly sampled). WV tends to be better with practical information, like transportation options from the airport, and how to use public transit. I think the general idea is that power users on travel community sites mostly choose to use WV at this point, but WT tends to rank higher in searches, so it may get more edits from other people.

vzaliva 5y ago

Yandex does a plenty their own filtering. For example they filter all the sites in this list https://eais.rkn.gov.ru/en/ https://eais.rkn.gov.ru/en/ which is used for political censorship in Russia.

tomrod 5y ago

Is it not the case that almost every non-Google, US-based search engine is basically Bing under the hood? It's a market I am not too familiar with, but seems I've heard that repeatedly over the years.

burnte 5y ago
No, that's not true. Yahoo's search was for a while, but DDG isn't just Bing, it's an aggregator.
- EVa5I7bHFq9mnYK 5y ago
  Interesting, what percentage of DDG search results comes from Bing and what percentage comes from Yandex.
wly_cdgr 5y ago
The Brave search, based on work done by Cliqz, is independent

cheald 5y ago

Brave has their own index and Wikivoyage is happily found there: https://search.brave.com/search?q=wikivoyage https://search.brave.com/search?q=wikivoyage