6 ms·

There are a few less obvious reasons why you wouldn't want to allow people to link to your content: - Privacy: You want to be in control of how the information

There are a few less obvious reasons why you wouldn't want to allow people to link to your content: - Privacy: You want to be in control of how the information on your site is procured. This is common on government sites that generally allow public access, but require all sorts of forms and certifications to gain that access. - Competition: If you are competing with Google or another to create the user's initial landing page, you would reasonably want to prevent them to access your own landing page content. - Trust/Reputation: Google tries to link to as many pages as it can, but it can't link to them all. If your goal is to provide a trusted comprehensive service, you may want to deter "easy" but "non-comprehensive" access to your content. For example, it's conceivable that a medical journal would prevent access because Google only indexes 40% of it's content, and end users may develop a false (and potentially deadly) understanding that Google is comprehensive.

fauigerzigerk 8y ago
Absolutely, but not getting linked to has always been an option. Publishers want Google to link to their content. At the same time they want Google to pay for the honor and prevent competitors from letting Google link to their content for free. The only reason this ridiculously brazen set of demands doesn't get laughed out of the room is that these publishers have tremendous political clout.
- kuschku 8y ago
  The big issue is that Google over time has moved to scrape the web. One big thing Google has done recently is they extract what you search for from a website, and show it inline. E.g. when I search for a recipe, I get the result in the Google search, and don’t need to visit their website. https://i.k8r.eu/aILGEg https://i.k8r.eu/aILGEg Google extracts reviews from Yelp!, and I can read them without visiting Yelp!. They’re doing this for everything, and AMP furthers this. Google is trying to keep the user as long as possible on Google, and show snippets instead of letting users leave. As result many of these sites lose out on ad impressions, and money. Google has no way to disable this feature short of entire unlisting your site.
  - cabalamat 8y ago
    > The big issue is that Google over time has moved to scrape the web. This is true, and a lot of Google's practices are monopolistic/hegemonic. However banning/taxing links is not the answer.
    - kuschku 8y ago
      But what are you gonna do about it then? Ban snippets? Tax snippets? Wait for Google to be the web? Specially regulate Google? There’s no good solutions.
      - cabalamat 8y ago
        > But what are you gonna do about it then? Ban snippets? Tax snippets? Wait for Google to be the web? Specially regulate Google? I think I would ban search engines from doing other things, ban or regulate targetted advertising, require social networks to be interoperable, and ban social networks from owning other social networks (so Facebook would have to sell off Instagram).
        kuschku 8y ago
        That’s an idea I can agree with, and that Vestager visibly sympathizes with, but that’d mean breaking apart Google, and as result would anger a lot of people.
  - fauigerzigerk 8y ago
    I can see the problem with that. But as I understand it the proposed "link tax" is not limited to this sort of thing.
    - kuschku 8y ago
      Yeah, it’s why I hate the law as it is, but I can obviously also understand why Google seems like a threat to the actors behind this.
- speedplane 8y ago
  Google's stated goal is to provide people access to information. But in reality, they provide access to what's easily accessible to them. Any robots.txt tool is far too blunt of an instrument. It does not respect other highly valued goals of privacy, competitive fairness, and comprehensiveness, among others. To put it more starkly, to assist with privacy, there should be a robots.txt tag/line that allows scraping, but prohibits indexing anything that is tagged as "personally identifiable information". It should be easy to Google what the newest bankruptcy law is, but should be a bit harder to figure out if a specific person filed for bankruptcy. Similarly, with comprehensiveness, robots.txt should be able to say "don't index these pages unless you also index those pages". For example, a medical site should be able to say, you can index these several name-brand drugs but only if you also index their generic equivalents. And as far as fairness, a better robots.txt should be able to tell Google to index the portions of its content to attract clicks, but prevent them from scraping their most valuable content. For example, Yelp may allow them to index a restaraunt's overall rating, but not summarize their reviews.
netheril96 8y ago
You can just have a `robots.txt` if you don't want Google to link to your content.
- ocdtrekkie 8y ago
  The issue is that sets a black and white option of not letting people find you through Google. But the alternative is letting Google scrape your content and give it to people directly from their site instead.
  - diffeomorphism 8y ago
    > But the alternative is letting Google scrape your content and give it to people directly from their site instead. No, that is not the alternative. The alternative is to allow citations as they are allowed everywhere else, i.e. small fractions of the content with attribution. For example, google currently may list the headline of an article and link to it. The proposal, which is based on a failed German law, is that google should pay for the privilege of sending viewers to newspapers this way. Last time they tried google simply said "no, thanks. Then we'll not list you". Newspapers immediately caved and said they will give free access. End result: monopolies get free access and smaller potential competitors have to pay. The law completely and predictably backfired. The, misguided IMO, hope now is that if it is an EU wide law, google will have to bow down and pay for the privilege of providing a google news like service.
    - ocdtrekkie 8y ago
      I believe the newer proposal addresses that flaw. But to be honest, if Google can't build a service without theft, it should simply not operate. New solutions will be able to fill the gap.

fauigerzigerk 8y ago

Absolutely, but not getting linked to has always been an option. Publishers want Google to link to their content. At the same time they want Google to pay for the honor and prevent competitors from letting Google link to their content for free. The only reason this ridiculously brazen set of demands doesn't get laughed out of the room is that these publishers have tremendous political clout.

kuschku 8y ago
The big issue is that Google over time has moved to scrape the web. One big thing Google has done recently is they extract what you search for from a website, and show it inline. E.g. when I search for a recipe, I get the result in the Google search, and don’t need to visit their website. https://i.k8r.eu/aILGEg https://i.k8r.eu/aILGEg Google extracts reviews from Yelp!, and I can read them without visiting Yelp!. They’re doing this for everything, and AMP furthers this. Google is trying to keep the user as long as possible on Google, and show snippets instead of letting users leave. As result many of these sites lose out on ad impressions, and money. Google has no way to disable this feature short of entire unlisting your site.
- cabalamat 8y ago
  > The big issue is that Google over time has moved to scrape the web. This is true, and a lot of Google's practices are monopolistic/hegemonic. However banning/taxing links is not the answer.
  - kuschku 8y ago
    But what are you gonna do about it then? Ban snippets? Tax snippets? Wait for Google to be the web? Specially regulate Google? There’s no good solutions.
    - cabalamat 8y ago
      > But what are you gonna do about it then? Ban snippets? Tax snippets? Wait for Google to be the web? Specially regulate Google? I think I would ban search engines from doing other things, ban or regulate targetted advertising, require social networks to be interoperable, and ban social networks from owning other social networks (so Facebook would have to sell off Instagram).
      - kuschku 8y ago
        That’s an idea I can agree with, and that Vestager visibly sympathizes with, but that’d mean breaking apart Google, and as result would anger a lot of people.
- fauigerzigerk 8y ago
  I can see the problem with that. But as I understand it the proposed "link tax" is not limited to this sort of thing.
  - kuschku 8y ago
    Yeah, it’s why I hate the law as it is, but I can obviously also understand why Google seems like a threat to the actors behind this.
speedplane 8y ago
Google's stated goal is to provide people access to information. But in reality, they provide access to what's easily accessible to them. Any robots.txt tool is far too blunt of an instrument. It does not respect other highly valued goals of privacy, competitive fairness, and comprehensiveness, among others. To put it more starkly, to assist with privacy, there should be a robots.txt tag/line that allows scraping, but prohibits indexing anything that is tagged as "personally identifiable information". It should be easy to Google what the newest bankruptcy law is, but should be a bit harder to figure out if a specific person filed for bankruptcy. Similarly, with comprehensiveness, robots.txt should be able to say "don't index these pages unless you also index those pages". For example, a medical site should be able to say, you can index these several name-brand drugs but only if you also index their generic equivalents. And as far as fairness, a better robots.txt should be able to tell Google to index the portions of its content to attract clicks, but prevent them from scraping their most valuable content. For example, Yelp may allow them to index a restaraunt's overall rating, but not summarize their reviews.

netheril96 8y ago

You can just have a `robots.txt` if you don't want Google to link to your content.

ocdtrekkie 8y ago
The issue is that sets a black and white option of not letting people find you through Google. But the alternative is letting Google scrape your content and give it to people directly from their site instead.
- diffeomorphism 8y ago
  > But the alternative is letting Google scrape your content and give it to people directly from their site instead. No, that is not the alternative. The alternative is to allow citations as they are allowed everywhere else, i.e. small fractions of the content with attribution. For example, google currently may list the headline of an article and link to it. The proposal, which is based on a failed German law, is that google should pay for the privilege of sending viewers to newspapers this way. Last time they tried google simply said "no, thanks. Then we'll not list you". Newspapers immediately caved and said they will give free access. End result: monopolies get free access and smaller potential competitors have to pay. The law completely and predictably backfired. The, misguided IMO, hope now is that if it is an EU wide law, google will have to bow down and pay for the privilege of providing a google news like service.
  - ocdtrekkie 8y ago
    I believe the newer proposal addresses that flaw. But to be honest, if Google can't build a service without theft, it should simply not operate. New solutions will be able to fill the gap.