4 ms·

If the sites don't want the traffic from links that drive their revenue then why don't they identify the servers which are prefetching content and IP ban them a

by shiado 8y ago

If the sites don't want the traffic from links that drive their revenue then why don't they identify the servers which are prefetching content and IP ban them and file abuse complaints. They probably have very identifiable User-Agent strings. Also this link tax reminds me that I think browser vendors should provide better privacy options for removing the referer header from link requests to prevent tracking.

reaperducer 8y ago
> They probably have very identifiable User-Agent strings Yes. Because bots never change or spoof their user agents.
- beagle3 8y ago
  "link tax" is aimed at mature aggregators and search engines, someone you can actually sue. GoogleBot, BingBot and friends do not change or spoof their user agents.
  - reaperducer 8y ago
    If you think being a large company means its user agent doesn't change, you haven't been monitoring user agents very long. The fact that you choose to use Bing as an example proves my point. Bing is only eight years old. Before that it identified itself as msnbot.
    - beagle3 8y ago
      Wow. Once in 8 years. Quite a point you have there /s Every single respectable search engines (which can be targets of lawsuits, and are thus targeted by this law) has proper identification. If a content owner worries so much that their data might be available somewhere (search engine, aggregator, ...) they did not approve, they should whitelist rather than blacklist. They can (and should) also whitelist/blacklist at the referrer level. But a law actually lets them have it both ways; They only sue those that they dislike, after the fact, letting others keep linking. At least that's the thinking - in practice, if such a law is written (which has happened before), Google, Facebook and others will drop those sources that don't give the info for free, those sources will see their numbers drop 90% over a month, and come back begging to be reindexed. I find it quite infuriating that laws are being put into place for problems which already have a very simple and precise technical solution.
      - reaperducer 8y ago
        Wow. Once in 8 years. Quite a point you have there /s You sound young. It's not unusual for serious code to run for decades without modification. The less it has to be touched, the better it was made from the start.
        beagle3 8y ago
        I have been programming professionally for nearly three decades now. I know that some 24 year old code of mine is still running, essentially unmodified, at the company I worked for at the time. And now that you got off my lawn ... I fail to see what you are trying to say. That a change once in 8 years in a user agent string is too frequent? Because that seems to be what you are saying. And if that is what you are saying, I don't think we have any common ground to base the discussion on.

by shiado 8y ago

reaperducer 8y ago

> They probably have very identifiable User-Agent strings Yes. Because bots never change or spoof their user agents.

beagle3 8y ago
"link tax" is aimed at mature aggregators and search engines, someone you can actually sue. GoogleBot, BingBot and friends do not change or spoof their user agents.
- reaperducer 8y ago
  If you think being a large company means its user agent doesn't change, you haven't been monitoring user agents very long. The fact that you choose to use Bing as an example proves my point. Bing is only eight years old. Before that it identified itself as msnbot.
  - beagle3 8y ago
    Wow. Once in 8 years. Quite a point you have there /s Every single respectable search engines (which can be targets of lawsuits, and are thus targeted by this law) has proper identification. If a content owner worries so much that their data might be available somewhere (search engine, aggregator, ...) they did not approve, they should whitelist rather than blacklist. They can (and should) also whitelist/blacklist at the referrer level. But a law actually lets them have it both ways; They only sue those that they dislike, after the fact, letting others keep linking. At least that's the thinking - in practice, if such a law is written (which has happened before), Google, Facebook and others will drop those sources that don't give the info for free, those sources will see their numbers drop 90% over a month, and come back begging to be reindexed. I find it quite infuriating that laws are being put into place for problems which already have a very simple and precise technical solution.
    - reaperducer 8y ago
      Wow. Once in 8 years. Quite a point you have there /s You sound young. It's not unusual for serious code to run for decades without modification. The less it has to be touched, the better it was made from the start.
      - beagle3 8y ago
        I have been programming professionally for nearly three decades now. I know that some 24 year old code of mine is still running, essentially unmodified, at the company I worked for at the time. And now that you got off my lawn ... I fail to see what you are trying to say. That a change once in 8 years in a user agent string is too frequent? Because that seems to be what you are saying. And if that is what you are saying, I don't think we have any common ground to base the discussion on.