5 ms·

Why is it a double standard? Google scraping usually benefits the site with increased traffic and revenue, in a way most other scraping does not. Saying "you ca

Why is it a double standard? Google scraping usually benefits the site with increased traffic and revenue, in a way most other scraping does not. Saying "you can scrape me if it benefits me" isn't totally in keeping with the principles of the open web, but it's not hypocritical.

tangue 10y ago
Google is basically algorithms built on top of a scraping service. It's unfair to competitors (and potential disruptors) to restrict access to data that Google can fetch without limits.
- collinmanderson 10y ago
  Maybe we (scrappers) just need to market ourselves as search engines. _Indexing_ is what we're doing. :)
  - lerpa 10y ago
    Exactly, market scrappers as search engines. And all smart websites should include a ToS that says you are not allowed to access their data, so they can sue for trespassing anyone that they don't like selectively. The far reaching of government into this, and also the pirating stuff (which I do not condone but think that arresting people for that is waay too much) is what makes me want for the system to collapse under it's own weight. Like some website suing members of congress for visiting it while violating the ToS in this case. I also secretly wanted Oracle to win vs Google so that cloning an API was piracy and that would extend to being a crime to purchase pirated goods which would make all clean room reverse engineering a criminal activity. That would lead to anyone that uses a PC without an authentic IBM BIOS (look up Phoenix BIOS) to be arrested, in theory, so even the US president would have to fall into that. It would have been a glorious shitstorm if Oracle won and IBM took that precedent to it's logical implications, the computer world would have failed, and the law would either be made even more arbitrary or be fixed, but at least it would be shown how idiotic the state of affairs was.
    - Grishnakh 10y ago
      Your idea about Oracle winning and society coming crashing to a halt is ridiculous and wouldn't have happened. Your flaw is believing that the law and the government will work with logical precision, so that a flaw in the law will, like an infinite recursive loop in programming code, cause complete disaster. It doesn't work that way. There's plenty of cases where the law is clearly broken (see civil forfeiture vs. the 4th Amendment to the US Constitution), yet nothing is done. That's because the government is run by humans, and they'll enforce things the way they want. Double standards happen all the time with law, and it takes big, expensive court cases to sort them out, and of course that only happens when some moneyed interest wants to fix it (which is why civil forfeiture is still a big thing--they're not going after extremely wealthy people or corporations with it). While IBM is certainly large enough to bring a big case like you suggested, the US government is far bigger and can simply invent a legal way of ignoring them, just as was done when the SCOTUS decided to rule in favor of using Eminent Domain to seize private property to hand over to commercial interests.
profeta 10y ago
because i may also come to the point where i am a direct competitor to google, but i will never get there because i can't scrap any site like they can. your next argument may very well be a very racist one with the very same excuse you used above.
- sjwright 10y ago
  And if you have some way to identify yourself as a potential competitor to google and not some jackass trying to scrape email addresses or spam comments forms, I'm all ears.
  - greglindahl 10y ago
    A majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.
    - sjwright 10y ago
      I agree, it's a difficult conundrum. It sucks.
      - greglindahl 10y ago
        There are worse barriers to entry for a search engine! DMCA take-downs... Right to be forgotten... click history...
nmstoker 10y ago
With a risk of stating the obvious, this is a double standard simply because there are two standards - one for Google and one for others. I can't speak for the poster you were replying to, but whilst I see it as logical self-interested behaviour by site owners, it still feels unfair.
- bendbro 10y ago
  There isn't: the function for this standard includes expected benefit as an input. Every standard has inputs, so that certainly isn't the quality for making something a double standard. The only remaining quality is how unfair it feels, so it would probably be better to just address that, since it is obviously the only thing you disagree about.
  - lugg 10y ago
    With that logic decreased wages for women are not a double standard due to the potential for maternity leave affecting their output at work. This is a double standard plain and simple, and a very dangerous one at that.
    - sp527 10y ago
      Your example is a case of discrimination, but the economic rationale is unquestionable. There is a tremendous upfront cost for new employees, who are not valuable contributors for some lengthy ramp up period and furthermore accrue experience over the course of employment. So the lifetime value curve for any given employee is typically skewed left.
    - bendbro 10y ago
      I think I'm just being difficult. My point was that when you call something a double standard, you're arguing two things of equal value have been judged differently under the same standard. But by acknowledging they've been judged differently, you're acknowledging that there is a judgement, a standard, that applies the same to both, and produces the results you object to. What you really object to is the fairness of the qualities checked by the standard. Since the outcome of calling things that, vs calling them a double standard is the same, I think most people already know and have no trouble with this. My protests were worthless. It could gain value if there were certain whitelisted judgable aspects (like expected value), and judgements that aren't based on things from the whitelist are considered outside the scope of a standard. Then, calling the standard unfair and calling it a double standard would have a different meaning (if only in some contrived way, since any aspect is just an argument away from the whitelist)

tangue 10y ago

Google is basically algorithms built on top of a scraping service. It's unfair to competitors (and potential disruptors) to restrict access to data that Google can fetch without limits.

collinmanderson 10y ago
Maybe we (scrappers) just need to market ourselves as search engines. _Indexing_ is what we're doing. :)
- lerpa 10y ago
  Exactly, market scrappers as search engines. And all smart websites should include a ToS that says you are not allowed to access their data, so they can sue for trespassing anyone that they don't like selectively. The far reaching of government into this, and also the pirating stuff (which I do not condone but think that arresting people for that is waay too much) is what makes me want for the system to collapse under it's own weight. Like some website suing members of congress for visiting it while violating the ToS in this case. I also secretly wanted Oracle to win vs Google so that cloning an API was piracy and that would extend to being a crime to purchase pirated goods which would make all clean room reverse engineering a criminal activity. That would lead to anyone that uses a PC without an authentic IBM BIOS (look up Phoenix BIOS) to be arrested, in theory, so even the US president would have to fall into that. It would have been a glorious shitstorm if Oracle won and IBM took that precedent to it's logical implications, the computer world would have failed, and the law would either be made even more arbitrary or be fixed, but at least it would be shown how idiotic the state of affairs was.
  - Grishnakh 10y ago
    Your idea about Oracle winning and society coming crashing to a halt is ridiculous and wouldn't have happened. Your flaw is believing that the law and the government will work with logical precision, so that a flaw in the law will, like an infinite recursive loop in programming code, cause complete disaster. It doesn't work that way. There's plenty of cases where the law is clearly broken (see civil forfeiture vs. the 4th Amendment to the US Constitution), yet nothing is done. That's because the government is run by humans, and they'll enforce things the way they want. Double standards happen all the time with law, and it takes big, expensive court cases to sort them out, and of course that only happens when some moneyed interest wants to fix it (which is why civil forfeiture is still a big thing--they're not going after extremely wealthy people or corporations with it). While IBM is certainly large enough to bring a big case like you suggested, the US government is far bigger and can simply invent a legal way of ignoring them, just as was done when the SCOTUS decided to rule in favor of using Eminent Domain to seize private property to hand over to commercial interests.

profeta 10y ago

because i may also come to the point where i am a direct competitor to google, but i will never get there because i can't scrap any site like they can. your next argument may very well be a very racist one with the very same excuse you used above.

sjwright 10y ago
And if you have some way to identify yourself as a potential competitor to google and not some jackass trying to scrape email addresses or spam comments forms, I'm all ears.
- greglindahl 10y ago
  A majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.
  - sjwright 10y ago
    I agree, it's a difficult conundrum. It sucks.
    - greglindahl 10y ago
      There are worse barriers to entry for a search engine! DMCA take-downs... Right to be forgotten... click history...

nmstoker 10y ago

With a risk of stating the obvious, this is a double standard simply because there are two standards - one for Google and one for others. I can't speak for the poster you were replying to, but whilst I see it as logical self-interested behaviour by site owners, it still feels unfair.

bendbro 10y ago
There isn't: the function for this standard includes expected benefit as an input. Every standard has inputs, so that certainly isn't the quality for making something a double standard. The only remaining quality is how unfair it feels, so it would probably be better to just address that, since it is obviously the only thing you disagree about.
- lugg 10y ago
  With that logic decreased wages for women are not a double standard due to the potential for maternity leave affecting their output at work. This is a double standard plain and simple, and a very dangerous one at that.
  - sp527 10y ago
    Your example is a case of discrimination, but the economic rationale is unquestionable. There is a tremendous upfront cost for new employees, who are not valuable contributors for some lengthy ramp up period and furthermore accrue experience over the course of employment. So the lifetime value curve for any given employee is typically skewed left.
  - bendbro 10y ago
    I think I'm just being difficult. My point was that when you call something a double standard, you're arguing two things of equal value have been judged differently under the same standard. But by acknowledging they've been judged differently, you're acknowledging that there is a judgement, a standard, that applies the same to both, and produces the results you object to. What you really object to is the fairness of the qualities checked by the standard. Since the outcome of calling things that, vs calling them a double standard is the same, I think most people already know and have no trouble with this. My protests were worthless. It could gain value if there were certain whitelisted judgable aspects (like expected value), and judgements that aren't based on things from the whitelist are considered outside the scope of a standard. Then, calling the standard unfair and calling it a double standard would have a different meaning (if only in some contrived way, since any aspect is just an argument away from the whitelist)