4 ms·

Most probably it would be illegal, against each website's TOS. If such tool catches up, it will be removed - and you still need to account for upstream changes

Most probably it would be illegal, against each website's TOS. If such tool catches up, it will be removed - and you still need to account for upstream changes that periodically kill the functionality.

jacquesm 9y ago
Against the TOS does not make something illegal.
- eeZah7Ux 9y ago
  Not yet - but people seem to confuse legislators and corporations more and more frequently...
  - jacquesm 9y ago
    Clink or Hoosegow?
- dredmorbius 9y ago
  Unfortunately, that's precisely what the CFAA does, in the United States. EFF's interpretation: The Computer Fraud and Abuse Act (“CFAA”), 18 U.S.C. § 1030, is an amendment made in 1986 to the Counterfeit Access Device and Abuse Act that was passed in 1984 and essentially states that, whoever intentionally accesses a computer without authorization or exceeds authorized access, and thereby obtains information from any protected computer if the conduct involved an interstate or foreign communication shall be punished under the Act. In 1996 the CFAA was, again, broadened by an amendment that replaced the term “federal interest computer” with the term “protected computer.”18 U.S.C. § 1030. While the CFAA is primarily a criminal law intended to reduce the instances of malicious interferences with computer systems and to address federal computer offenses, an amendment in 1994 allows civil actions to brought under the statute, as well. https://ilt.eff.org/index.php/Computer_Fraud_and_Abuse_Act_%28CFAA%29 https://ilt.eff.org/index.php/Computer_Fraud_and_Abuse_Act_%...
TeMPOraL 9y ago
Thinking of such a tool as a service is a symptom of the same disease that got us where we are in the first place. If such tool would, instead, be open-source and self-hosted (or a component of trivially self-hosted) application, websites could do exactly shit about it. Traffic will be coming from everywhere, and the tool will get updated by people to overcome upstream interface changes. Ultimately, it's SaaS webapps that's the problem. The model has legitimate benefits (lack of installation, cross-platform), but it also is the reason the Web centralizes so much.
- alanfranzoni 9y ago
  Most probably the github repo will be suspended, even though it's self hosted, I think. And you still need to account for upstream changes.
  - TeMPOraL 9y ago
    Forget Github. It might be just a regular git repo on someone's server. IANAL, but could companies really go against a person hosting the source code of a software that scraps a website? Even if, that's probably solvable (by turning it into cat-and-mouse game, TPB/SciHub-style). Have a non-turing-complete set of conversion rules that do scrapping -> standardized-data-format via a general-purpose scrapper. Distribute those through P2P. There might be problems with malicious actors, but generally this should be possible to implement in a way that's mostly transparent for end users. > And you still need to account for upstream changes. Upstream changes will never be too extreme to handle manually, if we're talking scrapping solutions - because any such extreme changes would also piss off the regular browser users.

jacquesm 9y ago

Against the TOS does not make something illegal.

eeZah7Ux 9y ago
Not yet - but people seem to confuse legislators and corporations more and more frequently...
- jacquesm 9y ago
  Clink or Hoosegow?
dredmorbius 9y ago
Unfortunately, that's precisely what the CFAA does, in the United States. EFF's interpretation: The Computer Fraud and Abuse Act (“CFAA”), 18 U.S.C. § 1030, is an amendment made in 1986 to the Counterfeit Access Device and Abuse Act that was passed in 1984 and essentially states that, whoever intentionally accesses a computer without authorization or exceeds authorized access, and thereby obtains information from any protected computer if the conduct involved an interstate or foreign communication shall be punished under the Act. In 1996 the CFAA was, again, broadened by an amendment that replaced the term “federal interest computer” with the term “protected computer.”18 U.S.C. § 1030. While the CFAA is primarily a criminal law intended to reduce the instances of malicious interferences with computer systems and to address federal computer offenses, an amendment in 1994 allows civil actions to brought under the statute, as well. https://ilt.eff.org/index.php/Computer_Fraud_and_Abuse_Act_%28CFAA%29 https://ilt.eff.org/index.php/Computer_Fraud_and_Abuse_Act_%...

TeMPOraL 9y ago

Thinking of such a tool as a service is a symptom of the same disease that got us where we are in the first place. If such tool would, instead, be open-source and self-hosted (or a component of trivially self-hosted) application, websites could do exactly shit about it. Traffic will be coming from everywhere, and the tool will get updated by people to overcome upstream interface changes. Ultimately, it's SaaS webapps that's the problem. The model has legitimate benefits (lack of installation, cross-platform), but it also is the reason the Web centralizes so much.

alanfranzoni 9y ago
Most probably the github repo will be suspended, even though it's self hosted, I think. And you still need to account for upstream changes.
- TeMPOraL 9y ago
  Forget Github. It might be just a regular git repo on someone's server. IANAL, but could companies really go against a person hosting the source code of a software that scraps a website? Even if, that's probably solvable (by turning it into cat-and-mouse game, TPB/SciHub-style). Have a non-turing-complete set of conversion rules that do scrapping -> standardized-data-format via a general-purpose scrapper. Distribute those through P2P. There might be problems with malicious actors, but generally this should be possible to implement in a way that's mostly transparent for end users. > And you still need to account for upstream changes. Upstream changes will never be too extreme to handle manually, if we're talking scrapping solutions - because any such extreme changes would also piss off the regular browser users.