5 ms·

Using a 'browser' from their base-of-operations in an automated manner would fit the definition of scraping, which is not allowed by Twitter's TOS.

by niftich 10y ago

madeofpalk 10y ago
What are they going to do if they break the TOS? It's not like the developer agreement where they'll shut down your API key - the best they can do is try and blacklist your IP Address
- rdtsc 10y ago
  Sue?
  - bnj 10y ago
    Scrape using an IP address registered to a corporate body located in an unfriendly jurisdiction.
    - drdaeman 10y ago
      Twitter will just block it. Scraping should be done from multiple random addresses from the networks all around the globe, at random intervals. I guess, the only case is voluntary botnet, where users sympathetic to the cause would run limited (so, not prone to abuse) proxies. (To avoid users messing with tweets, just proxy TCP connections to the mothership and let it verify TLS.) However, this would hinder the service, as with the polling model it may not always notice all the tweets.
      - joshmn 10y ago
        luminati.io Zing.
        dottedmag 10y ago
        Luminati shut downs the accounts if the owners of the scrapped websites complain about it.
- mikeryan 10y ago
  See Craigslist vs 3taps. It's not clear if scraping is legal either. https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc.
  - dreamsofdragons 10y ago
    Wow, google must be on some pretty shaky ground then...
    - deleted 10y ago
      [deleted]
    - joshmn 10y ago
      Think about The Internet Archive!
    - sbierwagen 10y ago
      Google/Alphabet is very close to the Obama administration, so it is unlikely they would be prosecuted: https://theintercept.com/2016/04/22/googles-remarkably-close-relationship-with-the-obama-white-house-in-two-charts/ https://theintercept.com/2016/04/22/googles-remarkably-close...
    - comex 10y ago
      No, they're not. To quote from the linked Wikipedia article: > the court held that sending a cease-and-desist letter and enacting an IP address block is sufficient notice of online trespassing, which a plaintiff can use to claim a violation of the Computer Fraud and Abuse Act If someone wants to block Google, they can simply add a robots.txt entry. An IP block would work too. (Sending a cease-and-desist letter might or might not also work, but it doesn't really matter - context and intent matters in court; Google accepting the standard robots.txt as a 'go away' signal should be more than sufficient.)
- syshum 10y ago
  In todays shaking legal ground, it would not shock me if Twitter could not get some Agency in the federal government to Arrest the founders/programmers/executives of the site under CFAA and charge them with criminal felonies
mholt 10y ago
> "in an automated manner" I wonder where that line is drawn. A person using their browser, clicking a link to load a web page initiates a hundred different automated routines[1]. Is that not automation? Is it then automation to have a click initiate another click or two more clicks? Where do you draw the line? [1]: https://github.com/alex/what-happens-when https://github.com/alex/what-happens-when
- JeremyBanks 10y ago
  The spirit of the rule should be self-evident.
  - ominous 10y ago
    And yet it isn't. edit: the spirit of the rule is to grant Twitter control over when to apply the rule. Is that what you meant? Is [F5] + [CTRL+S](for save) automated? Is a browser auto-refresh capability? is `$ watch wget` (or something to that effect) automated? getpocket.com? an IFTTT recipe? A crawler? See what happens?
    - jacquesm 10y ago
      It is. Automated means: without human intervention. So anything that accesses more than one URL without human intervention would violate the TOS. I'm quite amazed that this would have to be spelled out. If you're going to break someone's TOS then at least be aware of it. That helps later on when you have to face the consequences. Just like I'd like to know when I'm speeding, rather than to pretend to be ignorant of the limit. That way I'd know my exposure.
      - ominous 10y ago
        It is like you are happy with how soon you truncate "intervention". Let me know when you find a non human "[F5] + [CTRL+S]". Or link me to the alien crawlers you seem to imply. Are there amazon instances of crawlers owned by non-human entities? Do you consider the "death by bullet" to be caused by humans? By the transfer of momentum of lead? By gunpowder combustion? By human fingers pressing triggers? By time? It time killing humans? Oh no. How long must the causal chain be for you to consider it non human? > [F5] + [CTRL+S] > a browser auto-refresh capability > $ watch wget > getpocket.com > an IFTTT recipe > A crawler I can put all of these in motion. I can also write a macro that triggers them. Is there a difference? Twitter seems to want to be the arbiter of it (if it makes a difference). That is the only spirit of the law: We reserve the right to consider your actions wrong.
010a 10y ago
Which doesn't matter if you are based in a country that doesn't respects US law.
- syshum 10y ago
  If your nation has a extradition Treaty with the US I would be cautious as well... US has been very good of late about charging people that have never been to the US for violation of US laws online. It is bullshit but it seems many nations refuse to protect their citizens from harassment by the US legal system
downandout 10y ago
Perhaps. But it would be up to Twitter to prove such a thing is actually occurring, and it would be difficult or impossible to prove that if setup properly. The information on the site could come from anywhere, since tweets can be viewed even when someone is not logged in. It is worth noting that TMZ and other sites have been storing and detecting deleted celebrity tweets for years, and still do it to this day. If you're not using the API, Twitter has no affirmative way of knowing how you obtained the information.
dingo_bat 10y ago
So if I just visit twitter without logging in, do I have to abide by their ToS?
- 746F7475 10y ago
  You may not do any of the following while accessing or using the Services: (i) access, tamper with, or use non-public areas of the Services, Twitter’s computer systems, or the technical delivery systems of Twitter’s providers; (ii) probe, scan, or test the vulnerability of any system or network or breach or circumvent any security or authentication measures; (iii) access or search or attempt to access or search the Services by any means (automated or otherwise) other than through our currently available, published interfaces that are provided by Twitter (and only pursuant to the applicable terms and conditions), unless you have been specifically allowed to do so in a separate agreement with Twitter (NOTE: crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without the prior consent of Twitter is expressly prohibited); (iv) forge any TCP/IP packet header or any part of the header information in any email or posting, or in any way use the Services to send altered, deceptive or false source-identifying information; or (v) interfere with, or disrupt, (or attempt to do so), the access of any user, host or network, including, without limitation, sending a virus, overloading, flooding, spamming, mail-bombing the Services, or by scripting the creation of Content in such a manner as to interfere with or create an undue burden on the Services. Relevant part: "You may not do any of the following while accessing or using the Services: (...) crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without the prior consent of Twitter is expressly prohibited"
  - dingo_bat 10y ago
    But the thing is, I didn't read or agree to the ToS at all. It was never presented to me and content was presented on my screen when I visited twiiter.com, or followed a link. I am not bound by terms I did not agree to. At least that is my layman logic. But we live in a world where ~50 page long EULAs exist, so layman logic may not be applicable in this case.
deleted 10y ago
[deleted]

Using a 'browser' from their base-of-operations in an automated manner would fit the definition of scraping, which is not allowed by Twitter's TOS.

by niftich 10y ago

Using a 'browser' from their base-of-operations in an automated manner would fit the definition of scraping, which is not allowed by Twitter's TOS.

madeofpalk 10y ago

What are they going to do if they break the TOS? It's not like the developer agreement where they'll shut down your API key - the best they can do is try and blacklist your IP Address

rdtsc 10y ago
Sue?
- bnj 10y ago
  Scrape using an IP address registered to a corporate body located in an unfriendly jurisdiction.
  - drdaeman 10y ago
    Twitter will just block it. Scraping should be done from multiple random addresses from the networks all around the globe, at random intervals. I guess, the only case is voluntary botnet, where users sympathetic to the cause would run limited (so, not prone to abuse) proxies. (To avoid users messing with tweets, just proxy TCP connections to the mothership and let it verify TLS.) However, this would hinder the service, as with the polling model it may not always notice all the tweets.
    - joshmn 10y ago
      luminati.io Zing.
      - dottedmag 10y ago
        Luminati shut downs the accounts if the owners of the scrapped websites complain about it.
mikeryan 10y ago
See Craigslist vs 3taps. It's not clear if scraping is legal either. https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc.
- dreamsofdragons 10y ago
  Wow, google must be on some pretty shaky ground then...
  - deleted 10y ago
    [deleted]
  - joshmn 10y ago
    Think about The Internet Archive!
  - sbierwagen 10y ago
    Google/Alphabet is very close to the Obama administration, so it is unlikely they would be prosecuted: https://theintercept.com/2016/04/22/googles-remarkably-close-relationship-with-the-obama-white-house-in-two-charts/ https://theintercept.com/2016/04/22/googles-remarkably-close...
  - comex 10y ago
    No, they're not. To quote from the linked Wikipedia article: > the court held that sending a cease-and-desist letter and enacting an IP address block is sufficient notice of online trespassing, which a plaintiff can use to claim a violation of the Computer Fraud and Abuse Act If someone wants to block Google, they can simply add a robots.txt entry. An IP block would work too. (Sending a cease-and-desist letter might or might not also work, but it doesn't really matter - context and intent matters in court; Google accepting the standard robots.txt as a 'go away' signal should be more than sufficient.)
syshum 10y ago
In todays shaking legal ground, it would not shock me if Twitter could not get some Agency in the federal government to Arrest the founders/programmers/executives of the site under CFAA and charge them with criminal felonies

mholt 10y ago

> "in an automated manner" I wonder where that line is drawn. A person using their browser, clicking a link to load a web page initiates a hundred different automated routines[1]. Is that not automation? Is it then automation to have a click initiate another click or two more clicks? Where do you draw the line? [1]: https://github.com/alex/what-happens-when https://github.com/alex/what-happens-when

JeremyBanks 10y ago
The spirit of the rule should be self-evident.
- ominous 10y ago
  And yet it isn't. edit: the spirit of the rule is to grant Twitter control over when to apply the rule. Is that what you meant? Is [F5] + [CTRL+S](for save) automated? Is a browser auto-refresh capability? is `$ watch wget` (or something to that effect) automated? getpocket.com? an IFTTT recipe? A crawler? See what happens?
  - jacquesm 10y ago
    It is. Automated means: without human intervention. So anything that accesses more than one URL without human intervention would violate the TOS. I'm quite amazed that this would have to be spelled out. If you're going to break someone's TOS then at least be aware of it. That helps later on when you have to face the consequences. Just like I'd like to know when I'm speeding, rather than to pretend to be ignorant of the limit. That way I'd know my exposure.
    - ominous 10y ago
      It is like you are happy with how soon you truncate "intervention". Let me know when you find a non human "[F5] + [CTRL+S]". Or link me to the alien crawlers you seem to imply. Are there amazon instances of crawlers owned by non-human entities? Do you consider the "death by bullet" to be caused by humans? By the transfer of momentum of lead? By gunpowder combustion? By human fingers pressing triggers? By time? It time killing humans? Oh no. How long must the causal chain be for you to consider it non human? > [F5] + [CTRL+S] > a browser auto-refresh capability > $ watch wget > getpocket.com > an IFTTT recipe > A crawler I can put all of these in motion. I can also write a macro that triggers them. Is there a difference? Twitter seems to want to be the arbiter of it (if it makes a difference). That is the only spirit of the law: We reserve the right to consider your actions wrong.

010a 10y ago

Which doesn't matter if you are based in a country that doesn't respects US law.

syshum 10y ago
If your nation has a extradition Treaty with the US I would be cautious as well... US has been very good of late about charging people that have never been to the US for violation of US laws online. It is bullshit but it seems many nations refuse to protect their citizens from harassment by the US legal system

downandout 10y ago

Perhaps. But it would be up to Twitter to prove such a thing is actually occurring, and it would be difficult or impossible to prove that if setup properly. The information on the site could come from anywhere, since tweets can be viewed even when someone is not logged in. It is worth noting that TMZ and other sites have been storing and detecting deleted celebrity tweets for years, and still do it to this day. If you're not using the API, Twitter has no affirmative way of knowing how you obtained the information.

dingo_bat 10y ago

So if I just visit twitter without logging in, do I have to abide by their ToS?

746F7475 10y ago
You may not do any of the following while accessing or using the Services: (i) access, tamper with, or use non-public areas of the Services, Twitter’s computer systems, or the technical delivery systems of Twitter’s providers; (ii) probe, scan, or test the vulnerability of any system or network or breach or circumvent any security or authentication measures; (iii) access or search or attempt to access or search the Services by any means (automated or otherwise) other than through our currently available, published interfaces that are provided by Twitter (and only pursuant to the applicable terms and conditions), unless you have been specifically allowed to do so in a separate agreement with Twitter (NOTE: crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without the prior consent of Twitter is expressly prohibited); (iv) forge any TCP/IP packet header or any part of the header information in any email or posting, or in any way use the Services to send altered, deceptive or false source-identifying information; or (v) interfere with, or disrupt, (or attempt to do so), the access of any user, host or network, including, without limitation, sending a virus, overloading, flooding, spamming, mail-bombing the Services, or by scripting the creation of Content in such a manner as to interfere with or create an undue burden on the Services. Relevant part: "You may not do any of the following while accessing or using the Services: (...) crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without the prior consent of Twitter is expressly prohibited"
- dingo_bat 10y ago
  But the thing is, I didn't read or agree to the ToS at all. It was never presented to me and content was presented on my screen when I visited twiiter.com, or followed a link. I am not bound by terms I did not agree to. At least that is my layman logic. But we live in a world where ~50 page long EULAs exist, so layman logic may not be applicable in this case.

deleted 10y ago

[deleted]