4 ms·

Scraping isn’t illegal, and to be honest, I’m not even sure it’s unethical. I’m assuming you think it so — if so, why? I’m not disagreeing, but haven’t given it

Scraping isn’t illegal, and to be honest, I’m not even sure it’s unethical. I’m assuming you think it so — if so, why? I’m not disagreeing, but haven’t given it much thought.

pmlnr 3y ago
> I’m not even sure it’s unethical. If it doesn't respect robots.txt, it is unethical.
- dizhn 3y ago
  No means no ! :)
- threeseed 3y ago
  a) It looks to be permitted according to Twitter's robots.txt b) Given Twitter is public, user generated content which they don't own but simply have a license I wouldn't call it unethical in the slightest.
  - AmericanChopper 3y ago
    If the background of the issue is as Musk described, then it certainly is not allowed by twitter’s robots.txt, which allows a maximum of one request per second. I do a lot of data scraping, so I’m sympathetic to the people who want to do it, but violating the robots.txt (or other published policies) is absolutely unethical, regardless of the license of the content the service is hosting. Another way of describing an unauthorised usecase taking a service offline is a denial of service attack, which (again, if Musk’s description of the problem is accurate) seems to be the issue Twitter was facing, with a choice between restricting services or scaling forever to meet the scrapers requirements. Personally I would have probably tried to start with a captcha, but all this dogpiling just looks like low effort Musk hate. The prevailing sentiment on HN has become so passionately anti-Musk that it’s hard to view any criticism of him or Twitter here with any credibility.
- echelon 3y ago
  "You wouldn't download a car." The only reason these websites and platforms aggregate any content at all is because they're effectively giant public squares.
- glimmung 3y ago
  Is it ethical for the "public square" to have a robots.txt? Musk is trying to have his cake and eat it... (Clearly it's not a public square, but his position is incoherent).
  - rnbaxter 3y ago
    Yes, it is ethical. In many countries it is legal for humans to walk around the public square and overhear all conversations. It is NOT legal to install cameras that record everyone's conversations, much less sell the laundered results. Pre-2023 people went on Twitter with the expectation that their output would be read by humans. A traditional search engine is different: It redirects to the original. A bastardized search engine that shows snippets is more questionable, but still miles away from the AI steal.
    - RobotToaster 3y ago
      Many countries have freedom of panorama, which means it is legal to video record the public square. I'm not aware if anywhere has specific laws on mounting the camera on a robot.
    - johnnyanmac 3y ago
      >Pre-2023 people went on Twitter with the expectation that their output would be read by huma ns Expectations =/= reality. And the reality is that bits have been reading comments for over a decade.
wiseowise 3y ago
It’s ethical when your average Joe does it on a small scale to scrap their favorite favorite YouTuber or to buy something when it becomes available. When you have financial incentive to build your business on someone’s data and you scrap literally millions if not billions of pages - it’s unethical.
- specproc 3y ago
  The thing with social media platforms is that this data is user-generated, so you've got the company "owning" user content. This data is often of great public value. I track conversations around a social issue as part of my work for a non-profit. I'd counter it's unethical to prevent people from accessing this data.
  - jnsaff2 3y ago
    I’m not disagreeing with your comment but > great public value Having been to twitter mostly through the most recent prominent war, man the signal to noise ratio is really low even when being careful about who to follow and who to block. There is so much disinformation, bad takes, uninformed opinions presented as facts, pure evil, etc. So I guess it could be used for training very specific things or cataloging the underbelly of humanity but for general human knowledge it’s a frigging cesspool.
    - specproc 3y ago
      OK, not gonna argue with that. There is, I guess, a perception that it matters because policy-makers, and the wonks and hacks that influence them are hooked. The value for me (and ergo the public, some classic NGO thinking there for you) lies in understanding those dynamics. I do not use the Twitters myself, and actively discourage others from doing so. Sends people bonkers.
    - lmeyerov 3y ago
      I mean, we have found election manipulations like large-scale inauthentic activity of out-of-staters explicitly targeting African Americans, and projects here even to the extent of the perpetrators getting indicted. Other projects were tracking vaccine side-effect self-reports faster than the CDC and other disaster intelligence. We were actually gearing up to switch to paid accounts as we found use cases that could subsidize these efforts... And then the starting price for reasonably small volumes shot up to like $500k/yr.
develop7 3y ago
Moreover, isn't making scraping impossible illegal per a couple-of-years-old bill?

Scraping isn’t illegal, and to be honest, I’m not even sure it’s unethical. I’m assuming you think it so — if so, why? I’m not disagreeing, but haven’t given it much thought.

pmlnr 3y ago

> I’m not even sure it’s unethical. If it doesn't respect robots.txt, it is unethical.

dizhn 3y ago
No means no ! :)
threeseed 3y ago
a) It looks to be permitted according to Twitter's robots.txt b) Given Twitter is public, user generated content which they don't own but simply have a license I wouldn't call it unethical in the slightest.
- AmericanChopper 3y ago
  If the background of the issue is as Musk described, then it certainly is not allowed by twitter’s robots.txt, which allows a maximum of one request per second. I do a lot of data scraping, so I’m sympathetic to the people who want to do it, but violating the robots.txt (or other published policies) is absolutely unethical, regardless of the license of the content the service is hosting. Another way of describing an unauthorised usecase taking a service offline is a denial of service attack, which (again, if Musk’s description of the problem is accurate) seems to be the issue Twitter was facing, with a choice between restricting services or scaling forever to meet the scrapers requirements. Personally I would have probably tried to start with a captcha, but all this dogpiling just looks like low effort Musk hate. The prevailing sentiment on HN has become so passionately anti-Musk that it’s hard to view any criticism of him or Twitter here with any credibility.
echelon 3y ago
"You wouldn't download a car." The only reason these websites and platforms aggregate any content at all is because they're effectively giant public squares.
glimmung 3y ago
Is it ethical for the "public square" to have a robots.txt? Musk is trying to have his cake and eat it... (Clearly it's not a public square, but his position is incoherent).
- rnbaxter 3y ago
  Yes, it is ethical. In many countries it is legal for humans to walk around the public square and overhear all conversations. It is NOT legal to install cameras that record everyone's conversations, much less sell the laundered results. Pre-2023 people went on Twitter with the expectation that their output would be read by humans. A traditional search engine is different: It redirects to the original. A bastardized search engine that shows snippets is more questionable, but still miles away from the AI steal.
  - RobotToaster 3y ago
    Many countries have freedom of panorama, which means it is legal to video record the public square. I'm not aware if anywhere has specific laws on mounting the camera on a robot.
  - johnnyanmac 3y ago
    >Pre-2023 people went on Twitter with the expectation that their output would be read by huma ns Expectations =/= reality. And the reality is that bits have been reading comments for over a decade.

wiseowise 3y ago

It’s ethical when your average Joe does it on a small scale to scrap their favorite favorite YouTuber or to buy something when it becomes available. When you have financial incentive to build your business on someone’s data and you scrap literally millions if not billions of pages - it’s unethical.

specproc 3y ago
The thing with social media platforms is that this data is user-generated, so you've got the company "owning" user content. This data is often of great public value. I track conversations around a social issue as part of my work for a non-profit. I'd counter it's unethical to prevent people from accessing this data.
- jnsaff2 3y ago
  I’m not disagreeing with your comment but > great public value Having been to twitter mostly through the most recent prominent war, man the signal to noise ratio is really low even when being careful about who to follow and who to block. There is so much disinformation, bad takes, uninformed opinions presented as facts, pure evil, etc. So I guess it could be used for training very specific things or cataloging the underbelly of humanity but for general human knowledge it’s a frigging cesspool.
  - specproc 3y ago
    OK, not gonna argue with that. There is, I guess, a perception that it matters because policy-makers, and the wonks and hacks that influence them are hooked. The value for me (and ergo the public, some classic NGO thinking there for you) lies in understanding those dynamics. I do not use the Twitters myself, and actively discourage others from doing so. Sends people bonkers.
  - lmeyerov 3y ago
    I mean, we have found election manipulations like large-scale inauthentic activity of out-of-staters explicitly targeting African Americans, and projects here even to the extent of the perpetrators getting indicted. Other projects were tracking vaccine side-effect self-reports faster than the CDC and other disaster intelligence. We were actually gearing up to switch to paid accounts as we found use cases that could subsidize these efforts... And then the starting price for reasonably small volumes shot up to like $500k/yr.

develop7 3y ago

Moreover, isn't making scraping impossible illegal per a couple-of-years-old bill?