4 ms·

A curious title. "So you want to scrape like the unethical boys?" I guess doesn't scan so well. Bad boys maybe? I'm pretty sure Internet Archive, etc don't in

A curious title. "So you want to scrape like the unethical boys?" I guess doesn't scan so well. Bad boys maybe? I'm pretty sure Internet Archive, etc don't in fact misrepresent what they are to crawl websites...

echelon 3y ago
> unethical Using and transforming information in useful ways is unethical if it results in a profit? That's what our brains do, too.
- llamaimperative 3y ago
  No, destroying incentives to produce and share information is unethical (and more importantly, self-defeating). Brains that consume information don’t destroy that incentive, they produce it. Intermediating that and capturing all of the value for yourself is the unethical part, just like all forms of rent-seeking.
  - echelon 3y ago
    > destroying incentives Internet usage and content creation are increasing, not decreasing. I continue to publish comments, code, and images that presumably get used to train models. My incentive hasn't been destroyed. > rent-seeking Supply and demand set the prices. Subscription services provide value and continue to invest in their product, catalog, and/or service. Property owners handle asset ownership and upkeep problems at scale. Inefficiencies will be met with competition, and businesses not providing value will be out-competed. Data under-availability is an inefficiency holding us back from bigger and better things.
vasco 3y ago
Tell that to the most used website in the world, which is basically a scrapping-and-sorting machine.
- blantonl 3y ago
  I can commit a code change in 2 seconds that would directly tell the most used website in the world to stop scrapping and sorting my data, and they would honor it and that would be the end of that. I'm under no illusions that they would or would not honor that in the future, but that's the state today.
CoastalCoder 3y ago
> "So you want to scrape like the unethical boys?" What's considered ethical is a very debated topic. An assertion that something is simply "unethical" should be seen as the starting point of a discussion, not as a self-evident fact.
- marginalia_nu 3y ago
  If someone tells you to go away via the robots exclusion standard, and puts up bot mitigation to prevent you, blocks your IPs, etc. then clearly you do not have their consent to help yourself to the data. I find it really hard to see how you could twist ignoring this clear lack of consent, and going to great lengths to circumvent what was clearly put into place to prevent you from doing the very thing you are doing, how you could twist that into an ethical action. It may or may not be technically illegal to do, you're but that is not a statement about what is ethical.
  - lambdaba 3y ago
    Surely the ethics are more complicated then just following robots.txt or not. The intended usage counts, and that isn't captured in robots.txt.
    - marginalia_nu 3y ago
      If you have a noble intent, you ask the webmaster for permission to use the data. Surely if they agree with your assessment that your intent is indeed noble, then you'll be given consent. I run a search engine and an internet crawler. I do this all the time. To this date I've never had a webmaster that didn't permit my crawler access when I've asked nicely.
      - bryanrasmussen 3y ago
        If you have a noble intent - identify members of fascist organizations - then obviously when you ask the top online fascist sites if you may scrape them to build up your list of online fascists - they will say no. OK less provocative, you have new algorithm to identify inaccessible websites, your automation is scary good, crawling a site you can identify many issues that most sites would have to pay for a full audit to get, but now these sites have problems - if you can identify their sites as being inaccessible then they have to fix these problems due to various accessibility standards that apply in the regions they operate in. But if they don't allow you access then they can maybe make an argument they are accessible due to audit they did last year, at any rate they don't want to be forced to spend money on accessibility issues right now which it sounds like they might have to if they let you crawl their site. Version 2 of above, some years ago I spoke about a job with a big time magazine publisher in Denmark and said one of the things that would make me a good employee is my knowledge of accessibility and their chief of development said they didn't have anyone with disabilities that used their site - so if I ask that guy to crawl their site why say yes? They have no users that would benefit!! Stop abusing our bandwidth bleeding heart guy.

echelon 3y ago

> unethical Using and transforming information in useful ways is unethical if it results in a profit? That's what our brains do, too.

llamaimperative 3y ago
No, destroying incentives to produce and share information is unethical (and more importantly, self-defeating). Brains that consume information don’t destroy that incentive, they produce it. Intermediating that and capturing all of the value for yourself is the unethical part, just like all forms of rent-seeking.
- echelon 3y ago
  > destroying incentives Internet usage and content creation are increasing, not decreasing. I continue to publish comments, code, and images that presumably get used to train models. My incentive hasn't been destroyed. > rent-seeking Supply and demand set the prices. Subscription services provide value and continue to invest in their product, catalog, and/or service. Property owners handle asset ownership and upkeep problems at scale. Inefficiencies will be met with competition, and businesses not providing value will be out-competed. Data under-availability is an inefficiency holding us back from bigger and better things.

vasco 3y ago

Tell that to the most used website in the world, which is basically a scrapping-and-sorting machine.

blantonl 3y ago
I can commit a code change in 2 seconds that would directly tell the most used website in the world to stop scrapping and sorting my data, and they would honor it and that would be the end of that. I'm under no illusions that they would or would not honor that in the future, but that's the state today.

CoastalCoder 3y ago

> "So you want to scrape like the unethical boys?" What's considered ethical is a very debated topic. An assertion that something is simply "unethical" should be seen as the starting point of a discussion, not as a self-evident fact.

marginalia_nu 3y ago
If someone tells you to go away via the robots exclusion standard, and puts up bot mitigation to prevent you, blocks your IPs, etc. then clearly you do not have their consent to help yourself to the data. I find it really hard to see how you could twist ignoring this clear lack of consent, and going to great lengths to circumvent what was clearly put into place to prevent you from doing the very thing you are doing, how you could twist that into an ethical action. It may or may not be technically illegal to do, you're but that is not a statement about what is ethical.
- lambdaba 3y ago
  Surely the ethics are more complicated then just following robots.txt or not. The intended usage counts, and that isn't captured in robots.txt.
  - marginalia_nu 3y ago
    If you have a noble intent, you ask the webmaster for permission to use the data. Surely if they agree with your assessment that your intent is indeed noble, then you'll be given consent. I run a search engine and an internet crawler. I do this all the time. To this date I've never had a webmaster that didn't permit my crawler access when I've asked nicely.
    - bryanrasmussen 3y ago
      If you have a noble intent - identify members of fascist organizations - then obviously when you ask the top online fascist sites if you may scrape them to build up your list of online fascists - they will say no. OK less provocative, you have new algorithm to identify inaccessible websites, your automation is scary good, crawling a site you can identify many issues that most sites would have to pay for a full audit to get, but now these sites have problems - if you can identify their sites as being inaccessible then they have to fix these problems due to various accessibility standards that apply in the regions they operate in. But if they don't allow you access then they can maybe make an argument they are accessible due to audit they did last year, at any rate they don't want to be forced to spend money on accessibility issues right now which it sounds like they might have to if they let you crawl their site. Version 2 of above, some years ago I spoke about a job with a big time magazine publisher in Denmark and said one of the things that would make me a good employee is my knowledge of accessibility and their chief of development said they didn't have anyone with disabilities that used their site - so if I ask that guy to crawl their site why say yes? They have no users that would benefit!! Stop abusing our bandwidth bleeding heart guy.