3 ms·

> Naturally, they're going to say "web scraping uses resources, stop it!" that's the expected cost of publishing something to the public internet. People are g

> Naturally, they're going to say "web scraping uses resources, stop it!" that's the expected cost of publishing something to the public internet. People are going to access it. No one has a right to complain when people access something that was put there for the public to see. Scrappers can be dicks about it too, they can get lazy and endlessly hammer away at some server or repeatedly pull down the same content because they messed up, but we don't need need litigation for that. If something raises to the level of DoS that's already covered under existing laws. > it's completely in keeping with what amoral corporations locked in constant battle would be expected to do: maximize benefits to themselves while minimizing benefits to others. Maybe we need to rethink giving some of these corporations the privilege of corporate personhood if they are just going to make things worse for everyone else while only enriching themselves. We don't need to allow parasites and pillagers to take whatever they want at our expense.

mrkeen 3y ago
> Scrappers can be dicks about it too It's not always about individual bad actors. You can have lots of small players causing problems. I wonder how many python developers there are right now trying to make their own offline copy of stackoverflow.com. Wikipedia has a great defence against this. They ask you not to scrape, and at the same time, provide torrents of the data (https://meta.wikimedia.org/wiki/Data_dump_torrents https://meta.wikimedia.org/wiki/Data_dump_torrents)
- rzzzt 3y ago
  Stack Exchange also provides one: https://archive.org/details/stackexchange https://archive.org/details/stackexchange There was a hiccup around June but that seems resolved now: https://meta.stackexchange.com/questions/389922/june-2023-data-dump-is-missing https://meta.stackexchange.com/questions/389922/june-2023-da...
  - genewitch 3y ago
    yeah i thought this was the case. There's a couple of other sites that - i hesitate to state this completely factual phrase - leave dumps for the public to pick up.
SubjectToChange 3y ago
that's the expected cost of publishing something to the public internet. People are going to access it. No one has a right to complain when people access something that was put there for the public to see. What is your opinion on spam email?
- Asooka 3y ago
  I hold the same view as the parent poster on public data online and my opinion on spam e-mail is that it's a consequence of naivete bordering on faulty design. It should have been set up with strong authentication (proof that this e-mail is from whom it says it is) and explicit consent (you can only message me if I allow you to message me). The latter could be as simple as rate-limiting e-mail from addresses which you have not explicitly allowed, or to which you have not previously sent letters, to one per week or so, with all such e-mails automatically going to the spam folder to be purged in a month.
- autoexec 3y ago
  Spam is a problem, but a very different one. Spam is often malicious/harmful. Accessing a publicly accessible website is usually benign even if the content being accessed gets saved to disk. Some people don't publicly publish their email address, they instead selectively give it out only to those they want to get email from, but their address gets leaked/sold and abused. Ultimately people who do publicly publish a contact address (email or even a physical mailing address) are basically on the hook for deciding what to do with whatever people send them. The spam situation got out of hand pretty fast though. The only thing that kept email spam from reaching the level of a DoS attack were blacklists and server-side filtering, and even with those things (plus client-side filters) spam is still a huge problem today. Spam is just a much bigger problem than web scraping. Even the junkmail the mailman delivers to my door has an environmental cost that's much worse than the "harm" of a web scraper's http GET requests. We have many alternative ways to contact each other online that aren't as vulnerable to spam, but for all of its shortcomings email continues to be widely used because at the end of the day people think giving strangers the ability to reach out to them uninvited is valuable. Anyone can set up a whitelist and trash everything that comes into their mailbox unless it's from an approved sender, but almost nobody does because they want to be more reachable than that.

mrkeen 3y ago

> Scrappers can be dicks about it too It's not always about individual bad actors. You can have lots of small players causing problems. I wonder how many python developers there are right now trying to make their own offline copy of stackoverflow.com. Wikipedia has a great defence against this. They ask you not to scrape, and at the same time, provide torrents of the data (https://meta.wikimedia.org/wiki/Data_dump_torrents https://meta.wikimedia.org/wiki/Data_dump_torrents)

rzzzt 3y ago
Stack Exchange also provides one: https://archive.org/details/stackexchange https://archive.org/details/stackexchange There was a hiccup around June but that seems resolved now: https://meta.stackexchange.com/questions/389922/june-2023-data-dump-is-missing https://meta.stackexchange.com/questions/389922/june-2023-da...
- genewitch 3y ago
  yeah i thought this was the case. There's a couple of other sites that - i hesitate to state this completely factual phrase - leave dumps for the public to pick up.

SubjectToChange 3y ago

that's the expected cost of publishing something to the public internet. People are going to access it. No one has a right to complain when people access something that was put there for the public to see. What is your opinion on spam email?

Asooka 3y ago
I hold the same view as the parent poster on public data online and my opinion on spam e-mail is that it's a consequence of naivete bordering on faulty design. It should have been set up with strong authentication (proof that this e-mail is from whom it says it is) and explicit consent (you can only message me if I allow you to message me). The latter could be as simple as rate-limiting e-mail from addresses which you have not explicitly allowed, or to which you have not previously sent letters, to one per week or so, with all such e-mails automatically going to the spam folder to be purged in a month.
autoexec 3y ago
Spam is a problem, but a very different one. Spam is often malicious/harmful. Accessing a publicly accessible website is usually benign even if the content being accessed gets saved to disk. Some people don't publicly publish their email address, they instead selectively give it out only to those they want to get email from, but their address gets leaked/sold and abused. Ultimately people who do publicly publish a contact address (email or even a physical mailing address) are basically on the hook for deciding what to do with whatever people send them. The spam situation got out of hand pretty fast though. The only thing that kept email spam from reaching the level of a DoS attack were blacklists and server-side filtering, and even with those things (plus client-side filters) spam is still a huge problem today. Spam is just a much bigger problem than web scraping. Even the junkmail the mailman delivers to my door has an environmental cost that's much worse than the "harm" of a web scraper's http GET requests. We have many alternative ways to contact each other online that aren't as vulnerable to spam, but for all of its shortcomings email continues to be widely used because at the end of the day people think giving strangers the ability to reach out to them uninvited is valuable. Anyone can set up a whitelist and trash everything that comes into their mailbox unless it's from an approved sender, but almost nobody does because they want to be more reachable than that.