3 ms·

> Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. Yup, and exposing just a key pie

> Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. Yup, and exposing just a key pieces of information (title, and some of the meta/og tags) without the body would limit the potential for abuse, while still being fairly useful for legitimate uses.

qwerty456127 6y ago
There hardly are any "illegitimate" uses. The web is meant to be machine-readable (we wouldn't have Google or anything nearly as convenient in the first place if it wasn't). Whatever have been published is public and should not come with artificial limitations on how do you read and process it. Blocking crawling should be outlawed as it clearly is a monopolistic practice. E.g. I want to build my own crawler to index and categorize the web subset I choose for me. I believe this is a perfectly legitimate use. But they will probably try to stop me.
- cblconfederate 6y ago
  > Blocking crawling should be outlawed That's overly broad. But maybe it should be illegal to have exceptions only for major monopolies.
- sokoloff 6y ago
  Turn it around at least for a few minutes. Does a website operator have to handle whatever arbitrary traffic you want to throw at them from your crawler? They’re the ones choosing to use tech that’s blocking you. Proposing to make it illegal for them to make that choice or to speak to you differently than they speak to other users of their site may give you some idea of the resistance you’re likely to face to this proposal.
  - samoa42 6y ago
    i think there is a line somewhere along with beeing accountable in a business sense. ie. if your internet host just hands out information, you are free to block/throttle as you please. as soon as you are taking money (operate as a business), you are accountable and must not discriminate. so: > Does a website operator have to handle whatever arbitrary traffic you want to throw at them absolutely, yes!

qwerty456127 6y ago

There hardly are any "illegitimate" uses. The web is meant to be machine-readable (we wouldn't have Google or anything nearly as convenient in the first place if it wasn't). Whatever have been published is public and should not come with artificial limitations on how do you read and process it. Blocking crawling should be outlawed as it clearly is a monopolistic practice. E.g. I want to build my own crawler to index and categorize the web subset I choose for me. I believe this is a perfectly legitimate use. But they will probably try to stop me.

cblconfederate 6y ago
> Blocking crawling should be outlawed That's overly broad. But maybe it should be illegal to have exceptions only for major monopolies.
sokoloff 6y ago
Turn it around at least for a few minutes. Does a website operator have to handle whatever arbitrary traffic you want to throw at them from your crawler? They’re the ones choosing to use tech that’s blocking you. Proposing to make it illegal for them to make that choice or to speak to you differently than they speak to other users of their site may give you some idea of the resistance you’re likely to face to this proposal.
- samoa42 6y ago
  i think there is a line somewhere along with beeing accountable in a business sense. ie. if your internet host just hands out information, you are free to block/throttle as you please. as soon as you are taking money (operate as a business), you are accountable and must not discriminate. so: > Does a website operator have to handle whatever arbitrary traffic you want to throw at them absolutely, yes!