6 ms·

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking <title> tag. They don't seem eager

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking <title> tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.cloudflare.com/t/attention-required-message-when-sharing-link/88999 https://community.cloudflare.com/t/attention-required-messag...

neilparikh 6y ago
> Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. Yup, and exposing just a key pieces of information (title, and some of the meta/og tags) without the body would limit the potential for abuse, while still being fairly useful for legitimate uses.
- qwerty456127 6y ago
  There hardly are any "illegitimate" uses. The web is meant to be machine-readable (we wouldn't have Google or anything nearly as convenient in the first place if it wasn't). Whatever have been published is public and should not come with artificial limitations on how do you read and process it. Blocking crawling should be outlawed as it clearly is a monopolistic practice. E.g. I want to build my own crawler to index and categorize the web subset I choose for me. I believe this is a perfectly legitimate use. But they will probably try to stop me.
  - cblconfederate 6y ago
    > Blocking crawling should be outlawed That's overly broad. But maybe it should be illegal to have exceptions only for major monopolies.
  - sokoloff 6y ago
    Turn it around at least for a few minutes. Does a website operator have to handle whatever arbitrary traffic you want to throw at them from your crawler? They’re the ones choosing to use tech that’s blocking you. Proposing to make it illegal for them to make that choice or to speak to you differently than they speak to other users of their site may give you some idea of the resistance you’re likely to face to this proposal.
    - samoa42 6y ago
      i think there is a line somewhere along with beeing accountable in a business sense. ie. if your internet host just hands out information, you are free to block/throttle as you please. as soon as you are taking money (operate as a business), you are accountable and must not discriminate. so: > Does a website operator have to handle whatever arbitrary traffic you want to throw at them absolutely, yes!
rainingcatndogs 6y ago
And unfortunately, cloudflare is everywhere. This trend will make it even harder for projects like a new search engine to enter the game.
- raverbashing 6y ago
  Because if you don't have it some a-hole will go and ddos your site or you want to prevent a hug-of-death because of reasons. It seems a lot of issues happen because bad players are continued to allowed to thrive, example: everybody uses a big provider because they're the only ones that solved the spam issue.
  - Kaze404 6y ago
    I use Zoho.com and I rarely get spam, if ever.
    - snazz 6y ago
      Zoho isn't Google-size, but it isn't irrelevant, either. Sending mail from a self-hosted email server is far harder since the big providers might put it in spam or drop it even earlier.
    - dzhiurgis 6y ago
      To add to sibling - running your own mail server is the only way to ensure your email is not read by someone else which is so messed up.
      - londons_explore 6y ago
        > running your own mail server is the only way to ensure your email is not read by someone else But any mail you send to someone else probably ends up read by Google/Microsoft anyway, since that's where their mailbox is. Also, email security is a joke. It's 2020, and even TLS encrypted SMTP connections tend not to check for a valid certificate, making them trivial to MITM.
skybrian 6y ago
Is this the case for any web crawler?
- cblconfederate 6y ago
  not sure but it s a very common problem: https://www.google.com/search?q=cloudflare+attention+required https://www.google.com/search?q=cloudflare+attention+require...
bgirard 6y ago
Long term, a new HTTP META method would be interesting. I wonder if something like that has ever been considered. Providers like Cloudflare would hopefully be more lenient with these requests.
- nbadg 6y ago
  Huh. It's certainly an interesting idea! Strictly speaking, individual people could implement this today, since nonstandard HTTP verbs don't break anything that doesn't know to request with them. (It wouldn't be of much use, because clients wouldn't know to use it, but still -- something that could easily be prototyped). I don't think FAAANG (or any other big players) would have much interest in making it happen in the standard though, since it would undercut their big-player advantage.
- fiddlerwoaroof 6y ago
  I wonder if Accept: application/json Would be a reasonable alternative? Wasn't this supposed to be the point of content negotiation?
  - stingraycharles 6y ago
    Maybe, but not really; seems like this thread is more about intent (“I just want a preview”) while content type is more about representation (“I want the content as json”). I can imagine that there will be websites that are actively using the accept parameter to distinguish between “regular visitors” and have their APIs at the same paths (didn’t Reddit do this at some point?), and thus your approach would break in this case. I guess what this is really about is, I hate to say it, but something in the direction of the semantic web, where web servers (and in this case, CloudFlare et al) actually gain a deeper understanding of the content they serve, and a web browser / crawler being able to query this content directly.
    - stickfigure 6y ago
      Accept: text/preview
sergiotapia 6y ago
The cloudflare and google catcha are terrible. It's so bad that at this point I just close the tab if they challenge me with it. I use Brave and always have Shields UP, it seems having it up makes the captchas extremely difficult. Mission accomplished I guess.
dmix 6y ago
I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply). I hope every web admin gives it serious second thought before adopting Cloudflare. Just like for cellphones OS/operator the one thing I’d dream of is a tool that offers a limited set of what Cloudflare does (DDOS protection, hosting privacy layer) but is pro internet and pro privacy. They seem hostile to it in many ways likely because it directly affects their bottom line. The bigger question is whether such a tool could be created without all the downsides. The two I listed I think yes. But their web app security system is overly strict and bad for the internet IMO. And I say that knowing they protect some serious defenders of human rights and face a lot of abuse from the ‘bad guys’. I just wished there was a better middle ground.
- pvg 6y ago
  because both the founder and employees of cloudflare are quite popular users on HN. It seems a lot more likely people aren't finding your argument as convincing as you'd like. Plenty of well-known users (and users who identify their employer) around whose companies' HN-perception fortunes change quite a bit over time.
- hombre_fatal 6y ago
  > But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. I don't think it gets much traction because you're barking up the wrong tree. Also, suggesting that YC is out to silence you and that nobody actually has a counter argument isn't very good for traction, either. Until my website can't get taken off by a $5 rental of an internet-of-shit botnet, Cloudflare gives me and my users recourse against the bad actors of the world. (I also enjoy its host cloaking for my privacy) You simply gloss over bad actors and attack one of the only solutions that works. The biggest threat to the open internet was its naive "there are no bad actors" design, not the people giving us one of the only bulwarks against bad design. I agree with your last sentence that it would be nice to have a better middle ground, but notice that's not the "cloudflare bad" thesis of your comment. The internet needs to be improved so that Cloudflare is redundant. It's not Cloudflare's fault that fundamental design oversights (like optional ISP egress filtering) have created a lucrative niche. And things like faster, unlimited data plans accessible by smart toasters and smart doorbells on top of the internet's naive architecture only entrenches Cloudflare further.
seniorgarcia 6y ago
I don't get what value link previews add. Someone shares a link with me (on skype, slack, teams... whatever) and I care about the content because the person sharing it with me thinks I could/should care about it, or someone shares a link on an aggregator and then I don't think it is too much to ask for that someone to write a summary. If the link is worth sharing writing 1 sentence to explain why isn't too much to ask. What is the value a link preview adds? And why should I, as a content provider care about the value you add? Cloudflare does something for me, what is your service doing for me and why should I whitelist you (or care about you)?
- dasil003 6y ago
  They're sending you traffic. Imagine Twitter or Facebook without link preview, it's much harder to use and overall reduces the change I'll click on a link. Do you think only Twitter and Facebook should be allowed publish previews?
  - einpoklum 6y ago
    > it's much harder to use Actually it's easier to use, in that the preview doesn't take up screen real estate. Perhaps you mean the experience is less pleasant?
  - def_true_false 6y ago
    Half the time the link preview picks the wrong picture and sometimes even the quote. Twitter and Facebook would both be improved by disabling it. Hell, it might even stop people from thinking they need a hero image for their 2 paragraph medium shitpost.
    - input_sh 6y ago
      I'd place that blame towards website owners. Both Facebook and Twitter are pretty open where they read that info from, and an owner can pretty easily pass those fields (it's just some <meta> tags in the <head> element). They also have their own validators: https://cards-dev.twitter.com/validator https://cards-dev.twitter.com/validator and https://developers.facebook.com/tools/debug/ https://developers.facebook.com/tools/debug/ The only issue I'm aware of is that Facebook's crawler breaks about every two months or so.
coderholic 6y ago
At https://host.io https://host.io we scrape every registered domain once a month, and make the meta data available freely over an API. You could use that to get a title for a domain (although not for a URL that's not the main domain), eg: $ curl https://host.io/api/web/facebook.com?token=$TOKEN { "domain": "facebook.com", "rank": 2, "url": "https://www.facebook.com/", "ip": "157.240.11.35", "date": "2020-08-26T17:39:17.981Z", "length": 160817, "encoding": "utf8", "copyright": "Facebook © 2020", "title": "Facebook - Log In or Sign Up", "description": "Create an account or log into Facebook. Connect with friends, family and other people you know. Share photos and videos, send messages and get updates.", "links": [ "messenger.com", "oculus.com" ] } See https://host.io/docs https://host.io/docs for more details about the API and what else you can do with it (eg. find backlinks to domains, domains with the same adsense ID etc)

neilparikh 6y ago

> Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. Yup, and exposing just a key pieces of information (title, and some of the meta/og tags) without the body would limit the potential for abuse, while still being fairly useful for legitimate uses.

qwerty456127 6y ago
There hardly are any "illegitimate" uses. The web is meant to be machine-readable (we wouldn't have Google or anything nearly as convenient in the first place if it wasn't). Whatever have been published is public and should not come with artificial limitations on how do you read and process it. Blocking crawling should be outlawed as it clearly is a monopolistic practice. E.g. I want to build my own crawler to index and categorize the web subset I choose for me. I believe this is a perfectly legitimate use. But they will probably try to stop me.
- cblconfederate 6y ago
  > Blocking crawling should be outlawed That's overly broad. But maybe it should be illegal to have exceptions only for major monopolies.
- sokoloff 6y ago
  Turn it around at least for a few minutes. Does a website operator have to handle whatever arbitrary traffic you want to throw at them from your crawler? They’re the ones choosing to use tech that’s blocking you. Proposing to make it illegal for them to make that choice or to speak to you differently than they speak to other users of their site may give you some idea of the resistance you’re likely to face to this proposal.
  - samoa42 6y ago
    i think there is a line somewhere along with beeing accountable in a business sense. ie. if your internet host just hands out information, you are free to block/throttle as you please. as soon as you are taking money (operate as a business), you are accountable and must not discriminate. so: > Does a website operator have to handle whatever arbitrary traffic you want to throw at them absolutely, yes!

rainingcatndogs 6y ago

And unfortunately, cloudflare is everywhere. This trend will make it even harder for projects like a new search engine to enter the game.

raverbashing 6y ago
Because if you don't have it some a-hole will go and ddos your site or you want to prevent a hug-of-death because of reasons. It seems a lot of issues happen because bad players are continued to allowed to thrive, example: everybody uses a big provider because they're the only ones that solved the spam issue.
- Kaze404 6y ago
  I use Zoho.com and I rarely get spam, if ever.
  - snazz 6y ago
    Zoho isn't Google-size, but it isn't irrelevant, either. Sending mail from a self-hosted email server is far harder since the big providers might put it in spam or drop it even earlier.
  - dzhiurgis 6y ago
    To add to sibling - running your own mail server is the only way to ensure your email is not read by someone else which is so messed up.
    - londons_explore 6y ago
      > running your own mail server is the only way to ensure your email is not read by someone else But any mail you send to someone else probably ends up read by Google/Microsoft anyway, since that's where their mailbox is. Also, email security is a joke. It's 2020, and even TLS encrypted SMTP connections tend not to check for a valid certificate, making them trivial to MITM.

skybrian 6y ago

Is this the case for any web crawler?

cblconfederate 6y ago
not sure but it s a very common problem: https://www.google.com/search?q=cloudflare+attention+required https://www.google.com/search?q=cloudflare+attention+require...

bgirard 6y ago

Long term, a new HTTP META method would be interesting. I wonder if something like that has ever been considered. Providers like Cloudflare would hopefully be more lenient with these requests.

nbadg 6y ago
Huh. It's certainly an interesting idea! Strictly speaking, individual people could implement this today, since nonstandard HTTP verbs don't break anything that doesn't know to request with them. (It wouldn't be of much use, because clients wouldn't know to use it, but still -- something that could easily be prototyped). I don't think FAAANG (or any other big players) would have much interest in making it happen in the standard though, since it would undercut their big-player advantage.
fiddlerwoaroof 6y ago
I wonder if Accept: application/json Would be a reasonable alternative? Wasn't this supposed to be the point of content negotiation?
- stingraycharles 6y ago
  Maybe, but not really; seems like this thread is more about intent (“I just want a preview”) while content type is more about representation (“I want the content as json”). I can imagine that there will be websites that are actively using the accept parameter to distinguish between “regular visitors” and have their APIs at the same paths (didn’t Reddit do this at some point?), and thus your approach would break in this case. I guess what this is really about is, I hate to say it, but something in the direction of the semantic web, where web servers (and in this case, CloudFlare et al) actually gain a deeper understanding of the content they serve, and a web browser / crawler being able to query this content directly.
  - stickfigure 6y ago
    Accept: text/preview

sergiotapia 6y ago

The cloudflare and google catcha are terrible. It's so bad that at this point I just close the tab if they challenge me with it. I use Brave and always have Shields UP, it seems having it up makes the captchas extremely difficult. Mission accomplished I guess.

dmix 6y ago

I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply). I hope every web admin gives it serious second thought before adopting Cloudflare. Just like for cellphones OS/operator the one thing I’d dream of is a tool that offers a limited set of what Cloudflare does (DDOS protection, hosting privacy layer) but is pro internet and pro privacy. They seem hostile to it in many ways likely because it directly affects their bottom line. The bigger question is whether such a tool could be created without all the downsides. The two I listed I think yes. But their web app security system is overly strict and bad for the internet IMO. And I say that knowing they protect some serious defenders of human rights and face a lot of abuse from the ‘bad guys’. I just wished there was a better middle ground.

pvg 6y ago
because both the founder and employees of cloudflare are quite popular users on HN. It seems a lot more likely people aren't finding your argument as convincing as you'd like. Plenty of well-known users (and users who identify their employer) around whose companies' HN-perception fortunes change quite a bit over time.
hombre_fatal 6y ago
> But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. I don't think it gets much traction because you're barking up the wrong tree. Also, suggesting that YC is out to silence you and that nobody actually has a counter argument isn't very good for traction, either. Until my website can't get taken off by a $5 rental of an internet-of-shit botnet, Cloudflare gives me and my users recourse against the bad actors of the world. (I also enjoy its host cloaking for my privacy) You simply gloss over bad actors and attack one of the only solutions that works. The biggest threat to the open internet was its naive "there are no bad actors" design, not the people giving us one of the only bulwarks against bad design. I agree with your last sentence that it would be nice to have a better middle ground, but notice that's not the "cloudflare bad" thesis of your comment. The internet needs to be improved so that Cloudflare is redundant. It's not Cloudflare's fault that fundamental design oversights (like optional ISP egress filtering) have created a lucrative niche. And things like faster, unlimited data plans accessible by smart toasters and smart doorbells on top of the internet's naive architecture only entrenches Cloudflare further.

seniorgarcia 6y ago

I don't get what value link previews add. Someone shares a link with me (on skype, slack, teams... whatever) and I care about the content because the person sharing it with me thinks I could/should care about it, or someone shares a link on an aggregator and then I don't think it is too much to ask for that someone to write a summary. If the link is worth sharing writing 1 sentence to explain why isn't too much to ask. What is the value a link preview adds? And why should I, as a content provider care about the value you add? Cloudflare does something for me, what is your service doing for me and why should I whitelist you (or care about you)?

dasil003 6y ago
They're sending you traffic. Imagine Twitter or Facebook without link preview, it's much harder to use and overall reduces the change I'll click on a link. Do you think only Twitter and Facebook should be allowed publish previews?
- einpoklum 6y ago
  > it's much harder to use Actually it's easier to use, in that the preview doesn't take up screen real estate. Perhaps you mean the experience is less pleasant?
- def_true_false 6y ago
  Half the time the link preview picks the wrong picture and sometimes even the quote. Twitter and Facebook would both be improved by disabling it. Hell, it might even stop people from thinking they need a hero image for their 2 paragraph medium shitpost.
  - input_sh 6y ago
    I'd place that blame towards website owners. Both Facebook and Twitter are pretty open where they read that info from, and an owner can pretty easily pass those fields (it's just some <meta> tags in the <head> element). They also have their own validators: https://cards-dev.twitter.com/validator https://cards-dev.twitter.com/validator and https://developers.facebook.com/tools/debug/ https://developers.facebook.com/tools/debug/ The only issue I'm aware of is that Facebook's crawler breaks about every two months or so.

coderholic 6y ago

At https://host.io https://host.io we scrape every registered domain once a month, and make the meta data available freely over an API. You could use that to get a title for a domain (although not for a URL that's not the main domain), eg: $ curl https://host.io/api/web/facebook.com?token=$TOKEN { "domain": "facebook.com", "rank": 2, "url": "https://www.facebook.com/", "ip": "157.240.11.35", "date": "2020-08-26T17:39:17.981Z", "length": 160817, "encoding": "utf8", "copyright": "Facebook © 2020", "title": "Facebook - Log In or Sign Up", "description": "Create an account or log into Facebook. Connect with friends, family and other people you know. Share photos and videos, send messages and get updates.", "links": [ "messenger.com", "oculus.com" ] } See https://host.io/docs https://host.io/docs for more details about the API and what else you can do with it (eg. find backlinks to domains, domains with the same adsense ID etc)