11 ms·

Https hurts users far away from the server

121 points by antoinefink 10y ago

sp332 10y ago
If you're worried about a proprietary solution, you could host your own cache server in Australia or wherever your customers are having trouble.
- problems 10y ago
  Yeah, at a $200/mo cost, you could spin up a few VMs on DigitalOcean, Vultr or LightSail which have decent bandwidth and cache from there. Nice part about cloudflare though is that they can use anycast to determine location and then send the closest server IPs. For sub-$200/mo, you're not able to do that, you'd have to find a provider that could do it for you, I'm not sure anyone offers country-based anycast DNS alone. EDIT: Looks like easyDNS enterprise may be able to do it, https://fusion.easydns.com/Knowledgebase/Article/View/214/7/geodns https://fusion.easydns.com/Knowledgebase/Article/View/214/7/... for about $12.75/mo too. Might be a decent way to brew your own mini caching CDN for fairly cheap.
  - jdub 10y ago
    You can also use Route 53 for the same purpose, for a tiny premium on the standard rates for name resolution. (See latency based routing queries and geo DNS queries below, plus health checks for failover.) https://aws.amazon.com/route53/pricing/ https://aws.amazon.com/route53/pricing/
  - manigandham 10y ago
    > anycast to determine location and then send the closest server IPs Anycast doesn't determine location or send the closet IPs, it's all the same IP address announced using BGP (border gateway protocol) to automatically route to the closest (in network travel) server.
    - problems 10y ago
      Of course. Let me clarify - they use anycast DNS to send the closest CF caching proxy's IP.
      - manigandham 10y ago
        That's not how it works. Both their DNS and reverse proxy servers use anycast IPs without any DNS-based routing. They did recently release some features called traffic manager that lets you control the origin server based on geo. If you just need geo-balanced DNS though, AWS Route 53, Azure Traffic Manager, and NSOne offer DNS based routing.
        problems 10y ago
        Really? I didn't think they'd do anycast on their reverse proxy servers, that seems risky to me (ie: a TCP connection changes from one server to another due to a BGP change), but I suppose the odds are fairly low. I seem to remember getting different IPs from different locations, but it could just be random or I could be mistaken. EDIT: Tried now and it seems I'm getting the same IPs from Canada and Australia, so you are indeed correct.
        manigandham 10y ago
        Yes, they have a big address space and announce all ips from every location: https://blog.cloudflare.com/cloudflares-architecture-eliminating-single-p/ https://blog.cloudflare.com/cloudflares-architecture-elimina... Pretty much all major CDNs use anycast today for load balancing, rolling downtime and security/ddos protection. Http/tcp connections are usually short-lived, relatively cheap to setup (since it's an edge network anyway) and BGP route updates don't happen that often. LinkedIn switched to anycast too after testing: https://engineering.linkedin.com/network-performance/tcp-over-ip-anycast-pipe-dream-or-reality https://engineering.linkedin.com/network-performance/tcp-ove...
alvil 10y ago
There is also another problem on how much and how often is Googlebot indexing your site because your site speed is one of the factors of so called Google index budget. My users are in Germany so my VPS is also in Germany to be fast for local user (~130ms for http reply), but for US Googlebot is my site slow (~420ms for http reply). So you are penalized also for this.
- kijin 10y ago
  Isn't this somewhat compensated for by the extra credit Google gives you for using SSL? I would also assume that Google is smart enough to take the physical location of your server into account when calculating how much penalty to apply in which searches. Sites that load fast in Germany should have higher ranks in searches from Germany.
- KabuseCha 10y ago
  Hi - I know some tools report a slow site in this case, but these reports are not accurate - don't believe them! Google is not that stupid ! :D I am currently working as a dev in an SEO-Agency (in Austria), and we never believed this hypothesis - so we tested this once with a bunch of our sites: When moving sites with a German speaking audience to a VPS in America, your rankings at google.de/google.at will decrease (slightly - the effect is not that big) - the other way around your rankings will improve (slightly). However - even if your rankings would improve when moving to America I would recommend keeping your sites hosted in Europe: The increase in rankings will not offset the decrease in user satisfaction and therefore the decline in your conversion rates.
  - jacquesm 10y ago
    The whole idea that search engines should take preference over actual users is strange.
  - chatmasta 10y ago
    That sounds like a really interesting experiment. Kudos for taking a scientific approach to SEO. A bit off-topic, but out of curiosity, have you run any other interesting experiments like this? I would love to read a blog post about them.
Kiro 10y ago
I don't understand why I need to use https on a static marketing webpage. No login stuff, no JavaScript, nothing. Just straight up HTML and CSS. Right now I need to pay about $150 every year for something that's only used to satisfy Google PageRank (I can't use LetsEncrypt with my hosting provider). Why?
- Nullabillity 10y ago
  Yeah, why are you using a bad hosting provider?
  - Kiro 10y ago
    Legacy and one of the only hosting providers in my country. I don't want to risk worse localized SEO by hosting it outside.
    - RossM 10y ago
      I've been reading up on localisation SEO lately - as far as I understand, Google only uses "server IP is located in X country" as an indicator of where a site might be localised for, if it can't get any better information. For example, if you're using a ccTLD for the domain, or if it's a generic-TLD and you declare a country in Google Webmaster Tools, that will be a much stronger weighting. Of course, if that's wrong I'd love to know!
      - Kiro 10y ago
        Thank you. The actual web app is hosted on Linode in Frankfurt (using LetsEncrypt for https) so maybe I should host the marketing page there as well if that's true.
  - djsumdog 10y ago
    When I was in the EU, I saw both Vodaphone and Three inject banners on the top of websites in various countries. They're not as easy to get away from as you think.
- 10y ago
chatmasta 10y ago
What's wrong with the cloudflare free plan? You can host a static site on github pages with a custom domain and use the free cloudflare SSL cert.
- andrewaylett 10y ago
  Without Railgun, there's no guarantee that the CloudFlare nodes will have an open socket to your origin server, so your visitors may still have to pay the cost of the round-trip.
- amiraliakbari 10y ago
  The "Railgun" feature mentioned in the article is only available in some paid plans. Using the free plan wouldn't keep an open connection between your servers and Cloudflare's. It does improve the situation by terminating users' handshakes early, using better links, warm DNS cache, etc. among servers. But the latency hard limit is still present between your server and CF. Skipping https between your server and CL is not an option either for any site transferring user data.
  - chatmasta 10y ago
    Ah, I see. I did not realize that. Accordingly, I edited my comment to be less inflammatory. :) I understand that by using the generic CF free cert, https terminates at CF and the connection CF->Origin is over unencrypted HTTP. Is this why there is latency overhead? Because CF cannot connect to origin via https so it cannot open a persistent tunnel? Or is it because the overhead of keeping an open https tunnel per origin server is prohibitively expensive to maintain for every free customer? I assume that even though there is no persistent tunnel, CF still must still use persistent TCP connections?
    - floatboth 10y ago
      Maybe the cost is not completely prohibitively expensive but they do consider it a premium feature. Have to earn money :D
filleokus 10y ago
A related interesting topic is the possibility of secure cache servers that don't break the secure channel with "blind caches". Currently just a RFC draft and probably a long time from mass adoption, but nevertheless interesting. https://tools.ietf.org/html/draft-thomson-http-bc-00 https://tools.ietf.org/html/draft-thomson-http-bc-00, and Ericsson's article on it https://www.ericsson.com/thecompany/our_publications/ericsson_technology_review/archive/blind-cache https://www.ericsson.com/thecompany/our_publications/ericsso...
nprescott 10y ago
I really enjoyed the coverage of the same topic in High Performance Browser Networking[0]. It effectively explains the key performance influencers across various networks without being boring. [0]: https://hpbn.co/ https://hpbn.co/
andreareina 10y ago
One round trip over the course of the time that the user is using the same OS/browser installation isn't much. The Cloudflare Railgun is an interesting solution, and one that could be implemented in the context of an SPA over a websockets connection. Or conceivably some other consumer of an API.
hannob 10y ago
There are a couple of other things you can do with existing TLS technology that can improve your latency, e.g. using OCSP stapling, use modern crypto so browsers may use TLS false start, avoid too many ciphers or unnecessary certs in the chain to make the handshake smaller. It's a bit older, but here's some info, much of it is still valid: https://istlsfastyet.com/ https://istlsfastyet.com/
- citrin_ru 10y ago
  It is questionable if OCSP stapling reduces TLS handshake time. Without OCSP browser makes slow request to CA, but caches results for a long time so slow request happens not often. With OCSP stapling enabled more data is transferred between client and server on each TLS handshake. Main proponents of OCSP stapling are CA, because it saves them bandwidth/hardware.
  - pfg 10y ago
    Thinking about this a bit, it seems to be that clients talking to a server with OCSP stapling support could still make use of cached OCSP responses by simply omitting the "status_request" extension in the client hello, which would cause the server not to send the stapled OCSP response. I don't think any clients behave that way today, though. I'm not certain how session resumption plays into this either. If OCSP is skipped for resumed session as well (which would be my guess), you'd probably not take that small bandwidth hit all that often. As an aside, OCSP stapling improves your user's privacy quite a bit as well, by not giving your CA a list of all IP addresses connecting to a domain.
mfontani 10y ago
Yup, and that's why for thereg we started using Cloudflare's Railgun… with it, the connection to the servers (hosted in the UK) is "bearable"… without, it's abysmal: From a VPS in Sydney, with a Good Enough bandwidth: root@sydney:~# speedtest-cli 2>&1 | grep -e Download: -e Upload: Download: 721.20 Mbits/s Upload: 117.89 Mbits/s … doing the request through Railgun is "quite bearable": root@sydney:~# ./rg-diag -json https://www.theregister.co.uk/ | grep -e elapsed_time -e cloudflare_time -e origin_response_time "elapsed_time": "0.539365s", "origin_response_time": "0.045138s", "cloudflare_time": "0.494227s", Despite our "origin" server being quick enough, the main chunk of time is really "bytes having to travel half the world". Why does railgun help? Because this is what a user would get otherwise; the "whitepapers" site is hosted in the UK, and doesn't use Cloudflare or Railgun – it only uses Cloudflare for DNS: ./rg-diag -json http://whitepapers.theregister.co.uk/ | grep elapsed_time "elapsed_time": "0.706277s", … so that's ~200ms more, and _on http_. How much would https add, if it were done without Cloudflare's https and Railgun? That's easy to check, as our the whitepapers site has TLS (although admittedly not http/2): root@sydney:~# ./rg-diag -json https://whitepapers.theregister.co.uk/ | grep elapsed_time "elapsed_time": "1.559860s", that's quite a huge chunk of time that Cloudfalre HTTPS + Railgun just saves/shaves for us. Recommend it highly!
- pbarnes_1 10y ago
  Did you try CloudFlare without Railgun? That would be interesting.
  - mfontani 10y ago
    Sure, let's do just that… from the same location in Sydney; origin server hosting the content is in UK. This domain is on their "free" plan, as it gets hardly any traffic. root@sydney:~# ./rg-diag -json https://thereglabs.com/ | grep -e elapsed_time "elapsed_time": "0.863677s", So that's from Sydney to the UK, with https served by Cloudflare. The webapp serving that isn't the sharpest knife in the drawer, but when tested on localhost it replies in 0.015s – the rest is time taken moving bytes across the world. root@sydney:~# time curl -sH 'Host: thereglabs.com' -H 'Cf-Visitor: {"scheme":"https"}' http://THE_ORIGIN_SERVER/ -o/dev/null real 0m0.821s … and this is plain HTTP to the origin server: the free plan is great for offloading HTTPS at basically no cost in time added. We've got another domain on the business plan… so let's try that one. This is an _image_ request, which is _cached by cloudflare at the edge_: root@sydney:~# ./rg-diag -json https://regmedia.co.uk/2016/11/09/hypnotist_magician_smaller.jpg | grep elapsed_time "elapsed_time": "0.239641s", Lovely, the "local caching" of their CDN helps a ton! … compared to if we were to request the same file from the ORIGIN_SERVER over HTTP: root@sydney:~# ./rg-diag -json http://ORIGIN_SERVER/2016/11/09/hypnotist_magician_smaller.jpg | grep elapsed_time "elapsed_time": "0.704458s", … but our "origin server" _also_ is likely to have the image in the "memory cache"… … and that image was likely in their cache; so… let's add a parameter so they _will_ have to ask the origin server: $ pwgen 30 2 Eehacoh2phoo1Ooyengu6ohReWic2I Zeeyoe8ohpeeghie3doyeegoowiCei There you go… two new randomly generated values… root@sydney:~# ./rg-diag -json 'https://regmedia.co.uk/2016/11/09/hypnotist_magician_smaller.jpg?Eehacoh2phoo1Ooyengu6ohReWic2I=Zeeyoe8ohpeeghie3doyeegoowiCei' | grep elapsed_time "elapsed_time": "1.198940s", Yup, took quite a bit longer than the 200ms it took when the image URL was fully in their cache. All in all, from the point of view of being able to _easily_ serve people on the other side of the world with a "good enough" (not great, mind you!) response time, both "standard" Cloudflare, the "pro" offering _and specifically_ the "business" offering are just effin AWESOME.
jacquesm 10y ago
The biggest impact you can have on your users experience is to trim down the number of connections and the size of your pages. Long after that you can start worrying about round-trip-times to the server. This blog post is a nice example: 30 requests (ublock origin blocked another 12, with those enabled the time to load increases to a whopping 28 seconds), 2.5M transferred, 7 seconds load time. And all that for 4K payload + some images.
- lclarkmichalek 10y ago
  Number of connections isn't that relevant with HTTP2
  - valarauca1 10y ago
    Yes it is. The number of connections to one host isn't relevant in HTTP2. As ublock is blocking some ~20 connections these are going to different hosts. Connecting to a different host in HTTP2 is no different then HTTP1.1 If your HTTP2 is terminating at MANY boxes within your infrastructure you are failing to understand how HTTP2 works. Connections within a single TLS/TCP/IP connection are free, new TLS/TCP/IP connections cost exactly as much as before.
    - majewsky 10y ago
      Come to think of it, it's pretty ironic. For years, admins have been told to move static content to a different domain to trim down the request size (since browsers won't include the cookies for the main site). Now that might reverse, and it might be best to send all HTTP requests to the same server (for large sites: most likely some load balancer or haproxy or whatever) in order to benefit from HTTP/2 multiplexing.
      - valarauca1 10y ago
        HTTP/2 really seems more like a front end balancing and caching protocol. Handle your TLS/HTTP2 termination on the same boxes and use HTTP1.1 within the internal network.
hedora 10y ago
Presumably, cloudflare is up to its ears in NSL's, illegal wiretaps, etc. If you care at all about mass surveillance, censorship, oppressive governments (in the US, or the location of the cloudflare proxy) you probably should look elsewhere. It's probably controversial, but I'd love to see a yellow security icon in browsers when sites are using well known https relays that can see plaintext (or are doing other obviously bad things, like running software with known zero day exploits, etc)
- jacquesm 10y ago
  A mini audit along the lines of 'builtwith'. Hm. Good idea, why not go a step further and turn the 'no server signatures' advice on it's head: full disclosure, server signatures on, in fact, list each and every component in the stack so that end users can (through some plug-in) determine whether or not a site is safe to use. Of course nothing bad could ever come from that. /s I'm all for making the use of for instance Cloudflare less transparent so that users know who they are really talking to, but I'm confused about how you'd want to establish what a site is running without giving a potential attacker a lot of valuable information.
  - hedora 10y ago
    It would have to work without the site's permission, so the browser (or maybe a third party service) would do a basic vulnerability scan. Maybe orange could mean "a script kiddie could pwn this site in under an hour", and yellow would mean "we don't see how your ISP could mitm this, but server side providers (aws, google, azure cloudflare) definitely could." FWIW, my personal website uses let's encrypt, so it would be yellow or worse. Anyway, I like the idea of tying the security color in the url bar to an attacker model, since it at least gets people to think about attack models.
    - chatmasta 10y ago
      Wait, why does letsencrypt allow mitm by server provider?
c0nfused 10y ago
It seems to me that it is worth considering that HTTPS is not always a panacea of goodness. We should think about two things. First that almost every firewall out there right now supports https snooping via MITM. Example: https://www.paloaltonetworks.com/features/decryption https://www.paloaltonetworks.com/features/decryption Second, I just got back from rural China where most unblocked american webpages take between 5-15 seconds to load on my mobile phone many of them take upwards of a minute to load fully. This seems to be a fun combo of network latency, smaller than expected bandwidth, and pages using javascript with a series of different load events to display content. That dompageloaded->xmlhttprequest -> onreadystatechanged chain can ad some serious time on a 500ms round trip, and that's without talking about the css, the images, and the javascript. I forgot to pay me electric bill before I flew out and it took me nearly an hour to login, push pay my bill, accept the terms, and confirm payment. I was not a happy camper. It seems to me that while https is a very good thing, in some cases http and low bandwidth solutions might be worth implementing. It seems to me that one might actually want to tailor this to your audience, no one in their right mind is going to waste 5 minutes loading your web page. If they are so desperate they need to wait, they are going to hate you every minute they do it.
- jacquesm 10y ago
  > This seems to be a fun combo of network latency, smaller than expected bandwidth, and pages using javascript with a series of different load events to display content. You forgot about the great firewall of China playing merry MITM with your connections.
- chatmasta 10y ago
  Is there an easy way to pipeline those requests over one TCP connection? Or is that only possible with http/2? I wonder if it would be lower latency to open a single websocket tunnel on page load and download assets over the tunnel. Although at that point I suppose you're just replicating the functionality of http/2.
- rocqua 10y ago
  > First that almost every firewall out there right now supports https snooping via MITM. Example: https://www.paloaltonetworks.com/features/decryption https://www.paloaltonetworks.com/features/decryption Seems prudent to mention that this requires cooperation of the client bein MitMed. Specifically, the client needs to install a root certificate.
SEMW 10y ago
Funny coincidence, I was running into this exact issue earlier today. Had a customer complain about high response times from even our /time endpoint (which doesn't do anything except return server time) as measured by curl, and turns out it was just the TLS handshake: $ curl -o /dev/null -s -w "@time-format.txt" http://rest.ably.io/time time_namelookup: 0.012 time_connect: 0.031 time_appconnect: 0.000 time_pretransfer: 0.031 time_total: 0.053 $ curl -o /dev/null -s -w "@time-format.txt" https://rest.ably.io/time time_namelookup: 0.012 time_connect: 0.031 time_appconnect: 0.216 time_pretransfer: 0.216 time_total: 0.237 (as measured from my home computer, in the UK, so connecting to the aws eu-west region) Luckily not that much of an issue for us as when using an actual client library (unlike with curl) you get HTTP keep-alive, so at least the TCP connection doesn't need to be renewed for every request. And most customers who care about low latency are using a realtime library anyway, which just keeps a websocket, so sidesteps the whole issue. Certainly not enough to make us reconsider using TLS by default. Still, a bit annoying when you get someone who thinks they've discovered with curl that latency from them to us is 4x slower than to Pubnub, just because the Pubnub docs show the http versions of their endpoints, wheras ours show https, even though we're basically both using the same set of AWS regions...
aanm1988 10y ago
> In our case at Hunter, users were waiting on average 270ms for the handshake to be finished. Considering requests are handled in about 60ms on average, this was clearly too much. Why? Did it hurt user engagement? Were people complaining the site was slow?
- kuschku 10y ago
  The question is "is this the best we can do?". If it’s no, then clearly we should improve.
  - aanm1988 10y ago
    By spending time and effort on things that may not actually matter? To their credit this post talks about improving the performance, instead of just using it to complain about they can't use https because of a difference in a metric that may or may not actually cause end users any pain.
    - kuschku 10y ago
      You do realize that every latency you see on residential connections is magnified on mobile or even satellite connections? If you have a RTT of seconds, 10 additional roundtrips can cost an entire minute. These things become very noticeable very quickly if the system is in non-perfect environments. Additionally, even with modern browsers it takes far too long to open a website — it should be 100% instant, less than one or two frames (16 or 33ms). That's not possible, as RTT is usually around 18ms between users and CDN edges, but at least it should be below perceivable delays (100ms). EDIT: My best websites hover around 281ms to start of transfer, and 400ms to the site being finished. That's improvable, but most sites out there take literally half a minute to load. Now go on a 64kbps connection, and try again. Handshake takes seconds, start of transfer is almost after 30sec, and by the time your website arrives your coffee has gone cold (a few minutes for a google search). Years ago, Google was usable on dial-up. Now even the handshake takes as long as an entire search used to take. Notice anything?

121 points by antoinefink 10y ago

sp332 10y ago

If you're worried about a proprietary solution, you could host your own cache server in Australia or wherever your customers are having trouble.

problems 10y ago
Yeah, at a $200/mo cost, you could spin up a few VMs on DigitalOcean, Vultr or LightSail which have decent bandwidth and cache from there. Nice part about cloudflare though is that they can use anycast to determine location and then send the closest server IPs. For sub-$200/mo, you're not able to do that, you'd have to find a provider that could do it for you, I'm not sure anyone offers country-based anycast DNS alone. EDIT: Looks like easyDNS enterprise may be able to do it, https://fusion.easydns.com/Knowledgebase/Article/View/214/7/geodns https://fusion.easydns.com/Knowledgebase/Article/View/214/7/... for about $12.75/mo too. Might be a decent way to brew your own mini caching CDN for fairly cheap.
- jdub 10y ago
  You can also use Route 53 for the same purpose, for a tiny premium on the standard rates for name resolution. (See latency based routing queries and geo DNS queries below, plus health checks for failover.) https://aws.amazon.com/route53/pricing/ https://aws.amazon.com/route53/pricing/
- manigandham 10y ago
  > anycast to determine location and then send the closest server IPs Anycast doesn't determine location or send the closet IPs, it's all the same IP address announced using BGP (border gateway protocol) to automatically route to the closest (in network travel) server.
  - problems 10y ago
    Of course. Let me clarify - they use anycast DNS to send the closest CF caching proxy's IP.
    - manigandham 10y ago
      That's not how it works. Both their DNS and reverse proxy servers use anycast IPs without any DNS-based routing. They did recently release some features called traffic manager that lets you control the origin server based on geo. If you just need geo-balanced DNS though, AWS Route 53, Azure Traffic Manager, and NSOne offer DNS based routing.
      - problems 10y ago
        Really? I didn't think they'd do anycast on their reverse proxy servers, that seems risky to me (ie: a TCP connection changes from one server to another due to a BGP change), but I suppose the odds are fairly low. I seem to remember getting different IPs from different locations, but it could just be random or I could be mistaken. EDIT: Tried now and it seems I'm getting the same IPs from Canada and Australia, so you are indeed correct.
        manigandham 10y ago
        Yes, they have a big address space and announce all ips from every location: https://blog.cloudflare.com/cloudflares-architecture-eliminating-single-p/ https://blog.cloudflare.com/cloudflares-architecture-elimina... Pretty much all major CDNs use anycast today for load balancing, rolling downtime and security/ddos protection. Http/tcp connections are usually short-lived, relatively cheap to setup (since it's an edge network anyway) and BGP route updates don't happen that often. LinkedIn switched to anycast too after testing: https://engineering.linkedin.com/network-performance/tcp-over-ip-anycast-pipe-dream-or-reality https://engineering.linkedin.com/network-performance/tcp-ove...

alvil 10y ago

There is also another problem on how much and how often is Googlebot indexing your site because your site speed is one of the factors of so called Google index budget. My users are in Germany so my VPS is also in Germany to be fast for local user (~130ms for http reply), but for US Googlebot is my site slow (~420ms for http reply). So you are penalized also for this.

kijin 10y ago
Isn't this somewhat compensated for by the extra credit Google gives you for using SSL? I would also assume that Google is smart enough to take the physical location of your server into account when calculating how much penalty to apply in which searches. Sites that load fast in Germany should have higher ranks in searches from Germany.
KabuseCha 10y ago
Hi - I know some tools report a slow site in this case, but these reports are not accurate - don't believe them! Google is not that stupid ! :D I am currently working as a dev in an SEO-Agency (in Austria), and we never believed this hypothesis - so we tested this once with a bunch of our sites: When moving sites with a German speaking audience to a VPS in America, your rankings at google.de/google.at will decrease (slightly - the effect is not that big) - the other way around your rankings will improve (slightly). However - even if your rankings would improve when moving to America I would recommend keeping your sites hosted in Europe: The increase in rankings will not offset the decrease in user satisfaction and therefore the decline in your conversion rates.
- jacquesm 10y ago
  The whole idea that search engines should take preference over actual users is strange.
- chatmasta 10y ago
  That sounds like a really interesting experiment. Kudos for taking a scientific approach to SEO. A bit off-topic, but out of curiosity, have you run any other interesting experiments like this? I would love to read a blog post about them.

Kiro 10y ago

I don't understand why I need to use https on a static marketing webpage. No login stuff, no JavaScript, nothing. Just straight up HTML and CSS. Right now I need to pay about $150 every year for something that's only used to satisfy Google PageRank (I can't use LetsEncrypt with my hosting provider). Why?

Nullabillity 10y ago
Yeah, why are you using a bad hosting provider?
- Kiro 10y ago
  Legacy and one of the only hosting providers in my country. I don't want to risk worse localized SEO by hosting it outside.
  - RossM 10y ago
    I've been reading up on localisation SEO lately - as far as I understand, Google only uses "server IP is located in X country" as an indicator of where a site might be localised for, if it can't get any better information. For example, if you're using a ccTLD for the domain, or if it's a generic-TLD and you declare a country in Google Webmaster Tools, that will be a much stronger weighting. Of course, if that's wrong I'd love to know!
    - Kiro 10y ago
      Thank you. The actual web app is hosted on Linode in Frankfurt (using LetsEncrypt for https) so maybe I should host the marketing page there as well if that's true.
- djsumdog 10y ago
  When I was in the EU, I saw both Vodaphone and Three inject banners on the top of websites in various countries. They're not as easy to get away from as you think.
10y ago

chatmasta 10y ago

What's wrong with the cloudflare free plan? You can host a static site on github pages with a custom domain and use the free cloudflare SSL cert.

andrewaylett 10y ago
Without Railgun, there's no guarantee that the CloudFlare nodes will have an open socket to your origin server, so your visitors may still have to pay the cost of the round-trip.
amiraliakbari 10y ago
The "Railgun" feature mentioned in the article is only available in some paid plans. Using the free plan wouldn't keep an open connection between your servers and Cloudflare's. It does improve the situation by terminating users' handshakes early, using better links, warm DNS cache, etc. among servers. But the latency hard limit is still present between your server and CF. Skipping https between your server and CL is not an option either for any site transferring user data.
- chatmasta 10y ago
  Ah, I see. I did not realize that. Accordingly, I edited my comment to be less inflammatory. :) I understand that by using the generic CF free cert, https terminates at CF and the connection CF->Origin is over unencrypted HTTP. Is this why there is latency overhead? Because CF cannot connect to origin via https so it cannot open a persistent tunnel? Or is it because the overhead of keeping an open https tunnel per origin server is prohibitively expensive to maintain for every free customer? I assume that even though there is no persistent tunnel, CF still must still use persistent TCP connections?
  - floatboth 10y ago
    Maybe the cost is not completely prohibitively expensive but they do consider it a premium feature. Have to earn money :D

filleokus 10y ago

A related interesting topic is the possibility of secure cache servers that don't break the secure channel with "blind caches". Currently just a RFC draft and probably a long time from mass adoption, but nevertheless interesting. https://tools.ietf.org/html/draft-thomson-http-bc-00 https://tools.ietf.org/html/draft-thomson-http-bc-00, and Ericsson's article on it https://www.ericsson.com/thecompany/our_publications/ericsson_technology_review/archive/blind-cache https://www.ericsson.com/thecompany/our_publications/ericsso...

nprescott 10y ago

I really enjoyed the coverage of the same topic in High Performance Browser Networking[0]. It effectively explains the key performance influencers across various networks without being boring. [0]: https://hpbn.co/ https://hpbn.co/

andreareina 10y ago

One round trip over the course of the time that the user is using the same OS/browser installation isn't much. The Cloudflare Railgun is an interesting solution, and one that could be implemented in the context of an SPA over a websockets connection. Or conceivably some other consumer of an API.

hannob 10y ago

There are a couple of other things you can do with existing TLS technology that can improve your latency, e.g. using OCSP stapling, use modern crypto so browsers may use TLS false start, avoid too many ciphers or unnecessary certs in the chain to make the handshake smaller. It's a bit older, but here's some info, much of it is still valid: https://istlsfastyet.com/ https://istlsfastyet.com/

citrin_ru 10y ago
It is questionable if OCSP stapling reduces TLS handshake time. Without OCSP browser makes slow request to CA, but caches results for a long time so slow request happens not often. With OCSP stapling enabled more data is transferred between client and server on each TLS handshake. Main proponents of OCSP stapling are CA, because it saves them bandwidth/hardware.
- pfg 10y ago
  Thinking about this a bit, it seems to be that clients talking to a server with OCSP stapling support could still make use of cached OCSP responses by simply omitting the "status_request" extension in the client hello, which would cause the server not to send the stapled OCSP response. I don't think any clients behave that way today, though. I'm not certain how session resumption plays into this either. If OCSP is skipped for resumed session as well (which would be my guess), you'd probably not take that small bandwidth hit all that often. As an aside, OCSP stapling improves your user's privacy quite a bit as well, by not giving your CA a list of all IP addresses connecting to a domain.

mfontani 10y ago

Yup, and that's why for thereg we started using Cloudflare's Railgun… with it, the connection to the servers (hosted in the UK) is "bearable"… without, it's abysmal: From a VPS in Sydney, with a Good Enough bandwidth: root@sydney:~# speedtest-cli 2>&1 | grep -e Download: -e Upload: Download: 721.20 Mbits/s Upload: 117.89 Mbits/s … doing the request through Railgun is "quite bearable": root@sydney:~# ./rg-diag -json https://www.theregister.co.uk/ | grep -e elapsed_time -e cloudflare_time -e origin_response_time "elapsed_time": "0.539365s", "origin_response_time": "0.045138s", "cloudflare_time": "0.494227s", Despite our "origin" server being quick enough, the main chunk of time is really "bytes having to travel half the world". Why does railgun help? Because this is what a user would get otherwise; the "whitepapers" site is hosted in the UK, and doesn't use Cloudflare or Railgun – it only uses Cloudflare for DNS: ./rg-diag -json http://whitepapers.theregister.co.uk/ | grep elapsed_time "elapsed_time": "0.706277s", … so that's ~200ms more, and _on http_. How much would https add, if it were done without Cloudflare's https and Railgun? That's easy to check, as our the whitepapers site has TLS (although admittedly not http/2): root@sydney:~# ./rg-diag -json https://whitepapers.theregister.co.uk/ | grep elapsed_time "elapsed_time": "1.559860s", that's quite a huge chunk of time that Cloudfalre HTTPS + Railgun just saves/shaves for us. Recommend it highly!

pbarnes_1 10y ago
Did you try CloudFlare without Railgun? That would be interesting.
- mfontani 10y ago
  Sure, let's do just that… from the same location in Sydney; origin server hosting the content is in UK. This domain is on their "free" plan, as it gets hardly any traffic. root@sydney:~# ./rg-diag -json https://thereglabs.com/ | grep -e elapsed_time "elapsed_time": "0.863677s", So that's from Sydney to the UK, with https served by Cloudflare. The webapp serving that isn't the sharpest knife in the drawer, but when tested on localhost it replies in 0.015s – the rest is time taken moving bytes across the world. root@sydney:~# time curl -sH 'Host: thereglabs.com' -H 'Cf-Visitor: {"scheme":"https"}' http://THE_ORIGIN_SERVER/ -o/dev/null real 0m0.821s … and this is plain HTTP to the origin server: the free plan is great for offloading HTTPS at basically no cost in time added. We've got another domain on the business plan… so let's try that one. This is an _image_ request, which is _cached by cloudflare at the edge_: root@sydney:~# ./rg-diag -json https://regmedia.co.uk/2016/11/09/hypnotist_magician_smaller.jpg | grep elapsed_time "elapsed_time": "0.239641s", Lovely, the "local caching" of their CDN helps a ton! … compared to if we were to request the same file from the ORIGIN_SERVER over HTTP: root@sydney:~# ./rg-diag -json http://ORIGIN_SERVER/2016/11/09/hypnotist_magician_smaller.jpg | grep elapsed_time "elapsed_time": "0.704458s", … but our "origin server" _also_ is likely to have the image in the "memory cache"… … and that image was likely in their cache; so… let's add a parameter so they _will_ have to ask the origin server: $ pwgen 30 2 Eehacoh2phoo1Ooyengu6ohReWic2I Zeeyoe8ohpeeghie3doyeegoowiCei There you go… two new randomly generated values… root@sydney:~# ./rg-diag -json 'https://regmedia.co.uk/2016/11/09/hypnotist_magician_smaller.jpg?Eehacoh2phoo1Ooyengu6ohReWic2I=Zeeyoe8ohpeeghie3doyeegoowiCei' | grep elapsed_time "elapsed_time": "1.198940s", Yup, took quite a bit longer than the 200ms it took when the image URL was fully in their cache. All in all, from the point of view of being able to _easily_ serve people on the other side of the world with a "good enough" (not great, mind you!) response time, both "standard" Cloudflare, the "pro" offering _and specifically_ the "business" offering are just effin AWESOME.

jacquesm 10y ago

The biggest impact you can have on your users experience is to trim down the number of connections and the size of your pages. Long after that you can start worrying about round-trip-times to the server. This blog post is a nice example: 30 requests (ublock origin blocked another 12, with those enabled the time to load increases to a whopping 28 seconds), 2.5M transferred, 7 seconds load time. And all that for 4K payload + some images.

lclarkmichalek 10y ago
Number of connections isn't that relevant with HTTP2
- valarauca1 10y ago
  Yes it is. The number of connections to one host isn't relevant in HTTP2. As ublock is blocking some ~20 connections these are going to different hosts. Connecting to a different host in HTTP2 is no different then HTTP1.1 If your HTTP2 is terminating at MANY boxes within your infrastructure you are failing to understand how HTTP2 works. Connections within a single TLS/TCP/IP connection are free, new TLS/TCP/IP connections cost exactly as much as before.
  - majewsky 10y ago
    Come to think of it, it's pretty ironic. For years, admins have been told to move static content to a different domain to trim down the request size (since browsers won't include the cookies for the main site). Now that might reverse, and it might be best to send all HTTP requests to the same server (for large sites: most likely some load balancer or haproxy or whatever) in order to benefit from HTTP/2 multiplexing.
    - valarauca1 10y ago
      HTTP/2 really seems more like a front end balancing and caching protocol. Handle your TLS/HTTP2 termination on the same boxes and use HTTP1.1 within the internal network.

hedora 10y ago

Presumably, cloudflare is up to its ears in NSL's, illegal wiretaps, etc. If you care at all about mass surveillance, censorship, oppressive governments (in the US, or the location of the cloudflare proxy) you probably should look elsewhere. It's probably controversial, but I'd love to see a yellow security icon in browsers when sites are using well known https relays that can see plaintext (or are doing other obviously bad things, like running software with known zero day exploits, etc)

jacquesm 10y ago
A mini audit along the lines of 'builtwith'. Hm. Good idea, why not go a step further and turn the 'no server signatures' advice on it's head: full disclosure, server signatures on, in fact, list each and every component in the stack so that end users can (through some plug-in) determine whether or not a site is safe to use. Of course nothing bad could ever come from that. /s I'm all for making the use of for instance Cloudflare less transparent so that users know who they are really talking to, but I'm confused about how you'd want to establish what a site is running without giving a potential attacker a lot of valuable information.
- hedora 10y ago
  It would have to work without the site's permission, so the browser (or maybe a third party service) would do a basic vulnerability scan. Maybe orange could mean "a script kiddie could pwn this site in under an hour", and yellow would mean "we don't see how your ISP could mitm this, but server side providers (aws, google, azure cloudflare) definitely could." FWIW, my personal website uses let's encrypt, so it would be yellow or worse. Anyway, I like the idea of tying the security color in the url bar to an attacker model, since it at least gets people to think about attack models.
  - chatmasta 10y ago
    Wait, why does letsencrypt allow mitm by server provider?

c0nfused 10y ago

It seems to me that it is worth considering that HTTPS is not always a panacea of goodness. We should think about two things. First that almost every firewall out there right now supports https snooping via MITM. Example: https://www.paloaltonetworks.com/features/decryption https://www.paloaltonetworks.com/features/decryption Second, I just got back from rural China where most unblocked american webpages take between 5-15 seconds to load on my mobile phone many of them take upwards of a minute to load fully. This seems to be a fun combo of network latency, smaller than expected bandwidth, and pages using javascript with a series of different load events to display content. That dompageloaded->xmlhttprequest -> onreadystatechanged chain can ad some serious time on a 500ms round trip, and that's without talking about the css, the images, and the javascript. I forgot to pay me electric bill before I flew out and it took me nearly an hour to login, push pay my bill, accept the terms, and confirm payment. I was not a happy camper. It seems to me that while https is a very good thing, in some cases http and low bandwidth solutions might be worth implementing. It seems to me that one might actually want to tailor this to your audience, no one in their right mind is going to waste 5 minutes loading your web page. If they are so desperate they need to wait, they are going to hate you every minute they do it.

jacquesm 10y ago
> This seems to be a fun combo of network latency, smaller than expected bandwidth, and pages using javascript with a series of different load events to display content. You forgot about the great firewall of China playing merry MITM with your connections.
chatmasta 10y ago
Is there an easy way to pipeline those requests over one TCP connection? Or is that only possible with http/2? I wonder if it would be lower latency to open a single websocket tunnel on page load and download assets over the tunnel. Although at that point I suppose you're just replicating the functionality of http/2.
rocqua 10y ago
> First that almost every firewall out there right now supports https snooping via MITM. Example: https://www.paloaltonetworks.com/features/decryption https://www.paloaltonetworks.com/features/decryption Seems prudent to mention that this requires cooperation of the client bein MitMed. Specifically, the client needs to install a root certificate.

SEMW 10y ago

Funny coincidence, I was running into this exact issue earlier today. Had a customer complain about high response times from even our /time endpoint (which doesn't do anything except return server time) as measured by curl, and turns out it was just the TLS handshake: $ curl -o /dev/null -s -w "@time-format.txt" http://rest.ably.io/time time_namelookup: 0.012 time_connect: 0.031 time_appconnect: 0.000 time_pretransfer: 0.031 time_total: 0.053 $ curl -o /dev/null -s -w "@time-format.txt" https://rest.ably.io/time time_namelookup: 0.012 time_connect: 0.031 time_appconnect: 0.216 time_pretransfer: 0.216 time_total: 0.237 (as measured from my home computer, in the UK, so connecting to the aws eu-west region) Luckily not that much of an issue for us as when using an actual client library (unlike with curl) you get HTTP keep-alive, so at least the TCP connection doesn't need to be renewed for every request. And most customers who care about low latency are using a realtime library anyway, which just keeps a websocket, so sidesteps the whole issue. Certainly not enough to make us reconsider using TLS by default. Still, a bit annoying when you get someone who thinks they've discovered with curl that latency from them to us is 4x slower than to Pubnub, just because the Pubnub docs show the http versions of their endpoints, wheras ours show https, even though we're basically both using the same set of AWS regions...

aanm1988 10y ago

> In our case at Hunter, users were waiting on average 270ms for the handshake to be finished. Considering requests are handled in about 60ms on average, this was clearly too much. Why? Did it hurt user engagement? Were people complaining the site was slow?

kuschku 10y ago
The question is "is this the best we can do?". If it’s no, then clearly we should improve.
- aanm1988 10y ago
  By spending time and effort on things that may not actually matter? To their credit this post talks about improving the performance, instead of just using it to complain about they can't use https because of a difference in a metric that may or may not actually cause end users any pain.
  - kuschku 10y ago
    You do realize that every latency you see on residential connections is magnified on mobile or even satellite connections? If you have a RTT of seconds, 10 additional roundtrips can cost an entire minute. These things become very noticeable very quickly if the system is in non-perfect environments. Additionally, even with modern browsers it takes far too long to open a website — it should be 100% instant, less than one or two frames (16 or 33ms). That's not possible, as RTT is usually around 18ms between users and CDN edges, but at least it should be below perceivable delays (100ms). EDIT: My best websites hover around 281ms to start of transfer, and 400ms to the site being finished. That's improvable, but most sites out there take literally half a minute to load. Now go on a 64kbps connection, and try again. Handshake takes seconds, start of transfer is almost after 30sec, and by the time your website arrives your coffee has gone cold (a few minutes for a google search). Years ago, Google was usable on dial-up. Now even the handshake takes as long as an entire search used to take. Notice anything?