7 ms·

Why we don’t use a CDN: A story about SPDY and SSL (2014)

76 points by flipchart 12y ago

ckuehl 12y ago
Rather than copy their suggestions for SSL configuration, I think you can probably find better (and more well-maintained) advice on the Mozilla wiki: https://wiki.mozilla.org/Security/Server_Side_TLS https://wiki.mozilla.org/Security/Server_Side_TLS
bsder 12y ago
Is anybody else disgusted that 1.6 seconds is considered an acceptable page load time? When did this happen? Whatever happened to 200ms being the benchmark? Dear Lord, people, please stop loading 14 zillion different domains every time I hit a web page.
- FooBarWidget 12y ago
  If you can tell me how to make a decent looking site with a few MB of images load in 200 ms, I'm all ears. The only method I can think of, is by making the website ugly (i.e. few images and other design assets).
  - Nux 12y ago
    Ever heard of "less is more"?
  - bsder 12y ago
    1) Put your assets on the same domain. Did you see how much time was being used in multiple DNS requests? To an end user, everything not on your domain is CRAP. It's not there for me, the end user. It's there for you, the website owner so you can aggregate my eyeballs, pitch me something, use somebody's comment service, extract advertising revenue, push network bandwidth onto Google, jQuery, etc. 2) Why on Earth do you need megabytes of images? Really? WTF? Just because you took the picture with a gigapixel camera doesn't mean you need to serve every pixel. 3) Minimize the Javascript No, you don't need that Javascript framework. Shoot it. Especially on tablet/mobile browsers, a very small number of pages cache before the browser has to reload and regenerate (and that's REALLY slow). The research is against you. Every extra 100ms is a significant drop in retention rate. Your website will stick out because everything is so snappy. Users can feel this--especially on tablets and mobile.
    - tombrossman 12y ago
      Agree with the above comments about reducing bloat, and I'm surprised by the number of smart people who think nothing of setting up a Wordpress site with dozens of CSS and JS files. I just spent 20 minutes trying to find a link to an Apache function I found recently, which will join all your CSS or JS files together with something like an @include server-side automatically. Can't find it now but as I have been doing this manually I hope I find it again. Anyone know what this is? I seem to recall it was a native Apache function, not part of mod_pagespeed.
      - moehm 12y ago
        Do you mean modconcat? [0] There is also a nginx version by Alibaba available. [1] [0] https://code.google.com/p/modconcat/ https://code.google.com/p/modconcat/ [1] https://github.com/alibaba/nginx-http-concat https://github.com/alibaba/nginx-http-concat
        tombrossman 12y ago
        Nah, found that while searching for the other. I can probably use that, though. The one I'm thinking of had you putting a short instruction at the top of the file (like @include) if you wanted it to be part of the output file. Maybe I'm not remembering this correctly as I can find nothing on it now.
        e12e 12y ago
        Server Side Includes? (mod_include for apache, mod_ssi for some others)? Not sure I'd recommend that as a general solution today -- for static sites I think you're better of building the static html, and serve that up. With varnish, you might want to look at edge-side includes.
      - snowwrestler 12y ago
        Wordpress has a number of caching plugins that will do this at the application layer.
    - 12y ago
  - NKCSS 12y ago
    Try the dutch tech site http://tweakers.net/ http://tweakers.net/ it's one of the best sites in terms of page speed vs complexity and shows what can be done if you just do it right. If you do view-source, they also provide their stats: For me: 
    - tombrossman 12y ago
      Looks good and it is fast, but there are plenty of improvements to be made according to their Pingdom score: http://tools.pingdom.com/fpt/#!/bALFWp/http://tweakers.net/ http://tools.pingdom.com/fpt/#!/bALFWp/http://tweakers.net/ Surprised to see a total of 91 requests, for example.
  - realusername 12y ago
    Maybe that's part of the issue, websites don't needs 5MB worth of images to look good.
    - FooBarWidget 12y ago
      If your definition of "good" is Reddit and Hacker News, or a mostly text website like a blog, then yeah.
  - tomjen3 12y ago
    Concentrate on the CSS and HTML, then have the images loaded in after. That way it will look like the site has loaded quick and the images are what takes most of the time, anyway.
  - maybeathrowaway 12y ago
    Examples of those necessary images? A few MB is a lot unless you are talking about an artist's portfolio.
    - e12e 12y ago
      Even then, MBs on first page load is a lot. Judging from: http://www.sorryana.rocks/mavericks-wallpaper/ http://www.sorryana.rocks/mavericks-wallpaper/ (which loads some content/initial display pretty fast, btw) -- even a full-screen image intended for the Apple Retina iMac is "just" ~3MB. And if you're going to fill the entire viewport (and then some) -- you'll not be needing a lot of other images...
- thejosh 12y ago
  And this page took 9.7 seconds to open from Perth, Australia on a 10mbit connection. CDNs are useful because they load content closest to where the user is located.
  - dozzie 12y ago
    I think this would be more related to your 10 millibit connection.
  - jmkni 12y ago
    Aren't they also useful because if you and I use the same CDN for our libraries, we both use the same library/version, and a user visits my site and then visits your site, the libraries are already cached in the browser? If we both store those libraries locally, the library is loaded twice. That's my understanding of it anyway.
    - TheLoneWolfling 12y ago
      This is where links that have a hash would be useful. (I.e. "This links points to this file - if you already have the file with this hash use it instead")
    - eli 12y ago
      Yes, though there's some debate as to how often this actually matters. A surprisingly large number of the general web public comes in with a completely cold cache.
    - nahname 12y ago
      That would only occur if the sites referenced exactly the same version of the same library on the same CDN with the same URL.
    - michaelmior 12y ago
      Very often people concatenate and minify different assets, which is another reason that this sharing doesn't occur much in practice.
- bkchung 12y ago
  Google used to say "We're aiming for 100ms!" a lot quite some time ago. Now the their advocates are saying more like aim for 1000ms(1s) which is very realistic and good. 200ms will easily be eaten by SSL's 3 chunks of RTT(with the smaller initial cwnd cases).
- ipedrazas 12y ago
  Advertise bidders have a timeout of 200ms, so if your ads are going to need 200ms at least, your page is going to tae longer than that.
  - pirsquare 12y ago
    To add to that, 100ms-200ms is only the cutoff time to determine the bid winner from adexchange perspective. We still need to add the time for adexchange to notify bid winner + adserving load time for total ad response latency.
  - TazeTSchnitzel 12y ago
    Until your comment I had no idea per-impression real-time advertising bidding was a thing: https://en.wikipedia.org/wiki/Real-time_bidding https://en.wikipedia.org/wiki/Real-time_bidding
  - e12e 12y ago
    Not sure why you'd want to load ads synchronously. Surely you can just load the page with placeholder divs, and swap them out when the ads are ready? The extra jarring blink when ads arrive, should increase visibility anyway. And for the rest of us, that browse with ad-blockers the only change would be a quicker load time...
- mreiland 12y ago
  That was exactly my thought. 1.6 seconds is entirely too. fucking. slow.
  - sp332 12y ago
    1.6 seconds to seeing the main content of the page is too slow. But if you're just loading up "other articles" at the bottom, or "latest tweets" on the side, then it doesn't really matter if it's slow.
  - onion2k 12y ago
    What are you doing that you can't wait for less than 2 seconds for something?
    - oconnore 12y ago
      Browsing on a hotspot where 2s becomes 10+s?
    - bsder 12y ago
      2 seconds every time I switch a tab. I have 12 tabs open right now. Mobile browsers cache a LOT less after rendering than desktop browsers.
    - mreiland 12y ago
      It isn't a matter of can't, it's a matter of don't want to. Death by a thousand small cuts, a single 2 second wait isn't objectionable, the entire app being that slow means literally every time I do something I wait 2 seconds. It cuts into your mental flow.
- bkeroack 12y ago
  On the other hand, it's pretty amazing that a ~2MB application can be downloaded over the network and executed to a state of responsiveness within 2 seconds. Some of us remember when the MS Word binary was roughly that size and took ~20-30 seconds to start up (with a splash screen, etc). It's trivial to get a single static HTML hello world page with no assets to load in 100ms.
  - marssaxman 12y ago
    I'd really rather not be downloading 2MB applications from random internet sites, like, ever. I've started browsing with noscript turned on by default, and despite what everyone says about how useless the modern web is without it, things are pretty great - everything loads instantly and the stuff that doesn't load generally isn't much I care about.
bkchung 12y ago
My experience: https://medium.com/p/196b5024899c https://medium.com/p/196b5024899c Major browsers will not support h2c switching(switching from http, rather than TLS), and faster(but less secure) ciphers or null ciphers will not work for those browsers either, so speed in the HTTP/2 world will likely be something on top of SSL's latency.
jdub 12y ago
The big mistake here is using a split host name CDN (instead of an edge/origin CDN), and then putting the primary CSS on the split host name. That means you have to wait for DNS resolution and the TCP connection roundtrips (or worse, unoptimised TLS roundtrips) just to get the CSS. If they were on the same host name, and the CSS is the first thing in the HTML, it'll be downloaded immediately after the HTML. (And with SPDY, the browser could start downloading the CSS on the same connection as soon as it parses the link element.)
- youngtaff 12y ago
  Yeh would be interesting so what performance would be like with the origin served through the CDN or even using dns-prefetch or tcp-preconnect I think all we can gather from this example is that MaxCDNs TLS implementation wasn't optimal and they were working on it Would be interesting to see how this behaves for other CDNs or with link rel= optimisations in place
jgrahamc 12y ago
See also "Using CloudFlare to mix domain sharding and SPDY": https://blog.cloudflare.com/using-cloudflare-to-mix-domain-sharding-and-spdy/ https://blog.cloudflare.com/using-cloudflare-to-mix-domain-s...
mdekkers 12y ago
We optimise and host websites for our clients, use an https/spdy CDN exclusively, most sites we work on are dogs when it comes to performance, and aim for half second page load times on average. Anything over 1 second is unacceptable. I read this article before, I think it was posted to HN previously, and wasn't impressed at the time. Still not impressed now.... Long story short: Not using a CDN is stupid. Learn to CDN.
puppetmaster3 12y ago
Why we don't use a CDN: NIH.
- wampus 12y ago
  In this case, that's an appropriate attitude. If your environment is a mess and you add complexity with a CDN, you've got a complicated mess. But a local optimization like this benefits your entire organization and puts off the need for a CDN (a solution to a problem that no longer exists). They may grow to a point where a real strain is being put on their systems, they can't make local improvements, and involving a CDN is the right choice.
troels 12y ago
In the Australia comparison, the CDN version spends almost 1.5 second on DNS lookup for the main domain. In the non-CDN version this figure is much lower. It seems disingenuous to attribute this overhead to the CDN, no? And as an aside - Maybe there are some low hanging fruits in tuning their DNS setup instead? 1.5s seems extremely high to me.
dedalus 12y ago
http://www.webpagetest.org/result/150402_TV_f0fe942e49dc1191e818c0cecba48d76/ http://www.webpagetest.org/result/150402_TV_f0fe942e49dc1191... shows you take 8 seconds to load in Australia. Using http://www.webpagetest.org/result/150402_22_0fa16aec7ebc410a00737efcd1321d3e/ http://www.webpagetest.org/result/150402_22_0fa16aec7ebc410a... also shows 8 seconds from tokyo http://www.webpagetest.org/result/150402_AN_22f88770d4bab89c84bd986d75da50ae/ http://www.webpagetest.org/result/150402_AN_22f88770d4bab89c... shows 8 seconds from India http://www.webpagetest.org/result/150402_CT_49277f5267e82b6b75a5f2bc9a3cf589/ http://www.webpagetest.org/result/150402_CT_49277f5267e82b6b... shows 36 seconds from Amsetrdam http://www.webpagetest.org/result/150402_9G_0810eaa940dc2941d0ff83615913e959/ http://www.webpagetest.org/result/150402_9G_0810eaa940dc2941... shows 12 seconds from Buenos Aires Not so sure that you dont need a CDN based on this

76 points by flipchart 12y ago

Rather than copy their suggestions for SSL configuration, I think you can probably find better (and more well-maintained) advice on the Mozilla wiki: https://wiki.mozilla.org/Security/Server_Side_TLS https://wiki.mozilla.org/Security/Server_Side_TLS

bsder 12y ago

Is anybody else disgusted that 1.6 seconds is considered an acceptable page load time? When did this happen? Whatever happened to 200ms being the benchmark? Dear Lord, people, please stop loading 14 zillion different domains every time I hit a web page.

FooBarWidget 12y ago
If you can tell me how to make a decent looking site with a few MB of images load in 200 ms, I'm all ears. The only method I can think of, is by making the website ugly (i.e. few images and other design assets).
- Nux 12y ago
  Ever heard of "less is more"?
- bsder 12y ago
  1) Put your assets on the same domain. Did you see how much time was being used in multiple DNS requests? To an end user, everything not on your domain is CRAP. It's not there for me, the end user. It's there for you, the website owner so you can aggregate my eyeballs, pitch me something, use somebody's comment service, extract advertising revenue, push network bandwidth onto Google, jQuery, etc. 2) Why on Earth do you need megabytes of images? Really? WTF? Just because you took the picture with a gigapixel camera doesn't mean you need to serve every pixel. 3) Minimize the Javascript No, you don't need that Javascript framework. Shoot it. Especially on tablet/mobile browsers, a very small number of pages cache before the browser has to reload and regenerate (and that's REALLY slow). The research is against you. Every extra 100ms is a significant drop in retention rate. Your website will stick out because everything is so snappy. Users can feel this--especially on tablets and mobile.
  - tombrossman 12y ago
    Agree with the above comments about reducing bloat, and I'm surprised by the number of smart people who think nothing of setting up a Wordpress site with dozens of CSS and JS files. I just spent 20 minutes trying to find a link to an Apache function I found recently, which will join all your CSS or JS files together with something like an @include server-side automatically. Can't find it now but as I have been doing this manually I hope I find it again. Anyone know what this is? I seem to recall it was a native Apache function, not part of mod_pagespeed.
    - moehm 12y ago
      Do you mean modconcat? [0] There is also a nginx version by Alibaba available. [1] [0] https://code.google.com/p/modconcat/ https://code.google.com/p/modconcat/ [1] https://github.com/alibaba/nginx-http-concat https://github.com/alibaba/nginx-http-concat
      - tombrossman 12y ago
        Nah, found that while searching for the other. I can probably use that, though. The one I'm thinking of had you putting a short instruction at the top of the file (like @include) if you wanted it to be part of the output file. Maybe I'm not remembering this correctly as I can find nothing on it now.
        e12e 12y ago
        Server Side Includes? (mod_include for apache, mod_ssi for some others)? Not sure I'd recommend that as a general solution today -- for static sites I think you're better of building the static html, and serve that up. With varnish, you might want to look at edge-side includes.
    - snowwrestler 12y ago
      Wordpress has a number of caching plugins that will do this at the application layer.
  - 12y ago
- NKCSS 12y ago
  Try the dutch tech site http://tweakers.net/ http://tweakers.net/ it's one of the best sites in terms of page speed vs complexity and shows what can be done if you just do it right. If you do view-source, they also provide their stats: For me: 
  - tombrossman 12y ago
    Looks good and it is fast, but there are plenty of improvements to be made according to their Pingdom score: http://tools.pingdom.com/fpt/#!/bALFWp/http://tweakers.net/ http://tools.pingdom.com/fpt/#!/bALFWp/http://tweakers.net/ Surprised to see a total of 91 requests, for example.
- realusername 12y ago
  Maybe that's part of the issue, websites don't needs 5MB worth of images to look good.
  - FooBarWidget 12y ago
    If your definition of "good" is Reddit and Hacker News, or a mostly text website like a blog, then yeah.
- tomjen3 12y ago
  Concentrate on the CSS and HTML, then have the images loaded in after. That way it will look like the site has loaded quick and the images are what takes most of the time, anyway.
- maybeathrowaway 12y ago
  Examples of those necessary images? A few MB is a lot unless you are talking about an artist's portfolio.
  - e12e 12y ago
    Even then, MBs on first page load is a lot. Judging from: http://www.sorryana.rocks/mavericks-wallpaper/ http://www.sorryana.rocks/mavericks-wallpaper/ (which loads some content/initial display pretty fast, btw) -- even a full-screen image intended for the Apple Retina iMac is "just" ~3MB. And if you're going to fill the entire viewport (and then some) -- you'll not be needing a lot of other images...
thejosh 12y ago
And this page took 9.7 seconds to open from Perth, Australia on a 10mbit connection. CDNs are useful because they load content closest to where the user is located.
- dozzie 12y ago
  I think this would be more related to your 10 millibit connection.
- jmkni 12y ago
  Aren't they also useful because if you and I use the same CDN for our libraries, we both use the same library/version, and a user visits my site and then visits your site, the libraries are already cached in the browser? If we both store those libraries locally, the library is loaded twice. That's my understanding of it anyway.
  - TheLoneWolfling 12y ago
    This is where links that have a hash would be useful. (I.e. "This links points to this file - if you already have the file with this hash use it instead")
  - eli 12y ago
    Yes, though there's some debate as to how often this actually matters. A surprisingly large number of the general web public comes in with a completely cold cache.
  - nahname 12y ago
    That would only occur if the sites referenced exactly the same version of the same library on the same CDN with the same URL.
  - michaelmior 12y ago
    Very often people concatenate and minify different assets, which is another reason that this sharing doesn't occur much in practice.
bkchung 12y ago
Google used to say "We're aiming for 100ms!" a lot quite some time ago. Now the their advocates are saying more like aim for 1000ms(1s) which is very realistic and good. 200ms will easily be eaten by SSL's 3 chunks of RTT(with the smaller initial cwnd cases).
ipedrazas 12y ago
Advertise bidders have a timeout of 200ms, so if your ads are going to need 200ms at least, your page is going to tae longer than that.
- pirsquare 12y ago
  To add to that, 100ms-200ms is only the cutoff time to determine the bid winner from adexchange perspective. We still need to add the time for adexchange to notify bid winner + adserving load time for total ad response latency.
- TazeTSchnitzel 12y ago
  Until your comment I had no idea per-impression real-time advertising bidding was a thing: https://en.wikipedia.org/wiki/Real-time_bidding https://en.wikipedia.org/wiki/Real-time_bidding
- e12e 12y ago
  Not sure why you'd want to load ads synchronously. Surely you can just load the page with placeholder divs, and swap them out when the ads are ready? The extra jarring blink when ads arrive, should increase visibility anyway. And for the rest of us, that browse with ad-blockers the only change would be a quicker load time...
mreiland 12y ago
That was exactly my thought. 1.6 seconds is entirely too. fucking. slow.
- sp332 12y ago
  1.6 seconds to seeing the main content of the page is too slow. But if you're just loading up "other articles" at the bottom, or "latest tweets" on the side, then it doesn't really matter if it's slow.
- onion2k 12y ago
  What are you doing that you can't wait for less than 2 seconds for something?
  - oconnore 12y ago
    Browsing on a hotspot where 2s becomes 10+s?
  - bsder 12y ago
    2 seconds every time I switch a tab. I have 12 tabs open right now. Mobile browsers cache a LOT less after rendering than desktop browsers.
  - mreiland 12y ago
    It isn't a matter of can't, it's a matter of don't want to. Death by a thousand small cuts, a single 2 second wait isn't objectionable, the entire app being that slow means literally every time I do something I wait 2 seconds. It cuts into your mental flow.
bkeroack 12y ago
On the other hand, it's pretty amazing that a ~2MB application can be downloaded over the network and executed to a state of responsiveness within 2 seconds. Some of us remember when the MS Word binary was roughly that size and took ~20-30 seconds to start up (with a splash screen, etc). It's trivial to get a single static HTML hello world page with no assets to load in 100ms.
- marssaxman 12y ago
  I'd really rather not be downloading 2MB applications from random internet sites, like, ever. I've started browsing with noscript turned on by default, and despite what everyone says about how useless the modern web is without it, things are pretty great - everything loads instantly and the stuff that doesn't load generally isn't much I care about.

bkchung 12y ago

My experience: https://medium.com/p/196b5024899c https://medium.com/p/196b5024899c Major browsers will not support h2c switching(switching from http, rather than TLS), and faster(but less secure) ciphers or null ciphers will not work for those browsers either, so speed in the HTTP/2 world will likely be something on top of SSL's latency.

jdub 12y ago

The big mistake here is using a split host name CDN (instead of an edge/origin CDN), and then putting the primary CSS on the split host name. That means you have to wait for DNS resolution and the TCP connection roundtrips (or worse, unoptimised TLS roundtrips) just to get the CSS. If they were on the same host name, and the CSS is the first thing in the HTML, it'll be downloaded immediately after the HTML. (And with SPDY, the browser could start downloading the CSS on the same connection as soon as it parses the link element.)

youngtaff 12y ago
Yeh would be interesting so what performance would be like with the origin served through the CDN or even using dns-prefetch or tcp-preconnect I think all we can gather from this example is that MaxCDNs TLS implementation wasn't optimal and they were working on it Would be interesting to see how this behaves for other CDNs or with link rel= optimisations in place

jgrahamc 12y ago

See also "Using CloudFlare to mix domain sharding and SPDY": https://blog.cloudflare.com/using-cloudflare-to-mix-domain-sharding-and-spdy/ https://blog.cloudflare.com/using-cloudflare-to-mix-domain-s...

mdekkers 12y ago

We optimise and host websites for our clients, use an https/spdy CDN exclusively, most sites we work on are dogs when it comes to performance, and aim for half second page load times on average. Anything over 1 second is unacceptable. I read this article before, I think it was posted to HN previously, and wasn't impressed at the time. Still not impressed now.... Long story short: Not using a CDN is stupid. Learn to CDN.

puppetmaster3 12y ago

Why we don't use a CDN: NIH.

wampus 12y ago
In this case, that's an appropriate attitude. If your environment is a mess and you add complexity with a CDN, you've got a complicated mess. But a local optimization like this benefits your entire organization and puts off the need for a CDN (a solution to a problem that no longer exists). They may grow to a point where a real strain is being put on their systems, they can't make local improvements, and involving a CDN is the right choice.

troels 12y ago

In the Australia comparison, the CDN version spends almost 1.5 second on DNS lookup for the main domain. In the non-CDN version this figure is much lower. It seems disingenuous to attribute this overhead to the CDN, no? And as an aside - Maybe there are some low hanging fruits in tuning their DNS setup instead? 1.5s seems extremely high to me.

dedalus 12y ago

http://www.webpagetest.org/result/150402_TV_f0fe942e49dc1191e818c0cecba48d76/ http://www.webpagetest.org/result/150402_TV_f0fe942e49dc1191... shows you take 8 seconds to load in Australia. Using http://www.webpagetest.org/result/150402_22_0fa16aec7ebc410a00737efcd1321d3e/ http://www.webpagetest.org/result/150402_22_0fa16aec7ebc410a... also shows 8 seconds from tokyo http://www.webpagetest.org/result/150402_AN_22f88770d4bab89c84bd986d75da50ae/ http://www.webpagetest.org/result/150402_AN_22f88770d4bab89c... shows 8 seconds from India http://www.webpagetest.org/result/150402_CT_49277f5267e82b6b75a5f2bc9a3cf589/ http://www.webpagetest.org/result/150402_CT_49277f5267e82b6b... shows 36 seconds from Amsetrdam http://www.webpagetest.org/result/150402_9G_0810eaa940dc2941d0ff83615913e959/ http://www.webpagetest.org/result/150402_9G_0810eaa940dc2941... shows 12 seconds from Buenos Aires Not so sure that you dont need a CDN based on this