7 ms·

How to build your own CDN using BIND, GeoIP, Nginx, Varnish

100 points by nreece 16y ago

buro9 16y ago
Nice. The instructions assume some familiarity with BIND and Varnish, what I'd love to see would be a Linode article explaining how to use their datacenters to do this. Specifically I'd love to put noes in 2 US datacenters and the London one to help reduce my page load speed for the US visitors, whilst leaving the dynamic nodes all in London where most of my users are.
js4all 16y ago
Great article. I always wondered how this is done. I was aware of varnish and nginx, the missing link was the patched bind9 version. Many thanks. (btw. a little typo: varnish forwards to nginx on port 90 -> should be port 91)
komplex 16y ago
http://poramin.com/post/818944318/install-bind-with-support-geoip-on-ubuntu http://poramin.com/post/818944318/install-bind-with-support-...
pquerna 16y ago
If you want a non-patched server, built for this task, consider pgeodns: http://geo.bitnames.com/ http://geo.bitnames.com/ It is used by svn.apache.org, CPAN, the NTP pool, etc. I personally have my doubts about any value created for static pages doing this though, there are relatively cheap CDNs on the market now, rather than trying to find VMs in a dozen countries. In addition you have to be careful about services like OpenDNS which for example, made svn.apache.org always resolve to the EU even for users in the US, due to how their DNS anycast is setup.
- buro9 16y ago
  > I personally have my doubts about any value created for static pages doing this though, there are relatively cheap CDNs on the market now, rather than trying to find VMs in a dozen countries. Please share the options. My scenario is that I transfer around 800MB in static content (images, CSS and JS) per month. I'd simply want that part CDN'd. I used to use SimpleCDN but found them too expensive after they changed their pricing model. http://www.simplecdn.com/pricing http://www.simplecdn.com/pricing http://www.simplecdn.com/savings http://www.simplecdn.com/savings The cost of 2 additional Linodes would be (2 x $19.95) = $39.90 per month. My bandwidth usage isn't so great as to make the cost of an off-the-shelf CDN solution beneficial, yet it's great enough that a couple of Linodes would make a visible speed difference to the 100,000 unique visitors per month that visit my site from North America. To my mind there is a gap between small site and large site into which running a couple of VMs to provide a cheap CDN is an advantage. Although if you can show me cheaper drop-in (pass-through) alternatives to SimpleCDN then I'm very eager to hear about them.
  - pquerna 16y ago
    MaxCDN: http://www.maxcdn.com/pricing.php http://www.maxcdn.com/pricing.php Softlayer's CDN: http://www.softlayer.com/services/content-delivery-network/ http://www.softlayer.com/services/content-delivery-network/ (its re-branded Internap CDN, but they give you on-demand pricing) CacheFly, but you need to upload files to them: http://cachefly.com/pricing.html http://cachefly.com/pricing.html (Amazon CloudFiles too, but again, you need to upload files to them). In addition, if you are just using GeoDNS to load balancing between two US east/west locations, the big wins with GeoDNS are really about users in Australia, Asia or Europe, for closer localities, the only way really is to use an Anycast base CDN, because the odds of an ISPs DNS servers not being close enough for GeoDNS are far too high.
  - jasonkester 16y ago
    Sounds like you're looking for CloudFront. I run a site with similar usage to yours (at least on the upload side), and I see about $6 in S3 charges and another $8 for CloudFront each month. As the grandparent said, I just can't see why anybody would spend even a minute thinking about rolling their own CDN on their own servers for anything but the most extreme use cases. It's a solved problem, and it's ridiculously cheap.
  - buro9 16y ago
    I got my numbers wrong thanks to an early morning typo... 800GB bandwidth per month, not 800MB. Of the options recommended: Max CDN = $39.95 per month deal (1TB), usually $99.95 = Same price as DIY on the deal, or more expensive normally. SoftLayer = $140 per month (1TB) = More expensive. CacheFly = $299 per month (2TB) = More expensive and have to upload files to server. Rackspace CloudFiles = $176 per month (based on current 800GB bandwidth alone - other costs not yet factored in) = More expensive. Amazon CloudFront = $131 per month (based on current 800GB bandwidth alone - other costs not yet factored in) = More expensive and you have to upload files to the server. So yeah... DIY looks substantially cheaper. There's still a big gap between small site and large site. For a few days of my time I can easily save myself a lot of potential cost as well as speed up my website. I can't really see why I wouldn't set up my own CDN.
    - 16y ago
- jjoe 16y ago
  For part 2, we intend on showing how the CDN can cache dynamic content, not just static files. As a matter of fact, it's just one or two simple changes in Varnish VCL functions. Regards
papaf 16y ago
This is an interesting article but one thing I don't yet understand (which may become clear in part 2) is why both Nginx and Varnish are needed. Nginx can certainly do this on its own, and as far as I know, so can Varnish which has been designed to handle slow backends.
- forkqueue 16y ago
  I agree, nginx seems completely superfluous here. Actually, it's worse than superfluous - the way it's configured, nginx seems to be caching dynamic files but never expiring them.
- jjoe 16y ago
  1) Nginx is a fully fledged Web server and has some performance penalty compared to Varnish. The latter is lightening fast. 2) Nginx is no match for Varnish's VCL for header/cache manipulation Regards
  - hphp 16y ago
    1- yes but at the very highest loads 2- you don't need that much manip. for simple load balancing. 3- can you backup your claims with reference(s)?
    - jjoe 16y ago
      1 - not necessarily 2 - it depends on app complexity. Most people that read Y need it. 3 - Benchmarks? sure :) http://www.unixy.net/apache-vs-litespeed http://www.unixy.net/apache-vs-litespeed
      - zepolen 16y ago
        What does an Apache v Litespeed benchmark have to do with anything?
        jjoe 16y ago
        The above post proposed that Varnish is not needed because Nginx could handle the task well with the best performance. But Nginx is a bit slower than Varnish. It's been shown in benchmarks that Litespeed is a bit faster than Nginx. By inductive reasoning to show that Varnish is faster than Nginx, I posted a link to show that Varnish is faster than Litespeed, which in turn is faster than Nginx. So Varnish is faster than Nginx. Let me know if you have any question/comment. Regards
rmoriz 16y ago
I would use Powerdns instead of BIND. It's opensource and already comes with a geo backend (among others). http://doc.powerdns.com/geo.html#AEN6738 http://doc.powerdns.com/geo.html#AEN6738 The doc says, Wikimedia uses PowerDNS' geo backend.
davidu 16y ago
Relying on MaxMind for your CDN location targetting is not how I would recommend doing this. Anycast is better these days as it will taking routing topology into consideration over a hardcoded list of IP addresses. And routing topology can then be adjusted to account for congestion and other variables.
Goosey 16y ago
Great article to help me understand a bit more on how CDNs work! As others have pointed out the cost-effectiveness of deploying your own CDN is questionable in this climate, but the KNOWLEDGE of how your leased CDN works is certainly valuable!

100 points by nreece 16y ago

buro9 16y ago

Nice. The instructions assume some familiarity with BIND and Varnish, what I'd love to see would be a Linode article explaining how to use their datacenters to do this. Specifically I'd love to put noes in 2 US datacenters and the London one to help reduce my page load speed for the US visitors, whilst leaving the dynamic nodes all in London where most of my users are.

js4all 16y ago

Great article. I always wondered how this is done. I was aware of varnish and nginx, the missing link was the patched bind9 version. Many thanks. (btw. a little typo: varnish forwards to nginx on port 90 -> should be port 91)

komplex 16y ago

http://poramin.com/post/818944318/install-bind-with-support-geoip-on-ubuntu http://poramin.com/post/818944318/install-bind-with-support-...

pquerna 16y ago

If you want a non-patched server, built for this task, consider pgeodns: http://geo.bitnames.com/ http://geo.bitnames.com/ It is used by svn.apache.org, CPAN, the NTP pool, etc. I personally have my doubts about any value created for static pages doing this though, there are relatively cheap CDNs on the market now, rather than trying to find VMs in a dozen countries. In addition you have to be careful about services like OpenDNS which for example, made svn.apache.org always resolve to the EU even for users in the US, due to how their DNS anycast is setup.

buro9 16y ago
> I personally have my doubts about any value created for static pages doing this though, there are relatively cheap CDNs on the market now, rather than trying to find VMs in a dozen countries. Please share the options. My scenario is that I transfer around 800MB in static content (images, CSS and JS) per month. I'd simply want that part CDN'd. I used to use SimpleCDN but found them too expensive after they changed their pricing model. http://www.simplecdn.com/pricing http://www.simplecdn.com/pricing http://www.simplecdn.com/savings http://www.simplecdn.com/savings The cost of 2 additional Linodes would be (2 x $19.95) = $39.90 per month. My bandwidth usage isn't so great as to make the cost of an off-the-shelf CDN solution beneficial, yet it's great enough that a couple of Linodes would make a visible speed difference to the 100,000 unique visitors per month that visit my site from North America. To my mind there is a gap between small site and large site into which running a couple of VMs to provide a cheap CDN is an advantage. Although if you can show me cheaper drop-in (pass-through) alternatives to SimpleCDN then I'm very eager to hear about them.
- pquerna 16y ago
  MaxCDN: http://www.maxcdn.com/pricing.php http://www.maxcdn.com/pricing.php Softlayer's CDN: http://www.softlayer.com/services/content-delivery-network/ http://www.softlayer.com/services/content-delivery-network/ (its re-branded Internap CDN, but they give you on-demand pricing) CacheFly, but you need to upload files to them: http://cachefly.com/pricing.html http://cachefly.com/pricing.html (Amazon CloudFiles too, but again, you need to upload files to them). In addition, if you are just using GeoDNS to load balancing between two US east/west locations, the big wins with GeoDNS are really about users in Australia, Asia or Europe, for closer localities, the only way really is to use an Anycast base CDN, because the odds of an ISPs DNS servers not being close enough for GeoDNS are far too high.
- jasonkester 16y ago
  Sounds like you're looking for CloudFront. I run a site with similar usage to yours (at least on the upload side), and I see about $6 in S3 charges and another $8 for CloudFront each month. As the grandparent said, I just can't see why anybody would spend even a minute thinking about rolling their own CDN on their own servers for anything but the most extreme use cases. It's a solved problem, and it's ridiculously cheap.
- buro9 16y ago
  I got my numbers wrong thanks to an early morning typo... 800GB bandwidth per month, not 800MB. Of the options recommended: Max CDN = $39.95 per month deal (1TB), usually $99.95 = Same price as DIY on the deal, or more expensive normally. SoftLayer = $140 per month (1TB) = More expensive. CacheFly = $299 per month (2TB) = More expensive and have to upload files to server. Rackspace CloudFiles = $176 per month (based on current 800GB bandwidth alone - other costs not yet factored in) = More expensive. Amazon CloudFront = $131 per month (based on current 800GB bandwidth alone - other costs not yet factored in) = More expensive and you have to upload files to the server. So yeah... DIY looks substantially cheaper. There's still a big gap between small site and large site. For a few days of my time I can easily save myself a lot of potential cost as well as speed up my website. I can't really see why I wouldn't set up my own CDN.
  - 16y ago
jjoe 16y ago
For part 2, we intend on showing how the CDN can cache dynamic content, not just static files. As a matter of fact, it's just one or two simple changes in Varnish VCL functions. Regards

papaf 16y ago

This is an interesting article but one thing I don't yet understand (which may become clear in part 2) is why both Nginx and Varnish are needed. Nginx can certainly do this on its own, and as far as I know, so can Varnish which has been designed to handle slow backends.

forkqueue 16y ago
I agree, nginx seems completely superfluous here. Actually, it's worse than superfluous - the way it's configured, nginx seems to be caching dynamic files but never expiring them.
jjoe 16y ago
1) Nginx is a fully fledged Web server and has some performance penalty compared to Varnish. The latter is lightening fast. 2) Nginx is no match for Varnish's VCL for header/cache manipulation Regards
- hphp 16y ago
  1- yes but at the very highest loads 2- you don't need that much manip. for simple load balancing. 3- can you backup your claims with reference(s)?
  - jjoe 16y ago
    1 - not necessarily 2 - it depends on app complexity. Most people that read Y need it. 3 - Benchmarks? sure :) http://www.unixy.net/apache-vs-litespeed http://www.unixy.net/apache-vs-litespeed
    - zepolen 16y ago
      What does an Apache v Litespeed benchmark have to do with anything?
      - jjoe 16y ago
        The above post proposed that Varnish is not needed because Nginx could handle the task well with the best performance. But Nginx is a bit slower than Varnish. It's been shown in benchmarks that Litespeed is a bit faster than Nginx. By inductive reasoning to show that Varnish is faster than Nginx, I posted a link to show that Varnish is faster than Litespeed, which in turn is faster than Nginx. So Varnish is faster than Nginx. Let me know if you have any question/comment. Regards

rmoriz 16y ago

I would use Powerdns instead of BIND. It's opensource and already comes with a geo backend (among others). http://doc.powerdns.com/geo.html#AEN6738 http://doc.powerdns.com/geo.html#AEN6738 The doc says, Wikimedia uses PowerDNS' geo backend.

davidu 16y ago

Relying on MaxMind for your CDN location targetting is not how I would recommend doing this. Anycast is better these days as it will taking routing topology into consideration over a hardcoded list of IP addresses. And routing topology can then be adjusted to account for congestion and other variables.

Goosey 16y ago

Great article to help me understand a bit more on how CDNs work! As others have pointed out the cost-effectiveness of deploying your own CDN is questionable in this climate, but the KNOWLEDGE of how your leased CDN works is certainly valuable!