12 ms·

Optimizing web servers for high throughput and low latency

987 points by nuriaion 9y ago

indescions_2017 9y ago
Great write-up. And even if you use standard instances there is plenty to optimize. Kudos to Dropbox, Netflix, Cloudflare and everyone else for who demonstrates this level of transparency. And just for reference, AWS does provide enhanced networking capabilities on VPC: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-...
alinspired 9y ago
Great write up for traditional/kernel tuning! I guess i'm naively waiting for dpdk-based user space solutions to appear.
mfonda 9y ago
Great post! Contents of it aside, I very much like the disclaimer: > In this post we’ll be discussing lots of ways to tune web servers and proxies. Please do not cargo-cult them. For the sake of the scientific method, apply them one-by-one, measure their effect, and decide whether they are indeed useful in your environment. Far to often I see people apply ideas from posts they've read or talks they've seen without stopping to think whether or not it makes sense in the context they're applying it. Always think about context, and measure to make sure it actually works!
dragonwarrior 9y ago
Very interesting
mixedbit 9y ago
Does the physical location of a server matter for high-throughput use cases? If a client is downloading large files using all its available bandwith, is the download time noticeably better if the server is close to the client?
- SaveTheRbtz 9y ago
  Physical locations matters a lot for uploads, mostly because client TCP stack is way less sophisticated. For downloads high RTT can be mitigated by a congestion control that ignores constant packet-loss rates (which are common for high-rtt paths). Other tricks that you can try: fq+pacing and newer kernels with more sophisticated recovery heuristics.
- andy_ppp 9y ago
  I was actually looking into this the other day - latency from Japan to Virginia is over 200ms and then 32ms to grab the actual HTML the page. I reduced my SSL certificate from 4096 bits and it's definitely helped (latency with SSL was crazy, over 500ms). Even so, need to think at some point about a datacenter distributed code base :-/
stuxnet78 9y ago
Interesting article. Well, to be honest, some of the concepts were totally new to me and learning from this article found it interesting. And thanks for the other links too.
samber 9y ago
Last week, I was building, a live profiling platform (application level), based on Prometheus + Grafana. Still in a WIP status, but it can already display a live flamegraph of a running NodeJS app in Grafana: - https://github.com/samber/node-promfiler https://github.com/samber/node-promfiler - https://github.com/samber/grafana-flamegraph-panel https://github.com/samber/grafana-flamegraph-panel
siner 9y ago
"If you’ve read this far you probably want to work on solving these and other interesting problems! You’re in luck: Dropbox is looking for experienced SWEs, SREs, and Managers." Following the link will show "Open Positions" with, well, nothing to follow. They did not only optimize their servers for throughput but also HR!
Unisecure 9y ago
Know these Linux backup best practices to avoid data loss. https://buff.ly/2gJK5Ew https://buff.ly/2gJK5Ew
deleted 9y ago
[deleted]
dlecorfec 9y ago
I find it difficult to test a server in "real" conditions, these being at least 300k pps rx and 6 gbps tx, on 2x10G (with LACP) (context = http video streaming) Using a tool like iperf is useful for getting xx gbps, but I've never managed to reproduce high pps, while it seems that high rx pps is often the limiting factor (causing high softirq %cpu thus packet drops if reaching 100%).
justin66 9y ago
> You should keep your firmware up-to-date to avoid painful and lengthy troubleshooting sessions. Try to stay recent with CPU Microcode, Motherboard, NICs, and SSDs firmwares. I wonder if this is good advice. I would have said the opposite: do not mess around with any of that stuff unless there's a security advisory or a problem points to a specific piece of hardware. It's not like updating this stuff is without risk.
- SaveTheRbtz 9y ago
  > It's not like updating this stuff is without risk. Same goes for kernel, libc, all other libraries, and pretty much anything else. This is not an argument for not upgrading them though. Also, sadly, almost all firmware updates fix a "problem [that] points to a specific piece of hardware" that you use.
  - justin66 9y ago
    > Same goes for kernel, libc, all other libraries, and pretty much anything else. This is not an argument for not upgrading them though. If you have a budget to responsibly keep all that stuff up to date in your project and test to make sure you haven't broken anything, more power to you. It's a total waste most of the time, but why not? Especially if you're a consultant and getting paid by the hour Firmware is different. It's not always possible to back out a change and you risk bricking hardware every time you apply an update, especially in the realm of PC-based servers. If something like a NIC or motherboard is performing as expected, updating the firmware for no reason is generally a stupid thing to do. Edit: oh, you're the author. Please stop telling people to do this, or at least explain the risk. At a minimum, if the vendor provides a defect list with each update, it's not necessary to blindly apply updates. Take the changes if they're needed. It's possible you're accustomed to working on high-grade hardware that exhibits fewer of these firmware related blowups, but that doesn't mean it does not happen to people...
brendangregg 9y ago
Excellent!
skarap 9y ago
One congestion-control parameter which don't get it's fair share of articles is initcwnd. And for HTTP traffic over high bandwidth+latency links it has much larger impact than the choice of the cc algorithm. See https://www.cdnplanet.com/blog/tune-tcp-initcwnd-for-optimum-performance/ https://www.cdnplanet.com/blog/tune-tcp-initcwnd-for-optimum...
- bogomipz 9y ago
  Initcwnd is set 10 in just about every distro and has been for some time now. Initcwnd only comes into play at the beginning of a new connection, it isn't as important for LFNs as TCP buffer sizes. But at any rate this post is concerned with edge networks so there shouldn't be any LFN network at all.
- SaveTheRbtz 9y ago
  I've mentioned it very briefly in a Network Stack part, around the "upgrade your kernel" warning: > and I’m not even talking about IW10 (which is so 2010)[1] [1] https://developers.google.com/speed/protocols/Increasing_TCPs_Initial_Window_IETF78.pdf https://developers.google.com/speed/protocols/Increasing_TCP...
nvarsj 9y ago
This is really great. I love these kind of articles! I learned a few things (like about the non-cubic tcp congestion algorithms). Probably orthogonal to TFA, but I'd be interested to know though how they solve 0 downtime nginx deployments.
Unisecure 9y ago
If you are looking for web servers for high throughput & low latency then go for Unisecure, data center & hosting provider. Unisecure is also well known as one of the best Windows and Linux VPS hosting provider in US for startups with affordable and quality-driven services.
DamonHD 9y ago
Fab tour de force! Great! (Though I've been optimising my tiny low-traffic site on a solar-powered RPi 2 to apparently outperform a major CDN, at least on POHTTP...)
- baybal2 9y ago
  DPDK? One fabless SoC maker called Baikal claim that they can saturate 10gbs link with an 8 core arm chip running under 5W (phy power not counted in) for as long as users are using userspace driven network and hardware offloading
  - shaklee3 9y ago
    DPDK is typically benchmarked in packets per second instead of bitrate. The reason being that 64-byte packets at 10Gbps is much harder than 512+. Also, it depends on what the work is. You can do several hundred Gbps on a moderate Xeon, provided you're just dropping all the packets as they come in.
- turdnagel 9y ago
  Very interesting! What steps have you taken to optimize?
  - DamonHD 9y ago
    Well, here's a whole slew: http://www.earth.org.uk/note-on-site-technicals-2.html http://www.earth.org.uk/note-on-site-technicals-2.html
- gardnr 9y ago
  This sounds unbelievable
- cat199 9y ago
  not surprising imho - you probably have more 'dedicated hardware' (in an abstract sense) per request than a busy CDN.. if the site software is all cached in ram and cores are free, you're pretty much running 'wide open' on each request..
hartator 9y ago
I wish they could make a MacOSX app that doesn't use almost 100% of one core all the time.
- cabaalis 9y ago
  Sorry you're downvoted, but what you mention is a real concern.
  - ovao 9y ago
    Not to this topic.
Upvoter33 9y ago
This is a wonderful article. It also is indicative of why people who really understand systems will always be so employable - it's just so hard to make things run well.
Thaxll 9y ago
"people are still copy-pasting their old sysctls.conf that they’ve used to tune 2.6.18/2.6.32 kernels." This.
- korzun 9y ago
  .. or 32 bit systems.
excitom 9y ago
This is a great article and I've bookmarked it for future reference. I observe, though, that if you are tuning a system to this level of detail you likely have a number of web servers behind a load balancer. To be complete, the discussion should include optimization of interactions with the load balancer, e.g. where to terminate https, etc.
- annnnd 9y ago
  > should include optimization of interactions with the load balancer, e.g. where to terminate https, etc. I'm curious - is there a case where you want to terminate HTTPS on end device instead of (only) on load balancer?
  - toast0 9y ago
    Ignoring any privacy implications, terminating https on the load balancer means the load balancer will use more cpu and memory than if it was just a tcp terminating load balancer or working at layer 2. In a lot of architectures, the load balancer may not have as nice of a failover story as the hosts behind it, minimizing state on the load balancer could make state syncing possible for a quicker failover. If you're running load balancing appliances, those tend to be expensive, so you probably want to let them only do what they have to, so you don't need to buy more of them.
    - annnnd 9y ago
      Thanks!
- daenney 9y ago
  > To be complete, the discussion should include optimization of interactions with the load balancer, e.g. where to terminate https, etc. Not necessarily. There are other ways of spreading traffic, like DNS round robin for example, using the DNS delegation trick or even have client-side load balancing (which is perfectly feasible if you control the client, like your own app for example). The article itself mentions that what they're talking about is the Dropbox Edge network, an nginx proxy tier, which sounds like load balancers to me.
abc_lisper 9y ago
Just goes to show how much one should know in our field to make the machine work well for you. For somebody that can understand the article, the stuff is mostly known, but if you don't know it, the article is pretty dense. It would be nice if someone make a docker image with all the tuning set (except the hardware) It would have be nicer, if the author has shown what the end result of this optimization looks like, with numbers, comparing against a standard run-of-the-mill nginx setup.
- mayank 9y ago
  > It would be nice if someone make a docker image with all the tuning set (except the hardware) This degree of optimization is something that really depends on a specific use case and the precise configuration of the entire hardware+software stack, and not a general-purpose best practices list that can be put into a container. The author says as much at the top in the disclaimer: > Please do not cargo-cult them. For the sake of the scientific method, apply them one-by-one, measure their effect, and decide wether they are indeed useful in your environment. Better to take away the profiling and optimization methodologies from the article rather than specific settings.
- eugeneionesco 9y ago
  >It would be nice if someone make a docker image with all the tuning set (except the hardware) Have you not read the article? >In this post we’ll be discussing lots of ways to tune web servers and proxies. Please do not cargo-cult them. For the sake of the scientific method, apply them one-by-one, measure their effect, and decide whether they are indeed useful in your environment.
  - TomMarius 9y ago
    I don't think they meant for production, just testing and toying with it.
- tokenizerrr 9y ago
  The entire idea of tuning is that you tune it to your particular workload.
doubleorseven 9y ago
wow. This is the kind of article i'm expecting to see when I google for "Nginx best practices 2018". I am so far behind, maybe 20% of my usual setup include those recommendations. Thank you Dropbox. If someone can point me to a thorough article like this on the lua module, I will thank her/him forever.
- etiene 9y ago
  Maybe you will find some stuff here? https://openresty.org/en/presentations.html https://openresty.org/en/presentations.html
leetrout 9y ago
I'm surprised there was no mention of tcp_max_syn_backlog and netdev_max_backlog. When I've previously tuned a server I have used both of those to my advantage... Another comment on here talked about this ignoring an existing load balancer so maybe those sysctls are more appropriate on an LB?
- korzun 9y ago
  Those settings are irrelevant in this scenario. The point is to maximize the throughput from a single server without backing up the queue.
mozumder 9y ago
So how does Linux compare now with FreeBSD in terms of throughput and latency? I remember like 10 years ago Linux had issues with throughput, which is why Netflix went with FreeBSD. Are they similar now?
- Thaxll 9y ago
  Linux is probably faster and has always been for years now. Benchmark a recent kernel and prove me wrong.
  - 2trill2spill 9y ago
    > Linux is probably faster and has always been for years now. Benchmark a recent kernel and prove me wrong. How about you post the benchmarks you used to form your answer, otherwise your post is nothing but speculation and probably just favoritism.
  - rhizome 9y ago
    What good is benchmarking a kernel going to do when performance in this context has many more moving parts?
- 2trill2spill 9y ago
  > So how does Linux compare now with FreeBSD in terms of throughput and latency? I remember like 10 years ago Linux had issues with throughput, which is why Netflix went with FreeBSD. Are they similar now? I believe the open connect team at Netflix choose FreeBSD because a lot of them used to work at Yahoo and had lot's of FreeBSD experience. Not so much because of a performance difference between the two. As for now, the two network stacks are pretty equal when it comes to performance, some work loads are better on FreeBSD some are better on Linux.
  - ovao 9y ago
    Can you go into more detail about what workloads are better suited for Linux and FreeBSD?
jbergstroem 9y ago
I thoroughly appreciate how the majority of the article doesn't even go into the nginx config whereas most of the internet would discuss a result of the search query "optimized nginx config". Much [for me] to learn and much appreciated, Alexey.
- mfjordvald 9y ago
  One of my more popular blog posts is exactly about how to "optimise" nginx for high traffic loads and the gist of it is pretty much "you can't really do much, but here's some minor stuff". I consider that a good thing though, a web server shouldn't really require config tweaking to perform well.
  - derefr 9y ago
    I recall there being a good presentation by WhatsApp about optimizing not just the web server, but the OS (FreeBSD, in their case) to solve the C100K/C1M problem. It seemed that there was a lot more low-hanging fruit on the OS side than on the webserver side.
  - jbergstroem 9y ago
    Definitely! Sane defaults and config readability is pretty much the main reason I recommend nginx.
  - bogomipz 9y ago
    >"One of my more popular blog posts is exactly about how to "optimise" nginx for high traffic loads and the gist of it is pretty much "you can't really do much, but here's some minor stuff"." Could you provide a link for that blog post? I am not seeing it.
    - 62747478182 9y ago
      Maybe https://blog.martinfjordvald.com/2011/04/optimizing-nginx-for-high-traffic-loads/ https://blog.martinfjordvald.com/2011/04/optimizing-nginx-fo...
- 9y ago
tiffanyh 9y ago
My mind kind of explodes reading that article. So many bells & whistles and I don't even know where to begin.
cat199 9y ago
great article overall - but starts off by saying 'do not cargo cult this' and then proceeds to proscribe many 'mandates' without giving any rationale behind them..
- SaveTheRbtz 9y ago
  I've tried to: 1) give an bcc/perf example to check the need for tuning and verify its effect. 2) give code/docs/paper reference as an embedded link. 3) give a generic monitoring guideline at the start of a "chapter". Seems like I've (at least partially) failed. I'll do better next time.
pavs 9y ago
No mention of VPP - does it not apply to applications? or Routing/ switching? https://wiki.fd.io/view/VPP/What_is_VPP%3F https://wiki.fd.io/view/VPP/What_is_VPP%3F
- wmf 9y ago
  VPP has amazing performance, but it's mostly limited to switching, routing, NAT, and VPN at this point; it doesn't have (finished) TCP and porting a Web server like Nginx is a ways off. It's also questionable who needs something like serving 500 Gbps of Web traffic from a single server.
rcchen 9y ago
> If you are going with AMD, EPYC has quite impressive performance. Does this imply that Dropbox has started testing out EPYC metal?
korzun 9y ago
You can skip all of that nonsense and run FreeBSD.
- SaveTheRbtz 9y ago
  When I was at Yandex, we were a FreeBSD-CURRENT shop (FreeBSD 9 at that time). So, that said, FreeBSD will had exactly the same[0] issues on the low/mid levels, just not all of them had a good solution[1][2]. [0] https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerfo... [1] https://wiki.freebsd.org/NetworkPerformanceTuning https://wiki.freebsd.org/NetworkPerformanceTuning [2] https://wiki.freebsd.org/TransportProtocols https://wiki.freebsd.org/TransportProtocols
- jimktrains2 9y ago
  While I love freebsd, you can definitely take the time to tune it if your application requires it.
z3t4 9y ago
Would be cool with a story like, we did these adjustment so each server could handle 10% more requests. etc. This blog post seems to only cover software, you can also gain a lot of performance from hardware modding. Someone said optimizing is the root of all evil ... So first identify bottlenecks in real work-loads, no micro-benchmarking!
- raarts 9y ago
  That was: ''_premature_ optimization is the root of all evil".
exikyut 9y ago
I have a related question that most would probably consider relevant but which this article (quite rightly) doesn't answer (as it's not relevant for Netflix). Let's say I want to prepare a server to respond quickly to HTTP requests, from all over the world. How do I optimize where I put it? Generally there are three ways I can tackle this: 1. I configure/install my own server somewhere 2. Or rent a preconfigured dedicated server I can only do so much with 3. I rent Xen/KVM on a hopefully not-overcrowded/oversold host Obviously the 1st is the most expensive (I must own my own hardware; failures mean a trip to the DC or smart hands), the 2nd will remove some flexibility, and the 3rd will impose the most restrictions but be the cheapest. For reference, knowing how to pick a good network (#1) would be interesting to learn about. I've always been curious about that, although I don't exactly have anything to rack right now. Are there any physical locations in the world that will offer the lowest latency to the highest number of users? Do some providers have connections to better backbones? Etc. #2 is not impossible - https://cc.delimiter.com/cart/dedicated-servers/&step=0 https://cc.delimiter.com/cart/dedicated-servers/&step=0 currently lists a HP SL170s with dual L5360s, 24GB, 2TB and 20TB bandwidth @ 1Gbit for $50/mo. It's cool to know this kind of thing exists. But I don't know how good Delimiter's network(s) is/are (this is in Atlanta FWIW). #3 is what I'm the most interested in at this point, although this option does present the biggest challenge. Overselling is a tricky proposition. Hosting seems to be typically sold on the basis of how fast `dd` finishes (which is an atrocious and utterly wrong benchmark - most tests dd /dev/zero to a disk file, which will go through the disk cache). Not many people seem to setup a tuned Web server and then run ab or httperf on it from a remote with known-excellent networking. That's incredibly sad! Handling gaming or voice traffic is probably a good idea for the target I'd like to be able to hit - I don't want to do precisely that, but if my server's latency is good enough to handle that I'd be very happy.
- tomsthumb 9y ago
  Wouldn't dd'ing /dev/zero create an FS hole in your target file on many file systems? A similar (but maybe not isomorphic) situation in the Michael Kerrisk book is one of the excercises, and I'd bet ZFS does similar but suddenly we've assumed a lot of context.
  - 9y ago

987 points by nuriaion 9y ago

indescions_2017 9y ago

Great write-up. And even if you use standard instances there is plenty to optimize. Kudos to Dropbox, Netflix, Cloudflare and everyone else for who demonstrates this level of transparency. And just for reference, AWS does provide enhanced networking capabilities on VPC: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-...

alinspired 9y ago

Great write up for traditional/kernel tuning! I guess i'm naively waiting for dpdk-based user space solutions to appear.

mfonda 9y ago

Great post! Contents of it aside, I very much like the disclaimer: > In this post we’ll be discussing lots of ways to tune web servers and proxies. Please do not cargo-cult them. For the sake of the scientific method, apply them one-by-one, measure their effect, and decide whether they are indeed useful in your environment. Far to often I see people apply ideas from posts they've read or talks they've seen without stopping to think whether or not it makes sense in the context they're applying it. Always think about context, and measure to make sure it actually works!

dragonwarrior 9y ago

Very interesting

mixedbit 9y ago

Does the physical location of a server matter for high-throughput use cases? If a client is downloading large files using all its available bandwith, is the download time noticeably better if the server is close to the client?

SaveTheRbtz 9y ago
Physical locations matters a lot for uploads, mostly because client TCP stack is way less sophisticated. For downloads high RTT can be mitigated by a congestion control that ignores constant packet-loss rates (which are common for high-rtt paths). Other tricks that you can try: fq+pacing and newer kernels with more sophisticated recovery heuristics.
andy_ppp 9y ago
I was actually looking into this the other day - latency from Japan to Virginia is over 200ms and then 32ms to grab the actual HTML the page. I reduced my SSL certificate from 4096 bits and it's definitely helped (latency with SSL was crazy, over 500ms). Even so, need to think at some point about a datacenter distributed code base :-/

stuxnet78 9y ago

Interesting article. Well, to be honest, some of the concepts were totally new to me and learning from this article found it interesting. And thanks for the other links too.

samber 9y ago

Last week, I was building, a live profiling platform (application level), based on Prometheus + Grafana. Still in a WIP status, but it can already display a live flamegraph of a running NodeJS app in Grafana: - https://github.com/samber/node-promfiler https://github.com/samber/node-promfiler - https://github.com/samber/grafana-flamegraph-panel https://github.com/samber/grafana-flamegraph-panel

siner 9y ago

"If you’ve read this far you probably want to work on solving these and other interesting problems! You’re in luck: Dropbox is looking for experienced SWEs, SREs, and Managers." Following the link will show "Open Positions" with, well, nothing to follow. They did not only optimize their servers for throughput but also HR!

Unisecure 9y ago

Know these Linux backup best practices to avoid data loss. https://buff.ly/2gJK5Ew https://buff.ly/2gJK5Ew

deleted 9y ago

[deleted]

dlecorfec 9y ago

I find it difficult to test a server in "real" conditions, these being at least 300k pps rx and 6 gbps tx, on 2x10G (with LACP) (context = http video streaming) Using a tool like iperf is useful for getting xx gbps, but I've never managed to reproduce high pps, while it seems that high rx pps is often the limiting factor (causing high softirq %cpu thus packet drops if reaching 100%).

justin66 9y ago

> You should keep your firmware up-to-date to avoid painful and lengthy troubleshooting sessions. Try to stay recent with CPU Microcode, Motherboard, NICs, and SSDs firmwares. I wonder if this is good advice. I would have said the opposite: do not mess around with any of that stuff unless there's a security advisory or a problem points to a specific piece of hardware. It's not like updating this stuff is without risk.

SaveTheRbtz 9y ago
> It's not like updating this stuff is without risk. Same goes for kernel, libc, all other libraries, and pretty much anything else. This is not an argument for not upgrading them though. Also, sadly, almost all firmware updates fix a "problem [that] points to a specific piece of hardware" that you use.
- justin66 9y ago
  > Same goes for kernel, libc, all other libraries, and pretty much anything else. This is not an argument for not upgrading them though. If you have a budget to responsibly keep all that stuff up to date in your project and test to make sure you haven't broken anything, more power to you. It's a total waste most of the time, but why not? Especially if you're a consultant and getting paid by the hour Firmware is different. It's not always possible to back out a change and you risk bricking hardware every time you apply an update, especially in the realm of PC-based servers. If something like a NIC or motherboard is performing as expected, updating the firmware for no reason is generally a stupid thing to do. Edit: oh, you're the author. Please stop telling people to do this, or at least explain the risk. At a minimum, if the vendor provides a defect list with each update, it's not necessary to blindly apply updates. Take the changes if they're needed. It's possible you're accustomed to working on high-grade hardware that exhibits fewer of these firmware related blowups, but that doesn't mean it does not happen to people...

brendangregg 9y ago

Excellent!

skarap 9y ago

One congestion-control parameter which don't get it's fair share of articles is initcwnd. And for HTTP traffic over high bandwidth+latency links it has much larger impact than the choice of the cc algorithm. See https://www.cdnplanet.com/blog/tune-tcp-initcwnd-for-optimum-performance/ https://www.cdnplanet.com/blog/tune-tcp-initcwnd-for-optimum...

bogomipz 9y ago
Initcwnd is set 10 in just about every distro and has been for some time now. Initcwnd only comes into play at the beginning of a new connection, it isn't as important for LFNs as TCP buffer sizes. But at any rate this post is concerned with edge networks so there shouldn't be any LFN network at all.
SaveTheRbtz 9y ago
I've mentioned it very briefly in a Network Stack part, around the "upgrade your kernel" warning: > and I’m not even talking about IW10 (which is so 2010)[1] [1] https://developers.google.com/speed/protocols/Increasing_TCPs_Initial_Window_IETF78.pdf https://developers.google.com/speed/protocols/Increasing_TCP...

nvarsj 9y ago

This is really great. I love these kind of articles! I learned a few things (like about the non-cubic tcp congestion algorithms). Probably orthogonal to TFA, but I'd be interested to know though how they solve 0 downtime nginx deployments.

Unisecure 9y ago

If you are looking for web servers for high throughput & low latency then go for Unisecure, data center & hosting provider. Unisecure is also well known as one of the best Windows and Linux VPS hosting provider in US for startups with affordable and quality-driven services.

DamonHD 9y ago

Fab tour de force! Great! (Though I've been optimising my tiny low-traffic site on a solar-powered RPi 2 to apparently outperform a major CDN, at least on POHTTP...)

baybal2 9y ago
DPDK? One fabless SoC maker called Baikal claim that they can saturate 10gbs link with an 8 core arm chip running under 5W (phy power not counted in) for as long as users are using userspace driven network and hardware offloading
- shaklee3 9y ago
  DPDK is typically benchmarked in packets per second instead of bitrate. The reason being that 64-byte packets at 10Gbps is much harder than 512+. Also, it depends on what the work is. You can do several hundred Gbps on a moderate Xeon, provided you're just dropping all the packets as they come in.
turdnagel 9y ago
Very interesting! What steps have you taken to optimize?
- DamonHD 9y ago
  Well, here's a whole slew: http://www.earth.org.uk/note-on-site-technicals-2.html http://www.earth.org.uk/note-on-site-technicals-2.html
gardnr 9y ago
This sounds unbelievable
cat199 9y ago
not surprising imho - you probably have more 'dedicated hardware' (in an abstract sense) per request than a busy CDN.. if the site software is all cached in ram and cores are free, you're pretty much running 'wide open' on each request..

hartator 9y ago

I wish they could make a MacOSX app that doesn't use almost 100% of one core all the time.

cabaalis 9y ago
Sorry you're downvoted, but what you mention is a real concern.
- ovao 9y ago
  Not to this topic.

Upvoter33 9y ago

This is a wonderful article. It also is indicative of why people who really understand systems will always be so employable - it's just so hard to make things run well.

Thaxll 9y ago

"people are still copy-pasting their old sysctls.conf that they’ve used to tune 2.6.18/2.6.32 kernels." This.

korzun 9y ago
.. or 32 bit systems.

excitom 9y ago

This is a great article and I've bookmarked it for future reference. I observe, though, that if you are tuning a system to this level of detail you likely have a number of web servers behind a load balancer. To be complete, the discussion should include optimization of interactions with the load balancer, e.g. where to terminate https, etc.

annnnd 9y ago
> should include optimization of interactions with the load balancer, e.g. where to terminate https, etc. I'm curious - is there a case where you want to terminate HTTPS on end device instead of (only) on load balancer?
- toast0 9y ago
  Ignoring any privacy implications, terminating https on the load balancer means the load balancer will use more cpu and memory than if it was just a tcp terminating load balancer or working at layer 2. In a lot of architectures, the load balancer may not have as nice of a failover story as the hosts behind it, minimizing state on the load balancer could make state syncing possible for a quicker failover. If you're running load balancing appliances, those tend to be expensive, so you probably want to let them only do what they have to, so you don't need to buy more of them.
  - annnnd 9y ago
    Thanks!
daenney 9y ago
> To be complete, the discussion should include optimization of interactions with the load balancer, e.g. where to terminate https, etc. Not necessarily. There are other ways of spreading traffic, like DNS round robin for example, using the DNS delegation trick or even have client-side load balancing (which is perfectly feasible if you control the client, like your own app for example). The article itself mentions that what they're talking about is the Dropbox Edge network, an nginx proxy tier, which sounds like load balancers to me.

abc_lisper 9y ago

Just goes to show how much one should know in our field to make the machine work well for you. For somebody that can understand the article, the stuff is mostly known, but if you don't know it, the article is pretty dense. It would be nice if someone make a docker image with all the tuning set (except the hardware) It would have be nicer, if the author has shown what the end result of this optimization looks like, with numbers, comparing against a standard run-of-the-mill nginx setup.

mayank 9y ago
> It would be nice if someone make a docker image with all the tuning set (except the hardware) This degree of optimization is something that really depends on a specific use case and the precise configuration of the entire hardware+software stack, and not a general-purpose best practices list that can be put into a container. The author says as much at the top in the disclaimer: > Please do not cargo-cult them. For the sake of the scientific method, apply them one-by-one, measure their effect, and decide wether they are indeed useful in your environment. Better to take away the profiling and optimization methodologies from the article rather than specific settings.
eugeneionesco 9y ago
>It would be nice if someone make a docker image with all the tuning set (except the hardware) Have you not read the article? >In this post we’ll be discussing lots of ways to tune web servers and proxies. Please do not cargo-cult them. For the sake of the scientific method, apply them one-by-one, measure their effect, and decide whether they are indeed useful in your environment.
- TomMarius 9y ago
  I don't think they meant for production, just testing and toying with it.
tokenizerrr 9y ago
The entire idea of tuning is that you tune it to your particular workload.

doubleorseven 9y ago

wow. This is the kind of article i'm expecting to see when I google for "Nginx best practices 2018". I am so far behind, maybe 20% of my usual setup include those recommendations. Thank you Dropbox. If someone can point me to a thorough article like this on the lua module, I will thank her/him forever.

etiene 9y ago
Maybe you will find some stuff here? https://openresty.org/en/presentations.html https://openresty.org/en/presentations.html

leetrout 9y ago

I'm surprised there was no mention of tcp_max_syn_backlog and netdev_max_backlog. When I've previously tuned a server I have used both of those to my advantage... Another comment on here talked about this ignoring an existing load balancer so maybe those sysctls are more appropriate on an LB?

korzun 9y ago
Those settings are irrelevant in this scenario. The point is to maximize the throughput from a single server without backing up the queue.

mozumder 9y ago

So how does Linux compare now with FreeBSD in terms of throughput and latency? I remember like 10 years ago Linux had issues with throughput, which is why Netflix went with FreeBSD. Are they similar now?

Thaxll 9y ago
Linux is probably faster and has always been for years now. Benchmark a recent kernel and prove me wrong.
- 2trill2spill 9y ago
  > Linux is probably faster and has always been for years now. Benchmark a recent kernel and prove me wrong. How about you post the benchmarks you used to form your answer, otherwise your post is nothing but speculation and probably just favoritism.
- rhizome 9y ago
  What good is benchmarking a kernel going to do when performance in this context has many more moving parts?
2trill2spill 9y ago
> So how does Linux compare now with FreeBSD in terms of throughput and latency? I remember like 10 years ago Linux had issues with throughput, which is why Netflix went with FreeBSD. Are they similar now? I believe the open connect team at Netflix choose FreeBSD because a lot of them used to work at Yahoo and had lot's of FreeBSD experience. Not so much because of a performance difference between the two. As for now, the two network stacks are pretty equal when it comes to performance, some work loads are better on FreeBSD some are better on Linux.
- ovao 9y ago
  Can you go into more detail about what workloads are better suited for Linux and FreeBSD?

jbergstroem 9y ago

I thoroughly appreciate how the majority of the article doesn't even go into the nginx config whereas most of the internet would discuss a result of the search query "optimized nginx config". Much [for me] to learn and much appreciated, Alexey.

mfjordvald 9y ago
One of my more popular blog posts is exactly about how to "optimise" nginx for high traffic loads and the gist of it is pretty much "you can't really do much, but here's some minor stuff". I consider that a good thing though, a web server shouldn't really require config tweaking to perform well.
- derefr 9y ago
  I recall there being a good presentation by WhatsApp about optimizing not just the web server, but the OS (FreeBSD, in their case) to solve the C100K/C1M problem. It seemed that there was a lot more low-hanging fruit on the OS side than on the webserver side.
- jbergstroem 9y ago
  Definitely! Sane defaults and config readability is pretty much the main reason I recommend nginx.
- bogomipz 9y ago
  >"One of my more popular blog posts is exactly about how to "optimise" nginx for high traffic loads and the gist of it is pretty much "you can't really do much, but here's some minor stuff"." Could you provide a link for that blog post? I am not seeing it.
  - 62747478182 9y ago
    Maybe https://blog.martinfjordvald.com/2011/04/optimizing-nginx-for-high-traffic-loads/ https://blog.martinfjordvald.com/2011/04/optimizing-nginx-fo...
9y ago

tiffanyh 9y ago

My mind kind of explodes reading that article. So many bells & whistles and I don't even know where to begin.

cat199 9y ago

great article overall - but starts off by saying 'do not cargo cult this' and then proceeds to proscribe many 'mandates' without giving any rationale behind them..

SaveTheRbtz 9y ago
I've tried to: 1) give an bcc/perf example to check the need for tuning and verify its effect. 2) give code/docs/paper reference as an embedded link. 3) give a generic monitoring guideline at the start of a "chapter". Seems like I've (at least partially) failed. I'll do better next time.

pavs 9y ago

No mention of VPP - does it not apply to applications? or Routing/ switching? https://wiki.fd.io/view/VPP/What_is_VPP%3F https://wiki.fd.io/view/VPP/What_is_VPP%3F

wmf 9y ago
VPP has amazing performance, but it's mostly limited to switching, routing, NAT, and VPN at this point; it doesn't have (finished) TCP and porting a Web server like Nginx is a ways off. It's also questionable who needs something like serving 500 Gbps of Web traffic from a single server.

rcchen 9y ago

> If you are going with AMD, EPYC has quite impressive performance. Does this imply that Dropbox has started testing out EPYC metal?

korzun 9y ago

You can skip all of that nonsense and run FreeBSD.

SaveTheRbtz 9y ago
When I was at Yandex, we were a FreeBSD-CURRENT shop (FreeBSD 9 at that time). So, that said, FreeBSD will had exactly the same[0] issues on the low/mid levels, just not all of them had a good solution[1][2]. [0] https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerfo... [1] https://wiki.freebsd.org/NetworkPerformanceTuning https://wiki.freebsd.org/NetworkPerformanceTuning [2] https://wiki.freebsd.org/TransportProtocols https://wiki.freebsd.org/TransportProtocols
jimktrains2 9y ago
While I love freebsd, you can definitely take the time to tune it if your application requires it.

z3t4 9y ago

Would be cool with a story like, we did these adjustment so each server could handle 10% more requests. etc. This blog post seems to only cover software, you can also gain a lot of performance from hardware modding. Someone said optimizing is the root of all evil ... So first identify bottlenecks in real work-loads, no micro-benchmarking!

raarts 9y ago
That was: ''_premature_ optimization is the root of all evil".

exikyut 9y ago

I have a related question that most would probably consider relevant but which this article (quite rightly) doesn't answer (as it's not relevant for Netflix). Let's say I want to prepare a server to respond quickly to HTTP requests, from all over the world. How do I optimize where I put it? Generally there are three ways I can tackle this: 1. I configure/install my own server somewhere 2. Or rent a preconfigured dedicated server I can only do so much with 3. I rent Xen/KVM on a hopefully not-overcrowded/oversold host Obviously the 1st is the most expensive (I must own my own hardware; failures mean a trip to the DC or smart hands), the 2nd will remove some flexibility, and the 3rd will impose the most restrictions but be the cheapest. For reference, knowing how to pick a good network (#1) would be interesting to learn about. I've always been curious about that, although I don't exactly have anything to rack right now. Are there any physical locations in the world that will offer the lowest latency to the highest number of users? Do some providers have connections to better backbones? Etc. #2 is not impossible - https://cc.delimiter.com/cart/dedicated-servers/&step=0 https://cc.delimiter.com/cart/dedicated-servers/&step=0 currently lists a HP SL170s with dual L5360s, 24GB, 2TB and 20TB bandwidth @ 1Gbit for $50/mo. It's cool to know this kind of thing exists. But I don't know how good Delimiter's network(s) is/are (this is in Atlanta FWIW). #3 is what I'm the most interested in at this point, although this option does present the biggest challenge. Overselling is a tricky proposition. Hosting seems to be typically sold on the basis of how fast `dd` finishes (which is an atrocious and utterly wrong benchmark - most tests dd /dev/zero to a disk file, which will go through the disk cache). Not many people seem to setup a tuned Web server and then run ab or httperf on it from a remote with known-excellent networking. That's incredibly sad! Handling gaming or voice traffic is probably a good idea for the target I'd like to be able to hit - I don't want to do precisely that, but if my server's latency is good enough to handle that I'd be very happy.

tomsthumb 9y ago
Wouldn't dd'ing /dev/zero create an FS hole in your target file on many file systems? A similar (but maybe not isomorphic) situation in the Michael Kerrisk book is one of the excercises, and I'd bet ZFS does similar but suddenly we've assumed a lot of context.
- 9y ago