20 ms·
How we scaled Nginx
- LinuxBender 8y agoDoes Cloudflare submit all of it's improvements to Nginx upstream? Do nginx.org accept / merge the improvements?
- khc 8y agoAuthor here. I've talked to a nginx product manager and he's told me changes that are specific to one customer are unlikely to be accepted. Also in our implementation we took some shortcuts so it may not be suitable for upstream as is anyway
- LinuxBender 8y agoUnderstood. Perhaps you could make your improvements modular in some cases so that people can toggle them on, either as nginx modules, or compile flags in nginx core?
- khc 8y agoIn this case it's not possible to implement as nginx modules, but we are looking into releasing the patch as is.
- LinuxBender 8y agoVery nice! Thankyou for the improvements that your team is making.
- devwastaken 8y agoIt would be really great for some public projects if your internal modifications and updates to nginx were a public repo. That repo could be compiled and packaged for use of open source projects that benefit from those modifications. I say this because I've seen multiple patches from cloudflare around, but it's very difficult for 1 person to go through all of that, know what version of nginx it's for, and modify the patch for newer versions of NginX like security updates. If you modify nginx internally I don't doubt there's lots of various changes and improvements over time that don't get organized or published publicly. I think it'd be great if more companies released their own 'opinionated' versions that update with their infrastructure. Like if I wanted to host a OpenStreetMaps tiling server hypothetically using some features cloudflare has in their nginx builds. Makes it easy for white-hats to test, too. IIRC I've been intersted in HPACK for small http responses where the headers are larger than the body, but if I wanted to use the HPACK patch I have to re-impliment it every time an update comes out that modifies the file.
- bussetta 8y agolooking forward to it!
- thezilch 8y agoAny chance we can expect the patches to be applied to the nginx that ships with OpenResty? ;)
- always_good 8y agoI can't help but roll my eyes at these "hello, open source steward of the labor of others, checking in!" posts. Comes off as straight virtue-signaling.
- LinuxBender 8y agoI ask because I have had many challenges with nginx bugs and improvements. Maxim and I have fundamental disagreements about many things. Anything Cloudflare can do to make improvements upstream are appreciated.
- mp3geek 8y agoSubmit them to your own github repo?
- ksec 8y agoI see. I am wondering if you ever had a look at H20, which I think is being used by Fastly.
- jgrahamc 8y agoHere's the upstreamed patch: http://mailman.nginx.org/pipermail/nginx-devel/2018-August/011354.html http://mailman.nginx.org/pipermail/nginx-devel/2018-August/0...
- enitihas 8y agoNon blocking disk I/O is one thing where NT is really ahead of all nix OSes. Unlike say network IO where we have all sorts of platforms(go, node) which allow you to scale by doing async IO, there aren't much options for disk I/O, primarily because of lack of nix options.
- zzzcpan 8y agoIf you mean IOCP API that requires you to hold memory hostage for it, then no, NT sucks at it. The idea behind that API only works well if there is no switch between kernel space and user space. Otherwise you can emulate that awful API with threads on unix-like systems, it will perform the same. Don't get me wrong though, caching/file serving in nginx was always very hacky. And blowing latency by blocking on filesystem and yet relying on vfs cache underneath is just one side effect of that.
- acdha 8y ago> If you mean IOCP API that requires you to hold memory hostage for it, then no, NT sucks at it. “hold memory hostage” really doesn't help make your point. It's inflammatory wording one step above spelling it Micro$oft and it makes it sound like you don't understand the engineering trade-offs which the Microsoft engineers made. It'd be much better if you explained _why_ you disagree with their choice and especially whether there are differences in the type of work which you've used it for which might explain that opinion.
- brian-armstrong 8y agoThe preferred nomenclature is "Micro$haft".
- acdha 8y agoI remember Slashdot in 1997, too, and would like to avoid repeating it. HN is poorer for substance-free jokes.
- kev009 8y agohttps://www.slideshare.net/facepalmtarbz2/new-sendfile-in-english https://www.slideshare.net/facepalmtarbz2/new-sendfile-in-en...
- deleted 8y ago[deleted]
- brian-armstrong 8y agoIf they had written it in Rust this never would have happened
- dang 8y agoMaybe not, but please don't post unsubstantive comments here.
- deathanatos 8y agoMy understanding is that most of Rust's standard library translates to the same blocking calls on Linux, so it would be plagued with the same issues as C would be. (And the solution reached in the article would similarly work in Rust.) There are certainly async I/O libraries for Rust (e.g., Tokio), but those are going to be limited by the primitives the OS gives them. (AFAIK, Tokio's core libraries don't directly do async disk I/O; there is tokio-fs, and it does it by shunting the work to a threadpool.) The fact that disk I/O is so uniquely special on Linux effects any language, as it is an aspect of the kernel itself. I love Rust, and while there are a ton of compelling reasons to use it, I don't think it's fair to say it would have prevented this from happening, in this particular case.
- hjnilsson 8y agoPerhaps, but the first version of Rust appeared 6 years of nginx was launched, so it was never a choice. However, development of a new web server in Rust is an interesting project, and could certainly compete with nginx over time if enough effort was put into it.
- toast0 8y ago> Once in a while, a request gets slowed down enough to matter. My colleague Ivan Babrou ran some I/O benchmarks and saw read spikes of up to 1 second. Moreover, some of our SSDs have more performance outliers than others. If you're running Intel DC 3x10 SSDs, check for a firmware update that improves 'maximum latency' in some cases, the update was released some years ago, but people might not have noticed it.
- dev_dull 8y agoIs there a place for benchmarks where we can discover the “best” SSDs? I find their performance varies wildly.
- windows_tips 8y agohttps://www.tomshardware.com/reviews/best-ssds,3891.html https://www.tomshardware.com/reviews/best-ssds,3891.html
- olavgg 8y agoThese are consumer level SSD's that are very well known by professionals to be slow. They are good for quick bursts of traffic but throttles with heavy use over time. They also have horrible sync write performance https://forums.servethehome.com/index.php?threads/did-some-write-benchmarks-of-a-few-ssds.15231/ https://forums.servethehome.com/index.php?threads/did-some-w... They belong in a desktop/gaming machine not in a production web server. Here is an old review with enterprise drives and the Samsung 840 pro consumer drive which received excellent reviews at the time it was released, notice how it is the worst performer over time. https://www.storagereview.com/micron_m500dc_enterprise_ssd_review https://www.storagereview.com/micron_m500dc_enterprise_ssd_r... The exception is Optane 900p which is blazing fast! A gamechanger!
- toast0 8y agoSSDs aren't as easy to model as spinning drives, but either way, it depends on your use case. Definitely get as much data from others as possible, but very few people will be running the same use cases as you. You can probably figure out the general class of storage you need / can afford, which is interface and bits per cell, but then you probably need to try several well regarded drives (or several dodgy drives, if that's what you can afford) to see how they work for you. Or just go with whatever your hosting provider can supply ;)
- noncoml 8y agoRelevant stackoverflow question, which makes for an interesting read: https://stackoverflow.com/questions/22780822/linux-kernel-aio-open-system-call https://stackoverflow.com/questions/22780822/linux-kernel-ai...
- erikb 8y agoI love how happy they are to work around blocking open(). This is a very common way of thinking. But in fact there are only two ways to handle I/O. And no matter what you do, you always end up with one of them: Path 1, blocking I/O: When you have blocking I/O your process continues to the point where the I/O starts, then sends the corresponding request to the kernel and waits until it gets a respond, potentially forever. This is very low resource usage, but sometimes-hanging-forever is quite a huge price. So usually people put this I/O stuff in a thread/fork and use the parent to have a timeout waiting. Path 2, non-blocking I/O: In this version when the process hits the I/O it will almost-immediately fail when the desired resource (e.g. file, port, whatever) is not available. So usually you are writing a loop and constantly poll for the resource to become ready. This obviously has a rather high cost on resources, because your code gets more complicated (loops, exceptions, etc) and whatever you are I/Oing to has more activity (e.g. if you poll a webserver you constantly create load on that webserver for each client process). But also an advantage is that you can't hang forever, because you usually break the loop after x seconds or y amounts of retries. You might feel it sucks (at least I do) but there are not more options. Decide for one version that you can live with more easily, tune the variables you can fiddle with, like timeouts/retries, and then move on to other problems.
- iopuy 8y agoDoes nginx still force you to recompile the program to get access to the web application firewall? I remember this being a sticking point years ago when evaluating the product.
- merlincorey 8y agoNo, NginX supports dynamic modules now.