10 ms·

Let's Do Some Science: Zed's Test Stack for poll vs. epoll

66 points by gthank 16y ago

wglb 16y ago
Duplicate. See lively discussion at http://news.ycombinator.com/item?id=1570694 http://news.ycombinator.com/item?id=1570694
- jacquesm 16y ago
  Not exactly, this is where Zed presumably shows that using pipes is equivalent to real world testing and has nothing to do with using the loopback/localhost device. Personally I think that's nonsense. I downloaded the code, checked it out, there is nothing in there that changes my mind on the whole thing, the meat is in the C code and it's the same as what was linked from the article. The rest is a wrapper script and some stuff to visualise the results, which are what he showed them to be. The premise that web servers have the majority of their fds active is where the issue lies. FWIW I did some measuring and the results are pretty much in line with what I expected (though the spread is still puzzling, I really intend to find out the cause of that): http://news.ycombinator.com/item?id=1573145 http://news.ycombinator.com/item?id=1573145
  - lol 16y ago
    The massive thing that you're ignoring is that he's trying to make mongrel2 useful as more than a standard web server. Sure, if you ignore that massively obvious piece of information, you have a point. But if you take it into account, you really don't have one at all. As so often happens, you've spewed a lot of words, but said absolutely nothing of value.
  - zedshaw 16y ago
    If the test harness is wrong than every other test that everyone doing epoll comparisons with poll has done is wrong. This is the one they've all been using. So, either you write your own test to evaluate the same thing, without confounding it with other external factors (like this test does), or you use this test and prove me wrong by finding different data. However, I'm glad you finally went and got some metrics to show one webserver instead of the useless rhetoric you were spewing before. Not that it proves a damn thing since it has no external validity, but at least it's a step in the right direction. I also say that your findings of 40% are much higher than what you implied in your previous comments. You made it seem like it's close to the 10% range, but 40% is damn near ready to leap over into the equal performance levels. It's interesting that even after that you still can't see the value of either making epoll faster or using both, but whatever, you can't convince every idiot on the internet, even with Science.
    - jacquesm 16y ago
      You should read more carefully. 40% was the highest observed, 10% the lowest. Also, I said 'the majority', which is not like making it seem like it is only 10%. So it's not '10%' or '40%', for this particular workload, on average you're hovering somewhere between 25 and 30%, since you seem to prefer the higher number make that 30%. And if you read a bit more you would see that there are some factors that could pull that number down considerably in different situations and only one major one (large file serving, streaming) that would pull it up. If I get around to it (I'm on a holiday right now) I'm going to instrument some more sites to measure their active-to-total ratios to see if there is some kind of number (or a set of them) that we might be able to agree on when it comes to real life web traffic. One of them is a filedump, which serves up large media files to clients, that's the use case that I can come up with in which your scheme might pay off, and in which it is possibly advantageous (even if it is a very small advantage) to use poll. You keep 'challenging' people to come up with numbers. So, I did, from a real life webserver serving real customers, lots of them. If 60% is the cutoff point (and we're assuming that you are right about that) then 40% is comfortably under that, not 'much higher' than what I wrote in my previous comments, the majority (60 to 90%) of the fds is idle. So it's not in to 'equal performance levels' yet. But as far as I'm concerned even that is probably moot. (But that needs more testing) because I suspect that even this fairly high traffic server spends only very small amount of time on polling the OS, and a much larger time on shovelling bits in and out of the system. You keep using that word 'science' as though measuring anything at all is automatically science. But what you're doing is more like selecting the data that fits your 'who needs epoll' and 'epoll is bad' hypothesis and rejecting out of hand any experience and / or experimental data that seems to prove you wrong. That's not science. Science is to use all the data available to you, not just the juicy bits. Not just those bits that allow you to make bold claims about discoveries when in fact all you've done is to show that the underlying implementation of one systemcall is different and has different trade-offs than another, which may be specific to the way your system is set up. I agree that to test something you need to reduce it to the minimum of code required to do your tests. But when you're going that route you are doing what so many other database, CPU and application benchmarkers forget. The real world is nothing like a benchmark. You to suggest that this is 'not a localhost test' (http://news.ycombinator.com/item?id=1573195 http://news.ycombinator.com/item?id=1573195) and you say so pretty loud. That would mean that there may be some validity to your data that I have not been able to find to date in the stuff you've provided. I'm challenging you to provide the data that apparently is still missing that proves that this was not 'a localhost test'. And if you don't supply that data then I think you should take back (and try to do it gracriously for a change) that this wasn't a local host test, or you're going to have to come with some definition of what you consider a localhost test to be. For me, that definition is simple, a localhost test is a benchmark that never generates any network traffic. Feel free to disagree.
- lrm242 16y ago
  Not a dupe -- this is his follow-up post.
- wglb 16y ago
  My bad. This is a follow-up article. Apologies.
codexon 16y ago
Zed is probably right that poll is faster than epoll when all descriptors are active. There's no reason to doubt this since epoll has a callback for every file descriptor, while in poll, the OS only has to fill up an array of flags once per call. http://lwn.net/Articles/14168/ http://lwn.net/Articles/14168/ However, I will have to side with jacquesm in saying that regular internet servers will usually benefit from epoll. It is unfathomable that web servers have lots of active file descriptors especially considering the "keep-alive" and that even broadband latency is an eternity for modern cpus to wait. On the other hand though, using poll makes sense for a heavily trafficked file server.
- deleted 16y ago
  [deleted]
- derefr 16y ago
  I believe that, other than in the new-for-the-sake-of-new sysadminning camp, people almost never start by deploying Mongrel. They deploy Apache, because it's easy (every distro comes with an excellent default set of LAMP packages, for obvious reasons.) Thus, Mongrel (or any other non-mindshare-majority server) is only sought out by those who don't like the one they've got—that is, by people who are not getting the performance they desire. Mongrel is probably not used by Joe the VPS user, serving 20 simultaneous connections per hour, because Joe is doing just fine with the Apache install that came with his slice. Mongrel is deployed for high-traffic file servers, as the current weapon of choice. It's very likely that that will continue to be its niche (though it will expand out somewhat as larger shared-hosting providers start to default to it), so it's very likely to mostly see use in high-ATR deployment scenarios. Under that assumption, Zed's experiment is solid. I would still like to see a prologue tacked on about the composition of the Mongrel user base, though, if Zed has it :) Edit: rephrased to... sound less like Zed?
  - zedshaw 16y ago
    Exactly, this is why I was curious about the platitude that epoll wins all the time.
  - yummyfajitas 16y ago
lhorie 16y ago
I'd be interested to see if the idea of switching between epoll and poll at 60% ATR would have too much overhead as to be impractical. Re: > 60% active fd's being real-life or not, I don't really see what's the problem in wanting to handle that case in a project whose goal is basically to keep humming along nicely regardless of what you throw at it.
- pphaneuf 16y ago
  Adding/removing from the poll() array is more or less free, but with epoll, you have to use epoll_ctl(), so you'd definitely want an hysteresis to avoid bouncing things around too much.
  - zedshaw 16y ago
    Interesting, you're like the 3rd or 4th person who's mentioned hysteresis.
    - rbanffy 16y ago
      It's a fairly common idea.
    - pphaneuf 16y ago
      As in "only the 3rd or 4th" (so I'm possibly clever), or as in "a bunch of people"? I don't know your sample size. :-)
tzs 16y ago
The network interface in general on most operating systems seems kind of absurd from a server point of view. You've got data coming in from N clients. The data from all those clients gets physically multiplexed into a single data stream that comes in over the wire. Then in the kernel you've got something that demultiplexes it back into N separate streams, which it accumulates in N different buffers. The server software is aware of all N of these streams. Through some mechanism the kernel notifies the server software that data is available on some subset of the N buffers, and the server software has to ask for that data. If the server software is using a queue and thread model, it likely then gets that data which the kernel has gone to great effort to demultiplex into N streams and provide it through N separate file descriptors--and multiplexes back onto a single queue for the worker threads to pull from! Why not get rid of some of this demultiplexing and re-multiplexing? Let me have a single file descriptor, and everything that comes in destined to my port comes through that single file descriptor, as a message with a header that tells what remote IP address and port it came from and the length, followed by that many bytes of data. Instead of N separate buffers in the kernel, just have one big buffer per port. You'd still have to keep track of how many bytes in that buffer are associated with each remote connection, for purposes of dealing with TCP flow control, but I think this could be made to work reasonably (yes, I'm hand waving right now). Now you don't need poll or epoll. Heck, you don't even need to use select. I'll just have a blocking thread whose job is to just read that one file descriptor using blocking reads in a loop to pull in data and put it on my queue. Looking at this slightly different--everything destined for, say, port 80 is going to the web server, so why does the web server want this to come in through N different file descriptors? Especially when you consider that the code its going to run is the same no matter which it comes in on? All the multiple file descriptors are really accomplishing is providing a way to tell which bytes came from which client. Is that really the best use of file descriptors, as opposed to just having the kernel tell you which client the data came from when you do your read system call? I can see that the N file descriptor model made sense in the days before threads, when to get concurrency you had to use multiple processes. You then need to have data coming in on port X from different clients not necessarily all going to the same process. But for servers that are going to use the "single process with multiple threads" model to handle concurrency, a single stream network model would seem to be a lot nicer and efficient.

66 points by gthank 16y ago

wglb 16y ago

Duplicate. See lively discussion at http://news.ycombinator.com/item?id=1570694 http://news.ycombinator.com/item?id=1570694

jacquesm 16y ago
Not exactly, this is where Zed presumably shows that using pipes is equivalent to real world testing and has nothing to do with using the loopback/localhost device. Personally I think that's nonsense. I downloaded the code, checked it out, there is nothing in there that changes my mind on the whole thing, the meat is in the C code and it's the same as what was linked from the article. The rest is a wrapper script and some stuff to visualise the results, which are what he showed them to be. The premise that web servers have the majority of their fds active is where the issue lies. FWIW I did some measuring and the results are pretty much in line with what I expected (though the spread is still puzzling, I really intend to find out the cause of that): http://news.ycombinator.com/item?id=1573145 http://news.ycombinator.com/item?id=1573145
- lol 16y ago
  The massive thing that you're ignoring is that he's trying to make mongrel2 useful as more than a standard web server. Sure, if you ignore that massively obvious piece of information, you have a point. But if you take it into account, you really don't have one at all. As so often happens, you've spewed a lot of words, but said absolutely nothing of value.
- zedshaw 16y ago
  If the test harness is wrong than every other test that everyone doing epoll comparisons with poll has done is wrong. This is the one they've all been using. So, either you write your own test to evaluate the same thing, without confounding it with other external factors (like this test does), or you use this test and prove me wrong by finding different data. However, I'm glad you finally went and got some metrics to show one webserver instead of the useless rhetoric you were spewing before. Not that it proves a damn thing since it has no external validity, but at least it's a step in the right direction. I also say that your findings of 40% are much higher than what you implied in your previous comments. You made it seem like it's close to the 10% range, but 40% is damn near ready to leap over into the equal performance levels. It's interesting that even after that you still can't see the value of either making epoll faster or using both, but whatever, you can't convince every idiot on the internet, even with Science.
  - jacquesm 16y ago
    You should read more carefully. 40% was the highest observed, 10% the lowest. Also, I said 'the majority', which is not like making it seem like it is only 10%. So it's not '10%' or '40%', for this particular workload, on average you're hovering somewhere between 25 and 30%, since you seem to prefer the higher number make that 30%. And if you read a bit more you would see that there are some factors that could pull that number down considerably in different situations and only one major one (large file serving, streaming) that would pull it up. If I get around to it (I'm on a holiday right now) I'm going to instrument some more sites to measure their active-to-total ratios to see if there is some kind of number (or a set of them) that we might be able to agree on when it comes to real life web traffic. One of them is a filedump, which serves up large media files to clients, that's the use case that I can come up with in which your scheme might pay off, and in which it is possibly advantageous (even if it is a very small advantage) to use poll. You keep 'challenging' people to come up with numbers. So, I did, from a real life webserver serving real customers, lots of them. If 60% is the cutoff point (and we're assuming that you are right about that) then 40% is comfortably under that, not 'much higher' than what I wrote in my previous comments, the majority (60 to 90%) of the fds is idle. So it's not in to 'equal performance levels' yet. But as far as I'm concerned even that is probably moot. (But that needs more testing) because I suspect that even this fairly high traffic server spends only very small amount of time on polling the OS, and a much larger time on shovelling bits in and out of the system. You keep using that word 'science' as though measuring anything at all is automatically science. But what you're doing is more like selecting the data that fits your 'who needs epoll' and 'epoll is bad' hypothesis and rejecting out of hand any experience and / or experimental data that seems to prove you wrong. That's not science. Science is to use all the data available to you, not just the juicy bits. Not just those bits that allow you to make bold claims about discoveries when in fact all you've done is to show that the underlying implementation of one systemcall is different and has different trade-offs than another, which may be specific to the way your system is set up. I agree that to test something you need to reduce it to the minimum of code required to do your tests. But when you're going that route you are doing what so many other database, CPU and application benchmarkers forget. The real world is nothing like a benchmark. You to suggest that this is 'not a localhost test' (http://news.ycombinator.com/item?id=1573195 http://news.ycombinator.com/item?id=1573195) and you say so pretty loud. That would mean that there may be some validity to your data that I have not been able to find to date in the stuff you've provided. I'm challenging you to provide the data that apparently is still missing that proves that this was not 'a localhost test'. And if you don't supply that data then I think you should take back (and try to do it gracriously for a change) that this wasn't a local host test, or you're going to have to come with some definition of what you consider a localhost test to be. For me, that definition is simple, a localhost test is a benchmark that never generates any network traffic. Feel free to disagree.
lrm242 16y ago
Not a dupe -- this is his follow-up post.
wglb 16y ago
My bad. This is a follow-up article. Apologies.

codexon 16y ago

Zed is probably right that poll is faster than epoll when all descriptors are active. There's no reason to doubt this since epoll has a callback for every file descriptor, while in poll, the OS only has to fill up an array of flags once per call. http://lwn.net/Articles/14168/ http://lwn.net/Articles/14168/ However, I will have to side with jacquesm in saying that regular internet servers will usually benefit from epoll. It is unfathomable that web servers have lots of active file descriptors especially considering the "keep-alive" and that even broadband latency is an eternity for modern cpus to wait. On the other hand though, using poll makes sense for a heavily trafficked file server.

deleted 16y ago
[deleted]
derefr 16y ago
I believe that, other than in the new-for-the-sake-of-new sysadminning camp, people almost never start by deploying Mongrel. They deploy Apache, because it's easy (every distro comes with an excellent default set of LAMP packages, for obvious reasons.) Thus, Mongrel (or any other non-mindshare-majority server) is only sought out by those who don't like the one they've got—that is, by people who are not getting the performance they desire. Mongrel is probably not used by Joe the VPS user, serving 20 simultaneous connections per hour, because Joe is doing just fine with the Apache install that came with his slice. Mongrel is deployed for high-traffic file servers, as the current weapon of choice. It's very likely that that will continue to be its niche (though it will expand out somewhat as larger shared-hosting providers start to default to it), so it's very likely to mostly see use in high-ATR deployment scenarios. Under that assumption, Zed's experiment is solid. I would still like to see a prologue tacked on about the composition of the Mongrel user base, though, if Zed has it :) Edit: rephrased to... sound less like Zed?
- zedshaw 16y ago
  Exactly, this is why I was curious about the platitude that epoll wins all the time.
- yummyfajitas 16y ago

lhorie 16y ago

I'd be interested to see if the idea of switching between epoll and poll at 60% ATR would have too much overhead as to be impractical. Re: > 60% active fd's being real-life or not, I don't really see what's the problem in wanting to handle that case in a project whose goal is basically to keep humming along nicely regardless of what you throw at it.

pphaneuf 16y ago
Adding/removing from the poll() array is more or less free, but with epoll, you have to use epoll_ctl(), so you'd definitely want an hysteresis to avoid bouncing things around too much.
- zedshaw 16y ago
  Interesting, you're like the 3rd or 4th person who's mentioned hysteresis.
  - rbanffy 16y ago
    It's a fairly common idea.
  - pphaneuf 16y ago
    As in "only the 3rd or 4th" (so I'm possibly clever), or as in "a bunch of people"? I don't know your sample size. :-)

tzs 16y ago

The network interface in general on most operating systems seems kind of absurd from a server point of view. You've got data coming in from N clients. The data from all those clients gets physically multiplexed into a single data stream that comes in over the wire. Then in the kernel you've got something that demultiplexes it back into N separate streams, which it accumulates in N different buffers. The server software is aware of all N of these streams. Through some mechanism the kernel notifies the server software that data is available on some subset of the N buffers, and the server software has to ask for that data. If the server software is using a queue and thread model, it likely then gets that data which the kernel has gone to great effort to demultiplex into N streams and provide it through N separate file descriptors--and multiplexes back onto a single queue for the worker threads to pull from! Why not get rid of some of this demultiplexing and re-multiplexing? Let me have a single file descriptor, and everything that comes in destined to my port comes through that single file descriptor, as a message with a header that tells what remote IP address and port it came from and the length, followed by that many bytes of data. Instead of N separate buffers in the kernel, just have one big buffer per port. You'd still have to keep track of how many bytes in that buffer are associated with each remote connection, for purposes of dealing with TCP flow control, but I think this could be made to work reasonably (yes, I'm hand waving right now). Now you don't need poll or epoll. Heck, you don't even need to use select. I'll just have a blocking thread whose job is to just read that one file descriptor using blocking reads in a loop to pull in data and put it on my queue. Looking at this slightly different--everything destined for, say, port 80 is going to the web server, so why does the web server want this to come in through N different file descriptors? Especially when you consider that the code its going to run is the same no matter which it comes in on? All the multiple file descriptors are really accomplishing is providing a way to tell which bytes came from which client. Is that really the best use of file descriptors, as opposed to just having the kernel tell you which client the data came from when you do your read system call? I can see that the N file descriptor model made sense in the days before threads, when to get concurrency you had to use multiple processes. You then need to have data coming in on port X from different clients not necessarily all going to the same process. But for servers that are going to use the "single process with multiple threads" model to handle concurrency, a single stream network model would seem to be a lot nicer and efficient.