6 ms·

There are other major benefits of kqueue() when using sockets as a client that have essentially "forced" me to use bsd instead of Linux/epoll... The way epoll(

There are other major benefits of kqueue() when using sockets as a client that have essentially "forced" me to use bsd instead of Linux/epoll... The way epoll() is written triggers so many syscalls that things like socket clients can get triggered for every single packet. Whereas kqueue coordinates with other socket options set on the descriptor, such as not returning until x bytes have been received and/or the server has closed the connection. ...not to mention how much more it's tied to across the IO subsystem.

_wmd 8y ago
I might be missing something, but Linux caters to this already: SO_RCVLOWAT and SO_SNDLOWAT Specify the minimum number of bytes in the buffer until the socket layer will pass the data to the protocol (SO_SNDLOWAT) or the user on receiving (SO_RCVLOWAT). These two values are initialized to 1. On the transmit side: EPOLLET Sets the Edge Triggered behavior for the associated file descriptor. The default behavior for epoll is Level Triggered. See epoll(7) for more detailed informa‐ tion about Edge and Level Triggered event distribution architectures.
- benwills 8y ago
  Correct. You can set those socket options and change the triggering behavior of epoll. But those socket options do not actually affect how epoll handles the data when it comes in. They are completely ignored by epoll.
  - _wmd 8y ago
    Not sure how old your experience is, but I tested from Python via epoll a few minutes after posting (never used that option before!) and it worked as described.
    - benwills 8y ago
      My experience is across several years writing high performance http clients, writing in pure C.
      - kaendfinger 8y ago
        This changed recently so both of you are right in different ways.
        benwills 8y ago
        Do you have a link to either an announcement of this change, or C code that demonstrates it? If it has changed, I might start doing some things differently...
        caf 8y ago
        (2008) https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c7004482e8d https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
        benwills 8y ago
        I dealt with all of this in 2015, and spent three months trying to get epoll to perform similarly to kqueue; namely, to avoid returning on any and every packet received and, preferably, to abide by some sort of minimum bytes received (or connection closing) before returning. Unless I'm missing something (and I could very well be missing something), I'm not seeing how that is any different than anything I tried...and I went through every socket option I could find, even if it seemed irrelevant. Have you written code that actually performs this way? Or are you speculating? I really do wish I could do this with epoll, but I (and several others) just never were able to figure it out. And kqueue simply performed at least an order of magnitude better of any variation I could hack together with epoll.
        caf 8y ago
        I had neither written code to test it nor was speculating - I was reading the kernel source, which is pretty clear (if you know in general how file polling is implemented in the Linux kernel). However, because of what you have observed I wrote up a quick test, which confirms to my satisfaction that epoll_wait() respects SO_RCVLOWAT on TCP sockets: https://gist.github.com/keaston/d2473b8b996a34a5860b0744684f2017 https://gist.github.com/keaston/d2473b8b996a34a5860b0744684f... It seems likely that you must have been dealing with an old "enterprise" kernel in 2015 that hadn't had this fix backported to it?
    - benwills 8y ago
      Behind the scenes, Python may have another layer in there to check the socket options, etc, before returning. ie: Just because it looks like it's working on the Python level doesn't mean it's working the same way underneath.
  - deleted 8y ago
    [deleted]
- benwills 8y ago
  It's also worth noting that kqueue() also doesn't use the socket options for this. Instead, it uses a flag called EVFILT_READ. More here: https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2 https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2
cbsmith 8y ago
AIO != epoll()
- benwills 8y ago
  Given that I wasn't asserting that it is, I'm not sure what you're getting at. I was responding to the parent comment.
  - cbsmith 8y ago
    I guess I misunderstood. It sounded like you were limited by the epoll() interface and felt forced to kqueue() and BSD... but you have other alternatives...

_wmd 8y ago

I might be missing something, but Linux caters to this already: SO_RCVLOWAT and SO_SNDLOWAT Specify the minimum number of bytes in the buffer until the socket layer will pass the data to the protocol (SO_SNDLOWAT) or the user on receiving (SO_RCVLOWAT). These two values are initialized to 1. On the transmit side: EPOLLET Sets the Edge Triggered behavior for the associated file descriptor. The default behavior for epoll is Level Triggered. See epoll(7) for more detailed informa‐ tion about Edge and Level Triggered event distribution architectures.

benwills 8y ago
Correct. You can set those socket options and change the triggering behavior of epoll. But those socket options do not actually affect how epoll handles the data when it comes in. They are completely ignored by epoll.
- _wmd 8y ago
  Not sure how old your experience is, but I tested from Python via epoll a few minutes after posting (never used that option before!) and it worked as described.
  - benwills 8y ago
    My experience is across several years writing high performance http clients, writing in pure C.
    - kaendfinger 8y ago
      This changed recently so both of you are right in different ways.
      - benwills 8y ago
        Do you have a link to either an announcement of this change, or C code that demonstrates it? If it has changed, I might start doing some things differently...
        caf 8y ago
        (2008) https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c7004482e8d https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
        benwills 8y ago
        I dealt with all of this in 2015, and spent three months trying to get epoll to perform similarly to kqueue; namely, to avoid returning on any and every packet received and, preferably, to abide by some sort of minimum bytes received (or connection closing) before returning. Unless I'm missing something (and I could very well be missing something), I'm not seeing how that is any different than anything I tried...and I went through every socket option I could find, even if it seemed irrelevant. Have you written code that actually performs this way? Or are you speculating? I really do wish I could do this with epoll, but I (and several others) just never were able to figure it out. And kqueue simply performed at least an order of magnitude better of any variation I could hack together with epoll.
        caf 8y ago
        I had neither written code to test it nor was speculating - I was reading the kernel source, which is pretty clear (if you know in general how file polling is implemented in the Linux kernel). However, because of what you have observed I wrote up a quick test, which confirms to my satisfaction that epoll_wait() respects SO_RCVLOWAT on TCP sockets: https://gist.github.com/keaston/d2473b8b996a34a5860b0744684f2017 https://gist.github.com/keaston/d2473b8b996a34a5860b0744684f... It seems likely that you must have been dealing with an old "enterprise" kernel in 2015 that hadn't had this fix backported to it?
  - benwills 8y ago
    Behind the scenes, Python may have another layer in there to check the socket options, etc, before returning. ie: Just because it looks like it's working on the Python level doesn't mean it's working the same way underneath.
- deleted 8y ago
  [deleted]
benwills 8y ago
It's also worth noting that kqueue() also doesn't use the socket options for this. Instead, it uses a flag called EVFILT_READ. More here: https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2 https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2

cbsmith 8y ago

AIO != epoll()

benwills 8y ago
Given that I wasn't asserting that it is, I'm not sure what you're getting at. I was responding to the parent comment.
- cbsmith 8y ago
  I guess I misunderstood. It sounded like you were limited by the epoll() interface and felt forced to kqueue() and BSD... but you have other alternatives...