7 ms·

Dyad: Minimal, portable async networking library for C

97 points by api 12y ago

curiousDog 12y ago
Very neat! Although after using C#'s TAP, callbacks sound like hell :)
- regi 12y ago
  Callbacks can be avoided using co-routines. I started a project that uses asynchronous sockets but they appear synchronous. With a bit of abstraction, it's really easy to develop network applications even in C: https://github.com/reginaldl/librinoo https://github.com/reginaldl/librinoo
  - brown-dragon 12y ago
    Your approach looks very interesting - the examples are definitely much easier to read and understand over most async C code I have seen. How does it work internally?
    - baruch 12y ago
      All of the different coroutine/user-space-threads work by switching the stack and the the instruction pointer when the code is about to block on an async system call. So one would find that to read from the socket would block (by getting EWOULDBLOCK in errno) register to wait for this fd to become readable and switch to another coroutine. There is always one coroutine that gets scheduled from time to time and users select/poll/epoll to get all the fds that got data in them and wake up and schedule the coroutines that are waiting on each of these fds. It's a very neat concept that allows for cleaner code (sequential rather than callback-hell) and use one or a few threads for many actions but without the overhead of thread-per-socket.
      - brown-dragon 12y ago
        Isn't it effectively thread-per-socket except that the switches now only happen on an async system call? The difference, I guess, is you can optimize the context switch to be very small (and have smaller thread stacks) at the cost of giving up pre-emptive switching.
        baruch 12y ago
        It reduces context switches in the kernel which are more expensive and reduces the memory resources needed for kernel threads which are larger. There are also some hidden gains in terms of TLB caches and other costs of kernel threads switching. An additional advantage is that between user-space-threads you have fewer locking problem since they implicitly lock out each other between context switch points so you only need locks when you need to protect an area across several context switch points.
    - regi 12y ago
      It uses "user-space threads" similar to ucontext (swapcontext(3)). I have my own version: https://github.com/reginaldl/fcontext https://github.com/reginaldl/fcontext Once the stack allocated for a thread, context switches are almost as cheap as a function call. RiNOO has an event driven scheduler, based on epoll, which resume/release these user-space threads (that I call tasks) according to pending IOs. The library provides with IO functions (read, write...) which use the RiNOO scheduler. As a bonus, real threading is quite easy: just need to run a scheduler per thread (see examples with multi-threading).
      - brown-dragon 12y ago
        That's really interesting - thanks.
  - baruch 12y ago
    Didn't find this one before. I wrote my own user-space-threading library called libwire. Mostly for not liking the malloc-everywhere approach so common everywhere. The tradeoffs are different and the code is more verbose at times but I like the fact that there are no mallocs in code, at least not very explicit ones. I do provided a memory pool so it allocates memory but it is bounded in size. libwire: https://github.com/baruch/libwire https://github.com/baruch/libwire list of coroutine/user-space-threading libraries: https://github.com/baruch/libwire/wiki/Other-coroutine-libraries https://github.com/baruch/libwire/wiki/Other-coroutine-libra...
NkVczPkybiXICG 12y ago
It uses select, which is notorious for poor performance (although is relatively portable). Try not to use this project for anything high-load.
- regi 12y ago
  And anything that requires more than 1024 fds...
  - baruch 12y ago
    You can increase the define if you need more than 1024 fds but then the performance is getting very poor.
- Chromozon 12y ago
  How are you supposed to do socket programming without select?
  - BruceM 12y ago
    poll, epoll, kqueue, sigio and other options depending on OS, etc.
  - deathanatos 12y ago
    poll is fairly portable, but still has scaling problems. To get beyond scaling problems, you usually need something OS-specific, such as epoll (Linux), kqueue (BSDs of some kind), etc. One of the things I'd expect of a cross-platform networking library would be to use the best available abstraction, especially on major platforms such as linux, and perhaps fall back to select on odd platforms. Of course, your docs need to point out what the behavior is. The problem with poll/select is that you need to pass (transfer) to the kernel the entire list of events you're interested in, just to wait for a single event. Each time you re-enter your event loop you need to do this, and if you have thousands of connections, it will break down, because poll is O(number of connections). epoll, for example, works around this by having an "epoll FD", which on the kernel side contains all the events you're interested in. You wait by just waiting on that epoll fd with epoll_wait, but you don't specify the events you're interested in when you wait: you do that ahead of time, with other system calls. This allows you to change the event list only when you need to, which is much less frequent than some data arrived from somewhere. The API is supposed to be O(1), instead of O(number of connections) per wait call. My understanding is the kqueue works similarly, but I'm a Linux guy, so I can't really tell you. select also has other problems w.r.t. FDs with high numbers.
haberman 12y ago
Recommended additions to the docs / marketing material: "portable" -- portable to what? comparisons -- vs. libevent, libuv, etc. in the sample program -- when will dyad_getStreamCount() drop to zero? will it ever? what exactly is a "stream" anyway and how does it relate to individual connections?
codehero 12y ago
dyad_writef is certainly not 'async' when using the 'r' input specifier (which seems to redirect input from a FILE* to the output stream). And the way dyad_writef achieves 'async' is to buffer the entire contents of the output in RAM. So for 'r', why bother using the FILE* formatted IO if all you're going to do is pull single characters at a time? You can just use file descriptors.
kbaker 12y ago
Here is a libevent echo server for comparison: http://www.wangafu.net/~nickm/libevent-book/Ref8_listener.html#_example_code_an_echo_server http://www.wangafu.net/~nickm/libevent-book/Ref8_listener.ht... It looks like this library has only 2 weeks worth of commits so I'll reserve any criticism. My vote is still with libevent though!
larssonvomdach 12y ago
A friend of mine wrote something similar for C++: https://github.com/Kosta-Github/http-cpp https://github.com/Kosta-Github/http-cpp
shroukkhan 12y ago
does not seem like it supports ssl socket..or did i miss it ?
- eps 12y ago
  There is no such thing as "ssl socket".
floatboth 12y ago
Is it called Dyad for the reason I think it is called Dyad? ;-)
dimman 12y ago
Is it just me that wants to have complete control over stuff? I mean if you really have the need to use a library like libevent you most likely know why and could do your own handling rather easily. I'm not saying there's not a need for libraries like libevent, but IMO it's not needed for most applications. I might be biased because I want complete control and know what happens, I think that's important. I don't want to use a library before I know what happens in the "background". When I know that and know what I need, only then it might be appropriate to use a library.
- matthavener 12y ago
  I'm curious what you mean by "complete control"? Does that mean less abstraction? Or do you want to avoid inversion of control? That is, you want to avoid using callback-based libraries?
- ishbits 12y ago
  For me, libevent just provides the boiler plate but I'd be writing too often for fd registration and timeouts. I could do it myself, but it would just end up looking more and more like libevent each time I did it.

97 points by api 12y ago

curiousDog 12y ago

Very neat! Although after using C#'s TAP, callbacks sound like hell :)

regi 12y ago
Callbacks can be avoided using co-routines. I started a project that uses asynchronous sockets but they appear synchronous. With a bit of abstraction, it's really easy to develop network applications even in C: https://github.com/reginaldl/librinoo https://github.com/reginaldl/librinoo
- brown-dragon 12y ago
  Your approach looks very interesting - the examples are definitely much easier to read and understand over most async C code I have seen. How does it work internally?
  - baruch 12y ago
    All of the different coroutine/user-space-threads work by switching the stack and the the instruction pointer when the code is about to block on an async system call. So one would find that to read from the socket would block (by getting EWOULDBLOCK in errno) register to wait for this fd to become readable and switch to another coroutine. There is always one coroutine that gets scheduled from time to time and users select/poll/epoll to get all the fds that got data in them and wake up and schedule the coroutines that are waiting on each of these fds. It's a very neat concept that allows for cleaner code (sequential rather than callback-hell) and use one or a few threads for many actions but without the overhead of thread-per-socket.
    - brown-dragon 12y ago
      Isn't it effectively thread-per-socket except that the switches now only happen on an async system call? The difference, I guess, is you can optimize the context switch to be very small (and have smaller thread stacks) at the cost of giving up pre-emptive switching.
      - baruch 12y ago
        It reduces context switches in the kernel which are more expensive and reduces the memory resources needed for kernel threads which are larger. There are also some hidden gains in terms of TLB caches and other costs of kernel threads switching. An additional advantage is that between user-space-threads you have fewer locking problem since they implicitly lock out each other between context switch points so you only need locks when you need to protect an area across several context switch points.
  - regi 12y ago
    It uses "user-space threads" similar to ucontext (swapcontext(3)). I have my own version: https://github.com/reginaldl/fcontext https://github.com/reginaldl/fcontext Once the stack allocated for a thread, context switches are almost as cheap as a function call. RiNOO has an event driven scheduler, based on epoll, which resume/release these user-space threads (that I call tasks) according to pending IOs. The library provides with IO functions (read, write...) which use the RiNOO scheduler. As a bonus, real threading is quite easy: just need to run a scheduler per thread (see examples with multi-threading).
    - brown-dragon 12y ago
      That's really interesting - thanks.
- baruch 12y ago
  Didn't find this one before. I wrote my own user-space-threading library called libwire. Mostly for not liking the malloc-everywhere approach so common everywhere. The tradeoffs are different and the code is more verbose at times but I like the fact that there are no mallocs in code, at least not very explicit ones. I do provided a memory pool so it allocates memory but it is bounded in size. libwire: https://github.com/baruch/libwire https://github.com/baruch/libwire list of coroutine/user-space-threading libraries: https://github.com/baruch/libwire/wiki/Other-coroutine-libraries https://github.com/baruch/libwire/wiki/Other-coroutine-libra...

NkVczPkybiXICG 12y ago

It uses select, which is notorious for poor performance (although is relatively portable). Try not to use this project for anything high-load.

regi 12y ago
And anything that requires more than 1024 fds...
- baruch 12y ago
  You can increase the define if you need more than 1024 fds but then the performance is getting very poor.
Chromozon 12y ago
How are you supposed to do socket programming without select?
- BruceM 12y ago
  poll, epoll, kqueue, sigio and other options depending on OS, etc.
- deathanatos 12y ago
  poll is fairly portable, but still has scaling problems. To get beyond scaling problems, you usually need something OS-specific, such as epoll (Linux), kqueue (BSDs of some kind), etc. One of the things I'd expect of a cross-platform networking library would be to use the best available abstraction, especially on major platforms such as linux, and perhaps fall back to select on odd platforms. Of course, your docs need to point out what the behavior is. The problem with poll/select is that you need to pass (transfer) to the kernel the entire list of events you're interested in, just to wait for a single event. Each time you re-enter your event loop you need to do this, and if you have thousands of connections, it will break down, because poll is O(number of connections). epoll, for example, works around this by having an "epoll FD", which on the kernel side contains all the events you're interested in. You wait by just waiting on that epoll fd with epoll_wait, but you don't specify the events you're interested in when you wait: you do that ahead of time, with other system calls. This allows you to change the event list only when you need to, which is much less frequent than some data arrived from somewhere. The API is supposed to be O(1), instead of O(number of connections) per wait call. My understanding is the kqueue works similarly, but I'm a Linux guy, so I can't really tell you. select also has other problems w.r.t. FDs with high numbers.

haberman 12y ago

Recommended additions to the docs / marketing material: "portable" -- portable to what? comparisons -- vs. libevent, libuv, etc. in the sample program -- when will dyad_getStreamCount() drop to zero? will it ever? what exactly is a "stream" anyway and how does it relate to individual connections?

codehero 12y ago

dyad_writef is certainly not 'async' when using the 'r' input specifier (which seems to redirect input from a FILE* to the output stream). And the way dyad_writef achieves 'async' is to buffer the entire contents of the output in RAM. So for 'r', why bother using the FILE* formatted IO if all you're going to do is pull single characters at a time? You can just use file descriptors.

kbaker 12y ago

Here is a libevent echo server for comparison: http://www.wangafu.net/~nickm/libevent-book/Ref8_listener.html#_example_code_an_echo_server http://www.wangafu.net/~nickm/libevent-book/Ref8_listener.ht... It looks like this library has only 2 weeks worth of commits so I'll reserve any criticism. My vote is still with libevent though!

larssonvomdach 12y ago

A friend of mine wrote something similar for C++: https://github.com/Kosta-Github/http-cpp https://github.com/Kosta-Github/http-cpp

shroukkhan 12y ago

does not seem like it supports ssl socket..or did i miss it ?

eps 12y ago
There is no such thing as "ssl socket".

floatboth 12y ago

Is it called Dyad for the reason I think it is called Dyad? ;-)

dimman 12y ago

Is it just me that wants to have complete control over stuff? I mean if you really have the need to use a library like libevent you most likely know why and could do your own handling rather easily. I'm not saying there's not a need for libraries like libevent, but IMO it's not needed for most applications. I might be biased because I want complete control and know what happens, I think that's important. I don't want to use a library before I know what happens in the "background". When I know that and know what I need, only then it might be appropriate to use a library.

matthavener 12y ago
I'm curious what you mean by "complete control"? Does that mean less abstraction? Or do you want to avoid inversion of control? That is, you want to avoid using callback-based libraries?
ishbits 12y ago
For me, libevent just provides the boiler plate but I'd be writing too often for fd registration and timeouts. I could do it myself, but it would just end up looking more and more like libevent each time I did it.