6 ms·

LibPhenom, a high-performance C eventing framework from Facebook

107 points by jamesgpearce 13y ago

astrodust 13y ago
Does anyone know how this compares to the standard libevent or the quirky allegedly faster libev (http://software.schmorp.de/pkg/libev.html http://software.schmorp.de/pkg/libev.html)?
- kev009 13y ago
  There's a lot more framework in place for building an application here... hash tables, configuration files, JSON, performance counters. libevent/libev are easier to retrofit into existing applications. This looks like something you'd start a new application with. libuv is somewhere in the middle.
  - petsos 13y ago
    Exactly, I think it is more relevant to compare to e.g. glib.
- wezfurlong 13y ago
  We're a bit faster than libevent in terms of dispatch throughput; some benchmarks in this commit message: https://github.com/facebook/libphenom/commit/41b6106f04fe62cb1d86c9682ded55ab3be980a9 https://github.com/facebook/libphenom/commit/41b6106f04fe62c... The `tests/bench/iopipes.t` "test" allows you to play with some concurrency parameters to try this for yourself on your hardware. We haven't compared against libev. We've added some more APIs (buffers and sockets) since those benchmarks were done and we don't have numbers to share around those yet. One key difference between libevent, libev and libuv is that libphenom is inherently multithreaded in its IO dispatcher and timeout implementation. If you're dispatching purely CPU bound jobs, we get very close to linear scaling with the number of cores: https://github.com/facebook/libphenom/commit/c2753c2154a0cffc0bd70e8f28dd0ea884aab4fd https://github.com/facebook/libphenom/commit/c2753c2154a0cff...
  - mrb 13y ago
    What do you mean by "inherently multithreaded"? I have written a network application handling 100-500k TCP concurrent connections using libev. It was multithreaded to distribute the number of connections evenly per thread (typically from 10k to 100k connections per thread). This is a model that is perfectly supported by libev. And I observed a nice linear scaling of the network throughput with the number of cores since my jobs were also purely CPU-bound. Depending on how CPU intensive my jobs were exactly, my libev code needed anywhere from 2 to 10 threads to saturate a GbE link.
    - wezfurlong 13y ago
      The principal difference between the libphenom event dispatcher and the other event libraries is that libphenom can wakeup and dispatch IO events to any of the IO scheduling threads (no thread affinity). Contrast with the libevent approach of using an application specific scheme to assign descriptors to an event base associated with a thread (strong thread affinity). This makes more of a difference if you have chatty protocols and/or long lived sessions and no way to rebalance your fd -> event_base mapping.
- escaped_hn 13y ago
  what about libuv?
  - otterley 13y ago
    ... and libev?
codehero 13y ago
Why is it every framework handling run loops or communication makes its own implementation of printf? While printf is ubiquitous, I'd hardly call its semantics or syntax perfect. Why does everyone have to copy the same mistakes over and over again?
- zzzcpan 13y ago
  To make it more portable and consistent, something you can rely on. To avoid issues with locales (LC_*) and save some CPU in the meantime. To gain control. It would be wrong not to do that for such tiny amount of code.
  - to3m 13y ago
    To back this up, some specific printf issues I've seen in the past: - printf calls malloc - printf calls FPU emulator - platforms differ over whether %p's output includes a leading 0x - platforms differ over how you print int64_t/uint64_t - platforms differ over how you print size_t - some platforms have almost-ISO-but-not-quite semantics Additionally, on top of the difficulty of adding extra types in a cross-platform fashion, the printf system is tied to the FILE . FILE s are usually not extensible, and on Windows don't work with sockets. This is daft, and surprisingly shortsighted (perhaps it's the word "FILE" that causes people to come over all unimaginative?), because you could provide some system like Mac OS X's funopen, and then use fprintf for everything - maybe even replacing snprintf with it! - but functionality like this isn't as widely available as it should be. Anyway, if you write your own printf, you can fix all of this.
    - wezfurlong 13y ago
      All of these are factors in our choice for printf here. Another fun one: FILE on Solaris can only be used with file descriptors whose value fits in 8 bits due to an astonishing degree of backwards ABI compatibility. Also on Solaris, printf("%s", NULL) -> crash but on other systems will print "(null)". In our implementation we couldn't solve the frustrating size_t uint64_t stuff without disabling the compile time parameter checking that gcc provides; I value that more than the slight annoyance of PRIu64.
kev009 13y ago
Nice to see Concurrency Kit here. I wonder if they should have extended/patched libuv though.
- marshray 13y ago
  I hadn't heard of CK (http://concurrencykit.org/ http://concurrencykit.org/) before. Do you know of anything else that's using it?
peterwwillis 13y ago
Ooh. Maybe they can patch POE::XS to use this as a backend. The existing POE::XS code is 9 years old, and I can only assume probably doesn't work after that long without updates.
IPGlider 13y ago
How does this compare to libdispatch (https://libdispatch.macosforge.org/ https://libdispatch.macosforge.org/) in functionality and performance? I'm very interested in this.
- wezfurlong 13y ago
  We haven't looked at libdispatch specifically; would welcome your feedback on how we compare. We hope we do well here; we've had some nice results: https://github.com/facebook/libphenom/commit/c2753c2154a0cffc0bd70e8f28dd0ea884aab4fd https://github.com/facebook/libphenom/commit/c2753c2154a0cff...
  - wezfurlong 13y ago
    This is the test code referenced by that commit: https://github.com/facebook/libphenom/blob/master/tests/tpool.c https://github.com/facebook/libphenom/blob/master/tests/tpoo...
  - IPGlider 13y ago
    That looks nice. I used libdispatch to prototype a server and would be great to compare with other libraries before starting the project. libPhenom seems to have much more features, maybe even more than I need, but the documentation seems good. I will try to make another prototype with your lib. Thanks.
  - nwmcsween 13y ago
    Why do you do -fno-omit-frame-pointer on x86_64 the ABI requires dwarf or is there something I'm missing?
    - wezfurlong 13y ago
      Frame pointers make things easier for tools to get backtraces without requiring complex dwarf unwinders. The observability is something I value more than not being able to use that register for other things.
- throwaway812 13y ago
capkutay 13y ago
Can someone explain what eventing frameworks are used for? Are they related to event messaging or processing frameworks?
- wezfurlong 13y ago
  At a high level, they make it easier to write server (or client) software that deals with multiple concurrent connections (such as HTTP server software, or just about any network facing service these days). They do this by abstracting some of the details away so that you can focus on writing your application code; instead you declare callbacks that get invoked when you have data available. Traditional eventing frameworks focused on non-blocking I/O on a single thread on the basis that you don't need so many resources to scale up to a large number of clients when compared to a simple one-thread-per-client model. There's lot of good material discussing this at http://www.kegel.com/c10k.html http://www.kegel.com/c10k.html libphenom is a bit more than just an eventing framework though; we have a number of APIs that help with putting together the whole application. And we blur the lines a bit: we also have thread pooling and dispatch support for cases where your can't build 100% of your application in a non-blocking fashion.
iatrou 13y ago
Looks very promising! It would really help the adoption of such new projects if there was a bit more effort to present the current status as well as a roadmap. libphenom strives to be a core component for server applications. It also has to compete with similar Open Source projects that have been around for a while, with well known strengths and limitations. A more clear statement of its current status (is it used in production for Facebook? is it in beta? are there known caveats?) as well as the future goals would help to build confidence in the project.
- wezfurlong 13y ago
  Thanks; good feedback. It's not currently in production at Facebook, but like just about everything we do, we're quickly iterating. Regarding roadmap, we'll try to keep the issue tracker on Github sync'd up with the broad goals and milestones, and we welcome feedback there too; they're a bit more dynamic and easier to update in real time than the docs and website materials.

107 points by jamesgpearce 13y ago

Does anyone know how this compares to the standard libevent or the quirky allegedly faster libev (http://software.schmorp.de/pkg/libev.html http://software.schmorp.de/pkg/libev.html)?

kev009 13y ago
There's a lot more framework in place for building an application here... hash tables, configuration files, JSON, performance counters. libevent/libev are easier to retrofit into existing applications. This looks like something you'd start a new application with. libuv is somewhere in the middle.
- petsos 13y ago
  Exactly, I think it is more relevant to compare to e.g. glib.
wezfurlong 13y ago
We're a bit faster than libevent in terms of dispatch throughput; some benchmarks in this commit message: https://github.com/facebook/libphenom/commit/41b6106f04fe62cb1d86c9682ded55ab3be980a9 https://github.com/facebook/libphenom/commit/41b6106f04fe62c... The `tests/bench/iopipes.t` "test" allows you to play with some concurrency parameters to try this for yourself on your hardware. We haven't compared against libev. We've added some more APIs (buffers and sockets) since those benchmarks were done and we don't have numbers to share around those yet. One key difference between libevent, libev and libuv is that libphenom is inherently multithreaded in its IO dispatcher and timeout implementation. If you're dispatching purely CPU bound jobs, we get very close to linear scaling with the number of cores: https://github.com/facebook/libphenom/commit/c2753c2154a0cffc0bd70e8f28dd0ea884aab4fd https://github.com/facebook/libphenom/commit/c2753c2154a0cff...
- mrb 13y ago
  What do you mean by "inherently multithreaded"? I have written a network application handling 100-500k TCP concurrent connections using libev. It was multithreaded to distribute the number of connections evenly per thread (typically from 10k to 100k connections per thread). This is a model that is perfectly supported by libev. And I observed a nice linear scaling of the network throughput with the number of cores since my jobs were also purely CPU-bound. Depending on how CPU intensive my jobs were exactly, my libev code needed anywhere from 2 to 10 threads to saturate a GbE link.
  - wezfurlong 13y ago
    The principal difference between the libphenom event dispatcher and the other event libraries is that libphenom can wakeup and dispatch IO events to any of the IO scheduling threads (no thread affinity). Contrast with the libevent approach of using an application specific scheme to assign descriptors to an event base associated with a thread (strong thread affinity). This makes more of a difference if you have chatty protocols and/or long lived sessions and no way to rebalance your fd -> event_base mapping.
escaped_hn 13y ago
what about libuv?
- otterley 13y ago
  ... and libev?

codehero 13y ago

Why is it every framework handling run loops or communication makes its own implementation of printf? While printf is ubiquitous, I'd hardly call its semantics or syntax perfect. Why does everyone have to copy the same mistakes over and over again?

zzzcpan 13y ago
To make it more portable and consistent, something you can rely on. To avoid issues with locales (LC_*) and save some CPU in the meantime. To gain control. It would be wrong not to do that for such tiny amount of code.
- to3m 13y ago
  To back this up, some specific printf issues I've seen in the past: - printf calls malloc - printf calls FPU emulator - platforms differ over whether %p's output includes a leading 0x - platforms differ over how you print int64_t/uint64_t - platforms differ over how you print size_t - some platforms have almost-ISO-but-not-quite semantics Additionally, on top of the difficulty of adding extra types in a cross-platform fashion, the printf system is tied to the FILE . FILE s are usually not extensible, and on Windows don't work with sockets. This is daft, and surprisingly shortsighted (perhaps it's the word "FILE" that causes people to come over all unimaginative?), because you could provide some system like Mac OS X's funopen, and then use fprintf for everything - maybe even replacing snprintf with it! - but functionality like this isn't as widely available as it should be. Anyway, if you write your own printf, you can fix all of this.
  - wezfurlong 13y ago
    All of these are factors in our choice for printf here. Another fun one: FILE on Solaris can only be used with file descriptors whose value fits in 8 bits due to an astonishing degree of backwards ABI compatibility. Also on Solaris, printf("%s", NULL) -> crash but on other systems will print "(null)". In our implementation we couldn't solve the frustrating size_t uint64_t stuff without disabling the compile time parameter checking that gcc provides; I value that more than the slight annoyance of PRIu64.

kev009 13y ago

Nice to see Concurrency Kit here. I wonder if they should have extended/patched libuv though.

marshray 13y ago
I hadn't heard of CK (http://concurrencykit.org/ http://concurrencykit.org/) before. Do you know of anything else that's using it?

peterwwillis 13y ago

Ooh. Maybe they can patch POE::XS to use this as a backend. The existing POE::XS code is 9 years old, and I can only assume probably doesn't work after that long without updates.

IPGlider 13y ago

How does this compare to libdispatch (https://libdispatch.macosforge.org/ https://libdispatch.macosforge.org/) in functionality and performance? I'm very interested in this.

wezfurlong 13y ago
We haven't looked at libdispatch specifically; would welcome your feedback on how we compare. We hope we do well here; we've had some nice results: https://github.com/facebook/libphenom/commit/c2753c2154a0cffc0bd70e8f28dd0ea884aab4fd https://github.com/facebook/libphenom/commit/c2753c2154a0cff...
- wezfurlong 13y ago
  This is the test code referenced by that commit: https://github.com/facebook/libphenom/blob/master/tests/tpool.c https://github.com/facebook/libphenom/blob/master/tests/tpoo...
- IPGlider 13y ago
  That looks nice. I used libdispatch to prototype a server and would be great to compare with other libraries before starting the project. libPhenom seems to have much more features, maybe even more than I need, but the documentation seems good. I will try to make another prototype with your lib. Thanks.
- nwmcsween 13y ago
  Why do you do -fno-omit-frame-pointer on x86_64 the ABI requires dwarf or is there something I'm missing?
  - wezfurlong 13y ago
    Frame pointers make things easier for tools to get backtraces without requiring complex dwarf unwinders. The observability is something I value more than not being able to use that register for other things.
throwaway812 13y ago

capkutay 13y ago

Can someone explain what eventing frameworks are used for? Are they related to event messaging or processing frameworks?

wezfurlong 13y ago
At a high level, they make it easier to write server (or client) software that deals with multiple concurrent connections (such as HTTP server software, or just about any network facing service these days). They do this by abstracting some of the details away so that you can focus on writing your application code; instead you declare callbacks that get invoked when you have data available. Traditional eventing frameworks focused on non-blocking I/O on a single thread on the basis that you don't need so many resources to scale up to a large number of clients when compared to a simple one-thread-per-client model. There's lot of good material discussing this at http://www.kegel.com/c10k.html http://www.kegel.com/c10k.html libphenom is a bit more than just an eventing framework though; we have a number of APIs that help with putting together the whole application. And we blur the lines a bit: we also have thread pooling and dispatch support for cases where your can't build 100% of your application in a non-blocking fashion.

iatrou 13y ago

Looks very promising! It would really help the adoption of such new projects if there was a bit more effort to present the current status as well as a roadmap. libphenom strives to be a core component for server applications. It also has to compete with similar Open Source projects that have been around for a while, with well known strengths and limitations. A more clear statement of its current status (is it used in production for Facebook? is it in beta? are there known caveats?) as well as the future goals would help to build confidence in the project.

wezfurlong 13y ago
Thanks; good feedback. It's not currently in production at Facebook, but like just about everything we do, we're quickly iterating. Regarding roadmap, we'll try to keep the issue tracker on Github sync'd up with the broad goals and milestones, and we welcome feedback there too; they're a bit more dynamic and easier to update in real time than the docs and website materials.