8 ms·

Wangle – an asynchronous C++ networking and RPC library

164 points by doppp 11y ago

x5n1 11y ago
In 2016 I would write networking and RPC in go.
- swah 11y ago
  Me too - and I don't even know C++.
  - brandeis 11y ago
    Go is a toxic language, I wouldn't necessarily use C++, but jumping on the go bandwagon is going to get you burned.
    - tptacek 11y ago
      Articulating concerns with Go is a legitimate (if, for this particular thread, unproductive) way to use HN. Writing comments like "Go is a toxic language" is not; it's morally indistinguishable from trolling. The comment rooting this thread, suggesting the use of Go instead of C++, was also unproductive. I'm glad to be done with the C++ part of my career, but still happy to read about ambitious new libraries built in it. That upthread Go comment is the kind of tangent-driving chaff that made Slashdot unreadable. But this comment is even worse.
      - x5n1 11y ago
        Point taken. Made comment less polarizing.
  - swah 11y ago
    I was joking that the main reason for dismissing C++ for this task so quickly was not knowing it, and when and where C++ is used. How did I came through, to be so unvoted? I'm confused.
- tveita 11y ago
  "networking and rpc" must be written in the languages your server and client are written in, (or something compatible). Writing a networking library in Go does nothing for Facebook if their services are written in C++.
- thrownaway2424 11y ago
  Because you want to spend 10x as many CPUs?
  - x5n1 11y ago
    Mostly because it's easier, performant, and because of the nature of the language it's less likely I would make a mistake doing anything in the language compared to C++ or C.
  - deleted 11y ago
    [deleted]
educar 11y ago
Had to look up the meaning of wangle. "obtain (something that is desired) by persuading others to comply or by manipulating events." Does anyone know where facebook is coming up with all these words - folly/wangle?
- riyadparvez 11y ago
  Folly is a common word, I think. As non-native English speaker I know the meaning of folly. But I saw wangle for the first time.
  - educar 11y ago
    Yup, I know folly as well (non-native speaker as well). But it doesn't make sense to name a library 'folly' does it? What I was wondering is if this some self-deprecating humor or these words are popular phrases from some popular book/meme/play.
- DougN7 11y ago
  I was going to correct you and say you meant "wrangle". Turns out they can have the same meaning! Wonder if there are other examples of two words being one letter off that mean the same thing.
- cgh 11y ago
  "Folly" is a relatively common English word. "Wangle" is less common but I still see it from time to time. What I find more interesting than the naming is FB's willingness to remove features from Folly as they are made redundant by Boost or std. I'm unaccustomed to such a lack of arrogance from a major player.
  - educar 11y ago
    Doesn't removing features break backward compatibility? My favorite C++ library is Qt; they never remove public API for that reason. In Qt world, we update regularly to get the bug fixes and other features. Last thing I want to do is port my code to some other new API to update the library.
je42 11y ago
does anybody know why this library is not based on (boost::)asio ?
- vvanders 11y ago
  I'll take a shot in the dark and say they probably value their compile times. Every boost library I've ever used has the tendency to pull in the world from a single header and take forever to compile.
  - lhecker 11y ago
    ASIO can be used standalone with any C++11 compiler and the compile times are quite good. If I should make a guess then I'd say that the original author of that part of the library either started before ASIO was standalone and/or simply has a hatred for boost (which would be unwarranted), or he/she was simply more experienced with libevent... IMHO it's quite sad that they didn't use ASIO though as it's far more extensive.
    - vvanders 11y ago
      That's the thing about reputation though, really easy to burn and really hard to earn back. I've worked on projects where touching a main header incurred a 1 hour compile even with Incredibuild. You can understand how people can get sensitive with compile times in cases like that.
    - niekb 11y ago
      And interestingly, Chris Kohlhoff (asio author) is currently working on an executors library / c++ standard proposal. Other question: how would wangle compare to Cap'n Proto (as the post mentions that wangle is an RPC framework, and is zero-copy)
      - dunkelheit 11y ago
        Zero-copy in capnproto docs and in this blog post actually mean different things (the first means that serialized and in-memory representations of capnproto messages are the same and the second means the absence of copying between user space and kernel space when sending a file over the network).
thepumpkin1979 11y ago
I just tried to compile folly in OSX and it fails to compile with homebrew. https://github.com/facebook/folly/issues/400 https://github.com/facebook/folly/issues/400 Also, Once I manage to compile folly and then wangle, what kind of database drivers can I use in conjunction with this? Would I shoot myself in the foot if I use this with libpq under CPUThreadPoolExecutor? Same with Rust and dlang, I'm not sure how to apply the freedom to have pluggable executor into something useful in real life, I think this is the main reason Golang is killing it and it keeps getting better. For now I have my money on Go but can't wait for http://www.ponylang.org/ http://www.ponylang.org/ to become stable. Edit: Issue reported has been closed, I can now proceed to install wangle https://github.com/facebook/folly/issues/400 https://github.com/facebook/folly/issues/400 Edit 2: Found another issue, this time installing wangle https://github.com/facebook/wangle/issues/37 https://github.com/facebook/wangle/issues/37
- pcwalton 11y ago
  Go uses synchronous I/O backed with an M:N implementation. Rust and D use synchronous I/O backed with a 1:1 implementation. Semantically, there is no difference between the two. The difference is in the implementation, and 1:1 threading is not as slow as you think on Linux.
  - tptacek 11y ago
    Why are we talking about Go, Rust, and D on a thread about a new C++ library? I mean, I know the answer, but I wish we weren't. I ordinarily wouldn't care, but I usually feel like I have something to learn from ambitious networking libraries even when they're written in languages I prefer not to use.
    - pcwalton 11y ago
      For what it's worth, I try to not bring up Rust in threads about other languages; it just happens that most threads about Rust appear in threads about other languages. I would rather talk about the implementation of async I/O too.
    - 11y ago
euyyn 11y ago
I like that they explained the reasons behind the design decisions of the library. Looking forward to the post about the RPC part!
jhartmann 11y ago
I have used this library pretty heavily in some of my recent work and I love how easy it makes building performant async network servers in C++. Minor nitpick: It would be nice though, if they built shared libraries by default though, last release I took I had to tell cmake to make a shared library. Since this is used by fbthrift as a dependency, I think it really makes sense to build it that way by default. I also use fblualib and it builds with a very old version of some of these libraries probably because the FAIR team didn't want to mess with tweaking the builds so they can work with standalone wangle (parts of the library used to be part of folly).
soconfused1 11y ago
would this be useful for game development?
- jfasi 11y ago
  I can't speak for Wangle, but I've used Google's version (GRPC) to implement the networking transport for an N64 netplay hack[1] I wrote, and it was awesome. [1]: https://github.com/water-works https://github.com/water-works
averageJoeCoder 11y ago
I'm wondering, how does C++ fare against Nodejs now, if this library is implemented for creating servers?
- ec109685 11y ago
  Node JS unfortunately doesn't have the notion of separate IO and CPU threads.
rburhum 11y ago
After most compilers started properly supporting the newer C++ standards, it seems there has been an increase in the amount of love that C++ libraries are getting. A lot of people like the to say that [x] is superior to C or C++ - often ignoring that their favorite language's libraries/runtime/compilers/vm is most likely implemented in the very same languages they are putting down.
- derefr 11y ago
  > often ignoring that their favorite language's libraries/runtime/compilers/vm is most likely implemented in the very same languages they are putting down I don't see how that's an argument against a language being "superior to C or C++". You can't create a self-hosting compiler for your language without having already created a non-self-hosting compiler for your language. That means every language is going to have to be "bootstrapped" in terms of something else first. C is the usual candidate just because there are a plethora of useful compiler/runtime/vm-ish libraries people like to rely on for "language prototyping" that expose C bindings—and there are very few other languages than C/C++ that have zero-overhead, zero-friction C binding support. Rust is one of those languages—and it does indeed look like things are going in the direction of new languages getting prototyped/bootstrapped with a Rust compiler/runtime/vm instead.
  - santaclaus 11y ago
    > Rust is one of those languages—and it does indeed look like things are going in the direction of new languages getting prototyped/bootstrapped with a Rust compiler/runtime/vm instead. I'm curious what examples you have in mind here? LLVM seems to be the default go-to compiler infrastructure, at the moment.
- pjmlp 11y ago
  > After most compilers started properly supporting the newer C++ standards, it seems there has been an increase in the amount of love that C++ libraries are getting. Sadly there are many more compilers out there than just gcc, clang, icc and msvc++. Many of those aren't still C++11 compliant, specially in mainframes, classical commercial UNIXes, embedded real time OS and certified compilers. But it is true, thanks to the productivity of latest standards, there has been an increase in C++ visibility, specially thanks to Microsoft. People like to bash their C++'s compliance, but they are the commercial vendor with the more up to date support, even than icc last time I checked. Also they were the main sponsors for Back to native conferences and eventually making it together with CppCon. https://blogs.msdn.microsoft.com/vcblog/2014/04/03/cppcon-the-c-conference/ https://blogs.msdn.microsoft.com/vcblog/2014/04/03/cppcon-th... Also Windows Phone is the only mobile OS where C++ is a first class language. >. A lot of people like the to say that [x] is superior to C or C++ - often ignoring that their favorite language's libraries/runtime/compilers/vm is most likely implemented in the very same languages they are putting down. A lot of people like to point out that fact, ignoring that many compiler writers tend to use C or C++ out of convenience for existing tools instead of bootraping the full eco-system around a new programming language. Many use the same route as Go, using one of those languages and when the toolchain gets mature, those parts get re-writen. All in all, I do like the language, now if we could get rid of those C underpinnings...
vardump 11y ago
So Wangle is (mostly) a protocol agnostic platform for building asynchronous network clients and services in C++. What transport layers are supported and can it be extended? What I can gather, it only supports IPv4/IPv6 TCP or UDP transports. TLS is also supported. Notably, local pipes seem to be unsupported. Those can be sometimes useful for performance and security/isolation reasons.
jfasi 11y ago
The first thing that comes to mind after reading this is GRPC (https://github.com/grpc/grpc https://github.com/grpc/grpc). That library has the advantage of supporting just about every language, not just C++.
eva1984 11y ago
Good that they mentioned Netty as inspiration
deleted 11y ago
[deleted]
chetanahuja 11y ago
"Wangle is heavily influenced by the JVM-based libraries Netty," Stopped reading there. Have experience with using Netty in a java server application and engineers dread being assigned a bug or enhancement in that codebase. There's a tendency throw around the word "over-engineered" lightly but netty is a real, live and inexplicably popular example of exactly what that word means.
ausjke 11y ago
Good to see C++ is gaining more visibilities these days, considering the Oracle/Java combination I hope C++ can take away some JAVA dominance, at least nobody is going to sue you over C++ APIs. Failed to compile with CMake 2.8 CMake Error at /usr/share/cmake-2.8/Modules FindPackageHandleStandardArgs.cmake:91 (MESSAGE): REQUIRED_ARGS (missing: FOLLY_INCLUDE_DIR OLLY_LIBRARIES) Call Stack (most recent call first): /usr/share/cmake-2.8/Module/FindPackageHandleStandardArgs.cmake:252(_FPHSA_FAILURE_MESSAGE) cmake/FindFolly.cmake:23 (FIND_PACKAGE_HANDLE_STANDARD_ARGS) CMakeLists.txt:17 (find_package)
sametmax 11y ago
Also, check out crossbar.io and the WAMP protocol, offering transparent, clean and yet simple PUB/SUB & RPC accross languages, including C++, Python, PHP and JS (node and yes, browsers).
- Matthias247 11y ago
  WAMP is a protocol, Wangle is a framework that you can use to implement protocols (like WAMP), crossbar is a broker (an application that implements the protocol using a [python] framework. So it's not a direct comparison.

164 points by doppp 11y ago

x5n1 11y ago

In 2016 I would write networking and RPC in go.

swah 11y ago
Me too - and I don't even know C++.
- brandeis 11y ago
  Go is a toxic language, I wouldn't necessarily use C++, but jumping on the go bandwagon is going to get you burned.
  - tptacek 11y ago
    Articulating concerns with Go is a legitimate (if, for this particular thread, unproductive) way to use HN. Writing comments like "Go is a toxic language" is not; it's morally indistinguishable from trolling. The comment rooting this thread, suggesting the use of Go instead of C++, was also unproductive. I'm glad to be done with the C++ part of my career, but still happy to read about ambitious new libraries built in it. That upthread Go comment is the kind of tangent-driving chaff that made Slashdot unreadable. But this comment is even worse.
    - x5n1 11y ago
      Point taken. Made comment less polarizing.
- swah 11y ago
  I was joking that the main reason for dismissing C++ for this task so quickly was not knowing it, and when and where C++ is used. How did I came through, to be so unvoted? I'm confused.
tveita 11y ago
"networking and rpc" must be written in the languages your server and client are written in, (or something compatible). Writing a networking library in Go does nothing for Facebook if their services are written in C++.
thrownaway2424 11y ago
Because you want to spend 10x as many CPUs?
- x5n1 11y ago
  Mostly because it's easier, performant, and because of the nature of the language it's less likely I would make a mistake doing anything in the language compared to C++ or C.
- deleted 11y ago
  [deleted]

educar 11y ago

Had to look up the meaning of wangle. "obtain (something that is desired) by persuading others to comply or by manipulating events." Does anyone know where facebook is coming up with all these words - folly/wangle?

riyadparvez 11y ago
Folly is a common word, I think. As non-native English speaker I know the meaning of folly. But I saw wangle for the first time.
- educar 11y ago
  Yup, I know folly as well (non-native speaker as well). But it doesn't make sense to name a library 'folly' does it? What I was wondering is if this some self-deprecating humor or these words are popular phrases from some popular book/meme/play.
DougN7 11y ago
I was going to correct you and say you meant "wrangle". Turns out they can have the same meaning! Wonder if there are other examples of two words being one letter off that mean the same thing.
cgh 11y ago
"Folly" is a relatively common English word. "Wangle" is less common but I still see it from time to time. What I find more interesting than the naming is FB's willingness to remove features from Folly as they are made redundant by Boost or std. I'm unaccustomed to such a lack of arrogance from a major player.
- educar 11y ago
  Doesn't removing features break backward compatibility? My favorite C++ library is Qt; they never remove public API for that reason. In Qt world, we update regularly to get the bug fixes and other features. Last thing I want to do is port my code to some other new API to update the library.

je42 11y ago

does anybody know why this library is not based on (boost::)asio ?

vvanders 11y ago
I'll take a shot in the dark and say they probably value their compile times. Every boost library I've ever used has the tendency to pull in the world from a single header and take forever to compile.
- lhecker 11y ago
  ASIO can be used standalone with any C++11 compiler and the compile times are quite good. If I should make a guess then I'd say that the original author of that part of the library either started before ASIO was standalone and/or simply has a hatred for boost (which would be unwarranted), or he/she was simply more experienced with libevent... IMHO it's quite sad that they didn't use ASIO though as it's far more extensive.
  - vvanders 11y ago
    That's the thing about reputation though, really easy to burn and really hard to earn back. I've worked on projects where touching a main header incurred a 1 hour compile even with Incredibuild. You can understand how people can get sensitive with compile times in cases like that.
  - niekb 11y ago
    And interestingly, Chris Kohlhoff (asio author) is currently working on an executors library / c++ standard proposal. Other question: how would wangle compare to Cap'n Proto (as the post mentions that wangle is an RPC framework, and is zero-copy)
    - dunkelheit 11y ago
      Zero-copy in capnproto docs and in this blog post actually mean different things (the first means that serialized and in-memory representations of capnproto messages are the same and the second means the absence of copying between user space and kernel space when sending a file over the network).

thepumpkin1979 11y ago

I just tried to compile folly in OSX and it fails to compile with homebrew. https://github.com/facebook/folly/issues/400 https://github.com/facebook/folly/issues/400 Also, Once I manage to compile folly and then wangle, what kind of database drivers can I use in conjunction with this? Would I shoot myself in the foot if I use this with libpq under CPUThreadPoolExecutor? Same with Rust and dlang, I'm not sure how to apply the freedom to have pluggable executor into something useful in real life, I think this is the main reason Golang is killing it and it keeps getting better. For now I have my money on Go but can't wait for http://www.ponylang.org/ http://www.ponylang.org/ to become stable. Edit: Issue reported has been closed, I can now proceed to install wangle https://github.com/facebook/folly/issues/400 https://github.com/facebook/folly/issues/400 Edit 2: Found another issue, this time installing wangle https://github.com/facebook/wangle/issues/37 https://github.com/facebook/wangle/issues/37

pcwalton 11y ago
Go uses synchronous I/O backed with an M:N implementation. Rust and D use synchronous I/O backed with a 1:1 implementation. Semantically, there is no difference between the two. The difference is in the implementation, and 1:1 threading is not as slow as you think on Linux.
- tptacek 11y ago
  Why are we talking about Go, Rust, and D on a thread about a new C++ library? I mean, I know the answer, but I wish we weren't. I ordinarily wouldn't care, but I usually feel like I have something to learn from ambitious networking libraries even when they're written in languages I prefer not to use.
  - pcwalton 11y ago
    For what it's worth, I try to not bring up Rust in threads about other languages; it just happens that most threads about Rust appear in threads about other languages. I would rather talk about the implementation of async I/O too.
  - 11y ago

euyyn 11y ago

I like that they explained the reasons behind the design decisions of the library. Looking forward to the post about the RPC part!

jhartmann 11y ago

I have used this library pretty heavily in some of my recent work and I love how easy it makes building performant async network servers in C++. Minor nitpick: It would be nice though, if they built shared libraries by default though, last release I took I had to tell cmake to make a shared library. Since this is used by fbthrift as a dependency, I think it really makes sense to build it that way by default. I also use fblualib and it builds with a very old version of some of these libraries probably because the FAIR team didn't want to mess with tweaking the builds so they can work with standalone wangle (parts of the library used to be part of folly).

soconfused1 11y ago

would this be useful for game development?

jfasi 11y ago
I can't speak for Wangle, but I've used Google's version (GRPC) to implement the networking transport for an N64 netplay hack[1] I wrote, and it was awesome. [1]: https://github.com/water-works https://github.com/water-works

averageJoeCoder 11y ago

I'm wondering, how does C++ fare against Nodejs now, if this library is implemented for creating servers?

ec109685 11y ago
Node JS unfortunately doesn't have the notion of separate IO and CPU threads.

rburhum 11y ago

After most compilers started properly supporting the newer C++ standards, it seems there has been an increase in the amount of love that C++ libraries are getting. A lot of people like the to say that [x] is superior to C or C++ - often ignoring that their favorite language's libraries/runtime/compilers/vm is most likely implemented in the very same languages they are putting down.

derefr 11y ago
> often ignoring that their favorite language's libraries/runtime/compilers/vm is most likely implemented in the very same languages they are putting down I don't see how that's an argument against a language being "superior to C or C++". You can't create a self-hosting compiler for your language without having already created a non-self-hosting compiler for your language. That means every language is going to have to be "bootstrapped" in terms of something else first. C is the usual candidate just because there are a plethora of useful compiler/runtime/vm-ish libraries people like to rely on for "language prototyping" that expose C bindings—and there are very few other languages than C/C++ that have zero-overhead, zero-friction C binding support. Rust is one of those languages—and it does indeed look like things are going in the direction of new languages getting prototyped/bootstrapped with a Rust compiler/runtime/vm instead.
- santaclaus 11y ago
  > Rust is one of those languages—and it does indeed look like things are going in the direction of new languages getting prototyped/bootstrapped with a Rust compiler/runtime/vm instead. I'm curious what examples you have in mind here? LLVM seems to be the default go-to compiler infrastructure, at the moment.
pjmlp 11y ago
> After most compilers started properly supporting the newer C++ standards, it seems there has been an increase in the amount of love that C++ libraries are getting. Sadly there are many more compilers out there than just gcc, clang, icc and msvc++. Many of those aren't still C++11 compliant, specially in mainframes, classical commercial UNIXes, embedded real time OS and certified compilers. But it is true, thanks to the productivity of latest standards, there has been an increase in C++ visibility, specially thanks to Microsoft. People like to bash their C++'s compliance, but they are the commercial vendor with the more up to date support, even than icc last time I checked. Also they were the main sponsors for Back to native conferences and eventually making it together with CppCon. https://blogs.msdn.microsoft.com/vcblog/2014/04/03/cppcon-the-c-conference/ https://blogs.msdn.microsoft.com/vcblog/2014/04/03/cppcon-th... Also Windows Phone is the only mobile OS where C++ is a first class language. >. A lot of people like the to say that [x] is superior to C or C++ - often ignoring that their favorite language's libraries/runtime/compilers/vm is most likely implemented in the very same languages they are putting down. A lot of people like to point out that fact, ignoring that many compiler writers tend to use C or C++ out of convenience for existing tools instead of bootraping the full eco-system around a new programming language. Many use the same route as Go, using one of those languages and when the toolchain gets mature, those parts get re-writen. All in all, I do like the language, now if we could get rid of those C underpinnings...

vardump 11y ago

So Wangle is (mostly) a protocol agnostic platform for building asynchronous network clients and services in C++. What transport layers are supported and can it be extended? What I can gather, it only supports IPv4/IPv6 TCP or UDP transports. TLS is also supported. Notably, local pipes seem to be unsupported. Those can be sometimes useful for performance and security/isolation reasons.

jfasi 11y ago

The first thing that comes to mind after reading this is GRPC (https://github.com/grpc/grpc https://github.com/grpc/grpc). That library has the advantage of supporting just about every language, not just C++.

eva1984 11y ago

Good that they mentioned Netty as inspiration

deleted 11y ago

[deleted]

chetanahuja 11y ago

"Wangle is heavily influenced by the JVM-based libraries Netty," Stopped reading there. Have experience with using Netty in a java server application and engineers dread being assigned a bug or enhancement in that codebase. There's a tendency throw around the word "over-engineered" lightly but netty is a real, live and inexplicably popular example of exactly what that word means.

ausjke 11y ago

Good to see C++ is gaining more visibilities these days, considering the Oracle/Java combination I hope C++ can take away some JAVA dominance, at least nobody is going to sue you over C++ APIs. Failed to compile with CMake 2.8 CMake Error at /usr/share/cmake-2.8/Modules FindPackageHandleStandardArgs.cmake:91 (MESSAGE): REQUIRED_ARGS (missing: FOLLY_INCLUDE_DIR OLLY_LIBRARIES) Call Stack (most recent call first): /usr/share/cmake-2.8/Module/FindPackageHandleStandardArgs.cmake:252(_FPHSA_FAILURE_MESSAGE) cmake/FindFolly.cmake:23 (FIND_PACKAGE_HANDLE_STANDARD_ARGS) CMakeLists.txt:17 (find_package)

sametmax 11y ago

Also, check out crossbar.io and the WAMP protocol, offering transparent, clean and yet simple PUB/SUB & RPC accross languages, including C++, Python, PHP and JS (node and yes, browsers).

Matthias247 11y ago
WAMP is a protocol, Wangle is a framework that you can use to implement protocols (like WAMP), crossbar is a broker (an application that implements the protocol using a [python] framework. So it's not a direct comparison.