6 ms·

I just tried to compile folly in OSX and it fails to compile with homebrew. https://github.com/facebook/folly/issues/400 https://github.com/facebook/folly/issue

by thepumpkin1979 10y ago

I just tried to compile folly in OSX and it fails to compile with homebrew. https://github.com/facebook/folly/issues/400 https://github.com/facebook/folly/issues/400 Also, Once I manage to compile folly and then wangle, what kind of database drivers can I use in conjunction with this? Would I shoot myself in the foot if I use this with libpq under CPUThreadPoolExecutor? Same with Rust and dlang, I'm not sure how to apply the freedom to have pluggable executor into something useful in real life, I think this is the main reason Golang is killing it and it keeps getting better. For now I have my money on Go but can't wait for http://www.ponylang.org/ http://www.ponylang.org/ to become stable. Edit: Issue reported has been closed, I can now proceed to install wangle https://github.com/facebook/folly/issues/400 https://github.com/facebook/folly/issues/400 Edit 2: Found another issue, this time installing wangle https://github.com/facebook/wangle/issues/37 https://github.com/facebook/wangle/issues/37

pcwalton 10y ago
Go uses synchronous I/O backed with an M:N implementation. Rust and D use synchronous I/O backed with a 1:1 implementation. Semantically, there is no difference between the two. The difference is in the implementation, and 1:1 threading is not as slow as you think on Linux.
- tptacek 10y ago
  Why are we talking about Go, Rust, and D on a thread about a new C++ library? I mean, I know the answer, but I wish we weren't. I ordinarily wouldn't care, but I usually feel like I have something to learn from ambitious networking libraries even when they're written in languages I prefer not to use.
  - pcwalton 10y ago
    For what it's worth, I try to not bring up Rust in threads about other languages; it just happens that most threads about Rust appear in threads about other languages. I would rather talk about the implementation of async I/O too.
  - thepumpkin1979 10y ago
    I just wanted to point out the barriers a developer faces when trying to use C++. Sorry, I can't resist compare it to what I know, which happens to be other programming languages different to C++.
- dunkelheit 10y ago
  Does that mean that one can use system threads as frivolously as goroutines (e.g. spawning new thread for each incoming request)? Conventional C++ wisdom advices against it. But then it may be informed by some old and bad pthreads implementations.
  - jhartmann 10y ago
    A reasonable compromise is to use a threadpool (which is a big part of what comes with wangle) with a fixed number of threads and you queue the tasks to be run by the threads. It works quite well in practice, and you can use coroutines (I'm most familiar with the implementation in boost coroutine and the Ewan webserver personally) to go the whole hog and get something as lightweight as Erlang green threads or goroutines. I would only bother doing that of course if in your particular use case you find the threadpool doesn't perform well. With proper sizing and broken up units of work, it really works quite well in practice.
  - pcwalton 10y ago
    Usually memory usage ends up being the limiting factor, not thread spawning performance. Memory usage is independent of 1:1 vs. M:N, though, as pthread stack sizes are configurable.
  - otabdeveloper 10y ago
    > Does that mean that one can use system threads as frivolously as goroutines (e.g. spawning new thread for each incoming request)? Yes. In fact, under Linux oftentimes spawning a thread and then running a system call can be faster than just running a system call. > Conventional C++ wisdom advices against it. With 32 bit CPU's you might run out of virtual memory quickly, since each thread allocates virtual memory for its stack. With 64 bit CPU's there are no issues. (Also never forget that virtual memory isn't real memory, it's just a page counter.)
    - pcwalton 10y ago
      > Yes. In fact, under Linux oftentimes spawning a thread and then running a system call can be faster than just running a system call. Interesting, do you have any further info (references, cites) on this?
zzzcpan 10y ago
> what kind of database drivers can I use in conjunction with this? Usually synchronous database drivers have to be reimplemented in an asynchronous fashion to work with event loops. It's not very hard though, could be as simple as adding jump tables and context structures into synchronous functions. Emulating asynchronous execution with a thread is ok too. You do that to access filesystem anyway. But if your first thought is about accessing a database server - you probably don't need to go asynchronous at all.
- malkia 10y ago
  > Emulating asynchronous execution with a thread is ok too Straying away from the topic, but I used to support a file streaming library for the game studio I worked with, and while this approach worked, some things were not gracefully handled - for example canceling reading in the middle, or somehow giving all your I/O requests to the underlying system, so it can decide what's the best way to read them (aio?).
- jhartmann 10y ago
  The wangle native way of doing this is just to use an IOThreadPoolExecutor: https://github.com/facebook/wangle/blob/master/wangle/concurrent/IOThreadPoolExecutor.h https://github.com/facebook/wangle/blob/master/wangle/concur... Just make a block of threads to run your db tasks, and you can use a promise/future to return the results back to the async code.
- fredemmott 10y ago
  There's https://github.com/facebook/squangle https://github.com/facebook/squangle for MySQL, though it's lacking documentation, and requires a https://github.com/facebook/mysql-5.6 https://github.com/facebook/mysql-5.6 or WebScaleSQL client library
- jcoffland 10y ago
  MariaDB has an asynchronous C API.
  - infinite8s 10y ago
sreque 10y ago
The problem is the exact opposite. The question you should be asking, is: how can I use my go code in Rust/D/C++? The answer is, not very easy, given that go's runtime is largely incompatible with C. At least with C++, you can easily write C shim layers and your code becomes compatible with most runtimes in existence. The opposite is not true with Go.

pcwalton 10y ago

Go uses synchronous I/O backed with an M:N implementation. Rust and D use synchronous I/O backed with a 1:1 implementation. Semantically, there is no difference between the two. The difference is in the implementation, and 1:1 threading is not as slow as you think on Linux.

tptacek 10y ago
Why are we talking about Go, Rust, and D on a thread about a new C++ library? I mean, I know the answer, but I wish we weren't. I ordinarily wouldn't care, but I usually feel like I have something to learn from ambitious networking libraries even when they're written in languages I prefer not to use.
- pcwalton 10y ago
  For what it's worth, I try to not bring up Rust in threads about other languages; it just happens that most threads about Rust appear in threads about other languages. I would rather talk about the implementation of async I/O too.
- thepumpkin1979 10y ago
  I just wanted to point out the barriers a developer faces when trying to use C++. Sorry, I can't resist compare it to what I know, which happens to be other programming languages different to C++.
dunkelheit 10y ago
Does that mean that one can use system threads as frivolously as goroutines (e.g. spawning new thread for each incoming request)? Conventional C++ wisdom advices against it. But then it may be informed by some old and bad pthreads implementations.
- jhartmann 10y ago
  A reasonable compromise is to use a threadpool (which is a big part of what comes with wangle) with a fixed number of threads and you queue the tasks to be run by the threads. It works quite well in practice, and you can use coroutines (I'm most familiar with the implementation in boost coroutine and the Ewan webserver personally) to go the whole hog and get something as lightweight as Erlang green threads or goroutines. I would only bother doing that of course if in your particular use case you find the threadpool doesn't perform well. With proper sizing and broken up units of work, it really works quite well in practice.
- pcwalton 10y ago
  Usually memory usage ends up being the limiting factor, not thread spawning performance. Memory usage is independent of 1:1 vs. M:N, though, as pthread stack sizes are configurable.
- otabdeveloper 10y ago
  > Does that mean that one can use system threads as frivolously as goroutines (e.g. spawning new thread for each incoming request)? Yes. In fact, under Linux oftentimes spawning a thread and then running a system call can be faster than just running a system call. > Conventional C++ wisdom advices against it. With 32 bit CPU's you might run out of virtual memory quickly, since each thread allocates virtual memory for its stack. With 64 bit CPU's there are no issues. (Also never forget that virtual memory isn't real memory, it's just a page counter.)
  - pcwalton 10y ago
    > Yes. In fact, under Linux oftentimes spawning a thread and then running a system call can be faster than just running a system call. Interesting, do you have any further info (references, cites) on this?

zzzcpan 10y ago

> what kind of database drivers can I use in conjunction with this? Usually synchronous database drivers have to be reimplemented in an asynchronous fashion to work with event loops. It's not very hard though, could be as simple as adding jump tables and context structures into synchronous functions. Emulating asynchronous execution with a thread is ok too. You do that to access filesystem anyway. But if your first thought is about accessing a database server - you probably don't need to go asynchronous at all.

malkia 10y ago
> Emulating asynchronous execution with a thread is ok too Straying away from the topic, but I used to support a file streaming library for the game studio I worked with, and while this approach worked, some things were not gracefully handled - for example canceling reading in the middle, or somehow giving all your I/O requests to the underlying system, so it can decide what's the best way to read them (aio?).
jhartmann 10y ago
The wangle native way of doing this is just to use an IOThreadPoolExecutor: https://github.com/facebook/wangle/blob/master/wangle/concurrent/IOThreadPoolExecutor.h https://github.com/facebook/wangle/blob/master/wangle/concur... Just make a block of threads to run your db tasks, and you can use a promise/future to return the results back to the async code.
fredemmott 10y ago
There's https://github.com/facebook/squangle https://github.com/facebook/squangle for MySQL, though it's lacking documentation, and requires a https://github.com/facebook/mysql-5.6 https://github.com/facebook/mysql-5.6 or WebScaleSQL client library
jcoffland 10y ago
MariaDB has an asynchronous C API.
- infinite8s 10y ago

sreque 10y ago

The problem is the exact opposite. The question you should be asking, is: how can I use my go code in Rust/D/C++? The answer is, not very easy, given that go's runtime is largely incompatible with C. At least with C++, you can easily write C shim layers and your code becomes compatible with most runtimes in existence. The opposite is not true with Go.