3 ms·

Show HN: RaftLib – Stream Computation library for C++

45 points by jcbeard 11y ago

jcbeard 11y ago
thanks for checking out, awesome feedback so far!
warthog10 11y ago
So I saw in the medium post how this compares to pbzip2 with standard pthreads. how does it compare to an alternative (say threading building blocks). I also saw in one of the papers that with a Spark comparison, would like a compiled to compiled comparison though.
erichocean 11y ago
How does this differ from FastFlow?[0] [0] http://calvados.di.unipi.it/ http://calvados.di.unipi.it/
- jcbeard 11y ago
  I'm not totally familiar with FF, but reading through their manual and checking out the examples, also using this FastFlow example as a reference ( http://sourceforge.net/p/mc-fastflow/code/HEAD/tree/examples/pbzip2/ http://sourceforge.net/p/mc-fastflow/code/HEAD/tree/examples...): 1) FF still seems to require explicit locking 2) The job management in FF requires explicit function calls 3) Not illustrated in the example, but RaftLib can auto-parallelize using the raft::out keyword inline with the c++ stream (using SESE) within the DAG. 4) Auto-buffer management (FF appears to require explicit calls to allocate/free workers, and manage memory), RaftLib auto-manages buffer allocation based on queueing occupancy, queueing theory. 5) I like the FF2 explicit scheduling, RaftLib has a few, they're all selected by the run-time. Right now the default is the system scheduler since I'm trying to get the Win64 cross build stabilized before I go back to any crazier threading (there's a fiber plug-in too). 6) The interface is hugely simpler with RaftLib compared to FF for adding "workers" 7) RaftLib explicitly has lock-free communication between actors, seems FF2 requires at least the mutex, and there's not a simpler FIFO interface there (could be totally wrong, please correct me if I am...again, just looking at the docs). There's tons of other differences aesthetically/documentationally (I'm checking out now!) Thanks for the reference, this doesn't seem to pop up on any scholar.google hits and I hadn't run into it at any of the academic/industry conferences I've attended/presented this to (ISPASS, MASCOTS, HPCC, EuroPar, etc.) so sometimes it's hard to find related work. I'll definitely update my rbzip2 benchmark with their pbzip2 code tonight or tomorrow!
  - erichocean 11y ago
    1) FF still seems to require explicit locking Actually, it's main claim to fame is being lock free (on Intel architectures). At most it requires a fence on weaker architectures (e.g. ARM) for one of the operations. You don't use a mutex to communicate between workers, you send a message over a (lock free) queue. Anyway, the auto-scheduling stuff in RaftLib looks pretty cool. Thanks for the response.
- jcbeard 11y ago
  ooh, forgot to mention...there's also a fully distributed setup that's coming soon (well, already here...removed b/c I'm re-coding it). The original multi-node (UDP) stack came from a summer research student. It functioned, but not that well. Since I'm spending time optimizing that, I'm also adding container support so remote kernels will operated inside a container w/a common file system view. Another difference I probably didn't mention is that there is an instrumentation package that monitors the actual non-blocking service rate in real time (well, as real time as you can be across multiple clocking domains). The acyclic graph structure is awesome for using things like non-blocking service rate to optimize placement, make parallelization choices, etc. My original interest was using this framework for my online optimization work. Basically I've been using support vector machines and neural networks to select models and make buffer sizing decisions....

45 points by jcbeard 11y ago

thanks for checking out, awesome feedback so far!

So I saw in the medium post how this compares to pbzip2 with standard pthreads. how does it compare to an alternative (say threading building blocks). I also saw in one of the papers that with a Spark comparison, would like a compiled to compiled comparison though.

erichocean 11y ago

How does this differ from FastFlow?[0] [0] http://calvados.di.unipi.it/ http://calvados.di.unipi.it/

jcbeard 11y ago
I'm not totally familiar with FF, but reading through their manual and checking out the examples, also using this FastFlow example as a reference ( http://sourceforge.net/p/mc-fastflow/code/HEAD/tree/examples/pbzip2/ http://sourceforge.net/p/mc-fastflow/code/HEAD/tree/examples...): 1) FF still seems to require explicit locking 2) The job management in FF requires explicit function calls 3) Not illustrated in the example, but RaftLib can auto-parallelize using the raft::out keyword inline with the c++ stream (using SESE) within the DAG. 4) Auto-buffer management (FF appears to require explicit calls to allocate/free workers, and manage memory), RaftLib auto-manages buffer allocation based on queueing occupancy, queueing theory. 5) I like the FF2 explicit scheduling, RaftLib has a few, they're all selected by the run-time. Right now the default is the system scheduler since I'm trying to get the Win64 cross build stabilized before I go back to any crazier threading (there's a fiber plug-in too). 6) The interface is hugely simpler with RaftLib compared to FF for adding "workers" 7) RaftLib explicitly has lock-free communication between actors, seems FF2 requires at least the mutex, and there's not a simpler FIFO interface there (could be totally wrong, please correct me if I am...again, just looking at the docs). There's tons of other differences aesthetically/documentationally (I'm checking out now!) Thanks for the reference, this doesn't seem to pop up on any scholar.google hits and I hadn't run into it at any of the academic/industry conferences I've attended/presented this to (ISPASS, MASCOTS, HPCC, EuroPar, etc.) so sometimes it's hard to find related work. I'll definitely update my rbzip2 benchmark with their pbzip2 code tonight or tomorrow!
- erichocean 11y ago
  1) FF still seems to require explicit locking Actually, it's main claim to fame is being lock free (on Intel architectures). At most it requires a fence on weaker architectures (e.g. ARM) for one of the operations. You don't use a mutex to communicate between workers, you send a message over a (lock free) queue. Anyway, the auto-scheduling stuff in RaftLib looks pretty cool. Thanks for the response.
jcbeard 11y ago
ooh, forgot to mention...there's also a fully distributed setup that's coming soon (well, already here...removed b/c I'm re-coding it). The original multi-node (UDP) stack came from a summer research student. It functioned, but not that well. Since I'm spending time optimizing that, I'm also adding container support so remote kernels will operated inside a container w/a common file system view. Another difference I probably didn't mention is that there is an instrumentation package that monitors the actual non-blocking service rate in real time (well, as real time as you can be across multiple clocking domains). The acyclic graph structure is awesome for using things like non-blocking service rate to optimize placement, make parallelization choices, etc. My original interest was using this framework for my online optimization work. Basically I've been using support vector machines and neural networks to select models and make buffer sizing decisions....