I needed to do something like this and I ended up settling on a custom workflow connected through zeromq dumping data into into sharded sqlite databases. Single

I needed to do something like this and I ended up settling on a custom workflow connected through zeromq dumping data into into sharded sqlite databases. Single process handled writing to each database and lots of reads were mapped out and then reduced (also via zmq). This handled around 14 billion rows when I left that job. If you want to talk some more about it, my gmail is the same as my username.