I do something similar with the stuff for http://lifecode.co http://lifecode.co Couple of terrabytes processing easy but yeah limited to only a few queries per

I do something similar with the stuff for http://lifecode.co http://lifecode.co Couple of terrabytes processing easy but yeah limited to only a few queries per day. A lot of the effort goes into caching so disk reads and swap file use is minimised. Really that's where all the overhead is. All the public stuff for that is at http://theborgmatrix.com http://theborgmatrix.com