4 ms·
Tangential to the discussion, but I am currently working on a system that will handle some 100 million semi-structured rows. Despite all the buzz about "big dat
by karterk 14y ago
Tangential to the discussion, but I am currently working on a system that will handle some 100 million semi-structured rows. Despite all the buzz about "big data", hadoop and nosql etc. etc. there is literally no system that can give me a way to reliably store AND search through this data on multiple columns in near real-time (assuming I can throw a reasonable no. of nodes at it).
I looked at BigQuery, but they don't support updates. It's just dump and analyze.
- surrealize 14y agoHave you looked at hyperdex?
- karterk 14y agoYes, it's the closest to my current needs. But it's unlikely I will be able to use it currently as it's still pretty nascent and I don't feel comfortable just yet. Anyhow - I am looking to run some benchmarks on it next week.
- emidln 14y agoI needed to do something like this and I ended up settling on a custom workflow connected through zeromq dumping data into into sharded sqlite databases. Single process handled writing to each database and lots of reads were mapped out and then reduced (also via zmq). This handled around 14 billion rows when I left that job. If you want to talk some more about it, my gmail is the same as my username.
- karterk 14y agoThanks - I have emailed you.