3 ms·
Of these Crux, looks quite interesting. How well can it scale? Can it handle billions of documents? For us, history is mostly about being able to audit a recor
by Bedon292 6y ago
Of these Crux, looks quite interesting. How well can it scale? Can it handle billions of documents?
For us, history is mostly about being able to audit a record and understand it history, and occasionally undo a mistake. Don't need or want to go with things like SAP.
- refset 6y agoCrux is designed to scale directly based on how RocksDB (or LMDB) performs on a single node, in terms of: sustained ingestion throughput, KV seeks/sec, and the sheer quantity of KV data that can be supported on an array of local SSDs (i.e. easily many billions of small docs). At a higher level this means that point-in-time queries will maintain good performance regardless of how much history is stored, thanks to Z-order indexing [0], and the query algorithm only requires very modest amounts of memory because the KV indexes are lazily streamed out of Rocks and processed tuple-by-tuple (though having more memory is always going to speed things up!). Beyond the scope of a single Crux node, horizontal read scaling comes for free due to the transaction time model of history (i.e. you can spin up N identical nodes to service all manner of wholly unrelated use-cases with consistent reads). [0] https://en.wikipedia.org/wiki/Z-order_curve https://en.wikipedia.org/wiki/Z-order_curve