3 ms·

> If Escriva can accomplish the same thing with Consus, in a fully open source way, and gain traction enough to keep the project alive, I could see this as some

> If Escriva can accomplish the same thing with Consus, in a fully open source way, and gain traction enough to keep the project alive, I could see this as something I would consider using. Totally agree. I feel like there's a big void in the open-source world for this type of technology that has yet to be filled. There used to be FoundationDB and, well, we all know what happened there. CockroachDB is probably the only other active, open-source project right now that claims to have strong ACID guarantees for multi-database transactions. However, the team is not wholly focused on performance at this stage of development (which some of their early benchmarks demonstrate). If Consus can retain some of the performance characteristics of Hyperdex while providing completely open source distributed transactions, it will be an absolute game changer.

lobster_johnson 10y ago
It's promising, at least. I think a system like this would be the most useful if you're able to layer a query language on top of it, and apparently even FoundationDB (which had an SQL layer on top of a K/V store) didn't perform all that well. The Cockroach guys claim to have found a performant way of doing SQL on top of K/V, if I remember correctly, but we'll see. But having transactions in a distributed K/V store is certainly useful.
- spullara 10y ago
  If you are willing to optimize for a KV store with distributed transactions you can do a lot better than putting a SQL layer on top of it.
  - lobster_johnson 10y ago
    Not sure what you mean by that. Consus seems to be directly inspired by Google's Spanner, which famously has an SQL layer, F1, on top.
    - spullara 10y ago
      It does, but unless you know it is on top of a distributed KV store and design your schema and queries around that you will suffer horrible performance issues. Also, F1, similar to FDBs implementation, forces you to model your databases differently than you would in a "normal" oltp relational database.
      - rescrv 10y ago
        Almost every single host SQL solution I know of is built on a B-Tree for at least some of its properties. A B-Tree is nothing more than a lexicographically sorted key-value store. Consus is a lexicographically sorted key-value store. Where you will hit performance bottlenecks is when your query planner/executer makes traditional assumptions about performance that do not hold across machines. A good SQL implementation on top of a kvs should not force you to accommodate it any more than your current "normal" databases subconsciously train you to accommodate your disks (hint: they do).
        spullara 10y ago
        I would be very surprised if anyone is still running their primary data store on "disks" that cares about performance. BTW, look at the design of F1 and you will see that they do object like hierarchical storage of related tables to ensure that they don't have to go cross partition for queries.
spullara 10y ago
I've been watching this project as a possible replacement to FoundationDB as well. https://github.com/pingcap/tikv https://github.com/pingcap/tikv

lobster_johnson 10y ago

It's promising, at least. I think a system like this would be the most useful if you're able to layer a query language on top of it, and apparently even FoundationDB (which had an SQL layer on top of a K/V store) didn't perform all that well. The Cockroach guys claim to have found a performant way of doing SQL on top of K/V, if I remember correctly, but we'll see. But having transactions in a distributed K/V store is certainly useful.

spullara 10y ago
If you are willing to optimize for a KV store with distributed transactions you can do a lot better than putting a SQL layer on top of it.
- lobster_johnson 10y ago
  Not sure what you mean by that. Consus seems to be directly inspired by Google's Spanner, which famously has an SQL layer, F1, on top.
  - spullara 10y ago
    It does, but unless you know it is on top of a distributed KV store and design your schema and queries around that you will suffer horrible performance issues. Also, F1, similar to FDBs implementation, forces you to model your databases differently than you would in a "normal" oltp relational database.
    - rescrv 10y ago
      Almost every single host SQL solution I know of is built on a B-Tree for at least some of its properties. A B-Tree is nothing more than a lexicographically sorted key-value store. Consus is a lexicographically sorted key-value store. Where you will hit performance bottlenecks is when your query planner/executer makes traditional assumptions about performance that do not hold across machines. A good SQL implementation on top of a kvs should not force you to accommodate it any more than your current "normal" databases subconsciously train you to accommodate your disks (hint: they do).
      - spullara 10y ago
        I would be very surprised if anyone is still running their primary data store on "disks" that cares about performance. BTW, look at the design of F1 and you will see that they do object like hierarchical storage of related tables to ensure that they don't have to go cross partition for queries.

spullara 10y ago

I've been watching this project as a possible replacement to FoundationDB as well. https://github.com/pingcap/tikv https://github.com/pingcap/tikv