4 ms·

From their blog: "TiKV is a distributed key-value database. It is the core component of the TiDB project and is the open source implementation of Google Spanner

by eloff 9y ago

From their blog: "TiKV is a distributed key-value database. It is the core component of the TiDB project and is the open source implementation of Google Spanner." I think the Cockroach DB guys would take offence at that. So how do they use a Spanner-like design without specialized hardware? I had to really dig but the answer is "We are using the Timestamp Allocator introduced in Percolator, a paper published by Google in 2006. The pros of using the Timestamp Allocator are its easy implementation and no dependency on any hardware. The disadvantage lies in that if there are multiple datacenters, especially if these DCs are geologically distributed, the latency is really high." Using Spanner's design but without the real hardware that makes it practical seems like a step backward. A Calvin[1] based approach like FaunaDB would probably have been better - but they can't do that and still have MySQL compatibility (you can't do interactive client-server transactions with Calvin.) I'm skeptical of a database built over a generic KV-store. There was a much-hyped database a few years ago that failed to live up to the hype because of exactly that architecture. I can't even remember the name now when I was trying to find the post-mortem analysis for it in Google. I'm also skeptical of a database claiming to be good at both OLAP and OLTP. One requires a column store, the other a row store. You can be half-decent as an OLAP store and good as a OLTP store. There are no column-stores that are also good at OLTP. But OLAP is the real big-data problems of today, and using half-decent for that is likely to get you in trouble. There's no reason a database can't do both by using two separate storage engines under the hood, but that doesn't seem to be the case here. [1] https://fauna.com/blog/distributed-consistency-at-scale-spanner-vs-calvin https://fauna.com/blog/distributed-consistency-at-scale-span...

zzzcpan 9y ago
KV-store is probably the reason why they can claim OLAP, since distributing batch load on top of it is easier.
nemothekid 9y ago
>There was a much-hyped database a few years ago that failed to live up to the hype because of exactly that architecture. The only database I can remember that did something like this was FoundationDB. They had the core layer as a K/V, and a layer above that was FSQL or similar. They got acquired by Apple, and most of their material was taken off line. I don't know if they were a failure (IIRC, some people were pretty impressed with the core NoSQL DB), but an exit to Apple may not have been all bad - although I would bet that Apple is not using the technology internally. IME, my beef with the new NewSQL solutions is that they tend to be worse at both NoSQL and SQL workloads (specifically none are "the best" at OLAP and OLTP) except at one specific use case - global-replication. Getting global replication requires a cost (either monetary like Spanner, or performance wise like everything else), and global-replication just doesn't seem like a killer feature to me. Most times I need data in two datacenters globally, I wouldn't replicate the data anyways (don't want EU data in the US), and other times my SLAs just aren't tight enough to warrant the cost.
- zzzcpan 9y ago
  I can't remember, was it FoundationDB that didn't have interactive client-server transactions too?
jasonwatkinspdx 9y ago
> There was a much-hyped database a few years ago that failed to live up to the hype because of exactly that architecture. I can't even remember the name now when I was trying to find the post-mortem analysis for it in Google. FoundationDB, acquired and taken internal by Apple. There were a few different blog posts about the layering aspect of it, with mixed opinions. They built a transactionally consistent distributed database that was mysql client compatible and within 50% of mysql single node performance. If creating that and getting it bought by Apple is failing, I hope I'm lucky enough to fail like that. > I'm also skeptical of a database claiming to be good at both OLAP and OLTP. One requires a column store, the other a row store. You can be half-decent as an OLAP store and good as a OLTP store. This is a rigid and narrow view. There's lots of other design possibilities such as fractured mirrors, column decommission only within pages, etc. I wouldn't call a single db that can do both OLAP and OLTP easy or a slam dunk, but I also wouldn't rule it out. VoltDB is well proven and worth looking at.
- lobster_johnson 9y ago
  > If creating that and getting it bought by Apple is failing, I hope I'm lucky enough to fail like that. I think the parent is referring to performance. FoundationDB had a fast key-value store, but they supposedly weren't able to get good performance out of their SQL layer.
- jsmthrowaway 9y ago
  > If creating that and getting it bought by Apple is failing, I hope I'm lucky enough to fail like that. Careful, though, because one has to consider life after the acquisition if one loves the technology.
- eloff 9y ago
  Thank you, it was FoundationDB I was thinking of, post-mortem analysis from the VoltDB guys here[1]. I'll contend they failed though because Apple bought them as an acqui-hire and killed the product. They were acquiring good engineers with a track record of executing well at a premium. It's a common recruiting strategy for tech companies. It doesn't mean that FoundationDB was a successful product - certainly it means it was not successful enough for the founders to risk turning down a lucrative acqui-hire deal. > his is a rigid and narrow view. There's lots of other design possibilities such as fractured mirrors, column decommission only within pages, etc. Yes definitely I over-simplified to make my point, but I'll stand by it. OLAP and OLTP are different enough workloads and the engineering tradeoffs for being good at one of the other make it hard to excel at both. VoltDB is a row store, best suited for OLTP. Vertica is a column store best suited for OLAP. VoltDB can also make a decent OLAP database. Vertica does not make a good OLTP database. Both databases come out of Michael Stonebraker's work (although given the number of people required to make complex products like VoltDB and Vertica happen, it seems unfair to credit him unduly.) But especially in the case of a sharded, distributed database where communication overhead is likely to dominate many types of queries (VoltDB, TiDB), one might not notice the inefficiency of using a poorly-suited design, like a row store for OLAP. This can make the differences less striking. This might also hide the inefficiency of basing everything on a generic KV store in TiDB. [1]https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-...
jandrewrogers 9y ago
There are other (widely used, in fact) alternatives to a classic column store or row store that exist for precisely the points you raise. It is possible to get something like (made up number) 80% of the OLTP performance of a row store and 80% of the OLAP performance of a column store if you employ some of the fancier page engine designs from the last decade, which is a good tradeoff for workload flexibility even if technically suboptimal for read and write exclusively. However, I would argue that page layout doesn't matter that much in terms of system performance in very large scale-out systems. Other factors often dominate. I've seen row stores crush columnar stores for query performance, not because the row store was efficient (it wasn't) but because the inefficiency due to relative organization was offset by even greater efficiencies elsewhere in the architecture relative to the columnar store. Hence why using one of the "pretty good but not ideal for read and write" page engine designs and focusing elsewhere for query performance is a reasonable design choice. I would agree that a generic KV-store is not a particularly efficient basis for a general-purpose database. You need to get much more query selectivity out of the content-addressability than a KV-store typically gives you for non-trivial data models.
manigandham 9y ago
CockroachDB is also built on a key/value layer using a Spanner type design. MemSQL is OLAP that does OLTP well, and Citus can make Postgres handle OLAP pretty well too (as used at Heap). The article you're referring to is about Foundation DB written by John Hugg of VoltDB: https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-... The database world is rapidly changing and we're seeing new technologies that are enabling a lot more than we could do before. Just looking at some of the SQL on X engines alone shows massive improvements.

by eloff 9y ago

zzzcpan 9y ago

KV-store is probably the reason why they can claim OLAP, since distributing batch load on top of it is easier.

nemothekid 9y ago

>There was a much-hyped database a few years ago that failed to live up to the hype because of exactly that architecture. The only database I can remember that did something like this was FoundationDB. They had the core layer as a K/V, and a layer above that was FSQL or similar. They got acquired by Apple, and most of their material was taken off line. I don't know if they were a failure (IIRC, some people were pretty impressed with the core NoSQL DB), but an exit to Apple may not have been all bad - although I would bet that Apple is not using the technology internally. IME, my beef with the new NewSQL solutions is that they tend to be worse at both NoSQL and SQL workloads (specifically none are "the best" at OLAP and OLTP) except at one specific use case - global-replication. Getting global replication requires a cost (either monetary like Spanner, or performance wise like everything else), and global-replication just doesn't seem like a killer feature to me. Most times I need data in two datacenters globally, I wouldn't replicate the data anyways (don't want EU data in the US), and other times my SLAs just aren't tight enough to warrant the cost.

zzzcpan 9y ago
I can't remember, was it FoundationDB that didn't have interactive client-server transactions too?

jasonwatkinspdx 9y ago

> There was a much-hyped database a few years ago that failed to live up to the hype because of exactly that architecture. I can't even remember the name now when I was trying to find the post-mortem analysis for it in Google. FoundationDB, acquired and taken internal by Apple. There were a few different blog posts about the layering aspect of it, with mixed opinions. They built a transactionally consistent distributed database that was mysql client compatible and within 50% of mysql single node performance. If creating that and getting it bought by Apple is failing, I hope I'm lucky enough to fail like that. > I'm also skeptical of a database claiming to be good at both OLAP and OLTP. One requires a column store, the other a row store. You can be half-decent as an OLAP store and good as a OLTP store. This is a rigid and narrow view. There's lots of other design possibilities such as fractured mirrors, column decommission only within pages, etc. I wouldn't call a single db that can do both OLAP and OLTP easy or a slam dunk, but I also wouldn't rule it out. VoltDB is well proven and worth looking at.

lobster_johnson 9y ago
> If creating that and getting it bought by Apple is failing, I hope I'm lucky enough to fail like that. I think the parent is referring to performance. FoundationDB had a fast key-value store, but they supposedly weren't able to get good performance out of their SQL layer.
jsmthrowaway 9y ago
> If creating that and getting it bought by Apple is failing, I hope I'm lucky enough to fail like that. Careful, though, because one has to consider life after the acquisition if one loves the technology.
eloff 9y ago
Thank you, it was FoundationDB I was thinking of, post-mortem analysis from the VoltDB guys here[1]. I'll contend they failed though because Apple bought them as an acqui-hire and killed the product. They were acquiring good engineers with a track record of executing well at a premium. It's a common recruiting strategy for tech companies. It doesn't mean that FoundationDB was a successful product - certainly it means it was not successful enough for the founders to risk turning down a lucrative acqui-hire deal. > his is a rigid and narrow view. There's lots of other design possibilities such as fractured mirrors, column decommission only within pages, etc. Yes definitely I over-simplified to make my point, but I'll stand by it. OLAP and OLTP are different enough workloads and the engineering tradeoffs for being good at one of the other make it hard to excel at both. VoltDB is a row store, best suited for OLTP. Vertica is a column store best suited for OLAP. VoltDB can also make a decent OLAP database. Vertica does not make a good OLTP database. Both databases come out of Michael Stonebraker's work (although given the number of people required to make complex products like VoltDB and Vertica happen, it seems unfair to credit him unduly.) But especially in the case of a sharded, distributed database where communication overhead is likely to dominate many types of queries (VoltDB, TiDB), one might not notice the inefficiency of using a poorly-suited design, like a row store for OLAP. This can make the differences less striking. This might also hide the inefficiency of basing everything on a generic KV store in TiDB. [1]https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-...

jandrewrogers 9y ago

There are other (widely used, in fact) alternatives to a classic column store or row store that exist for precisely the points you raise. It is possible to get something like (made up number) 80% of the OLTP performance of a row store and 80% of the OLAP performance of a column store if you employ some of the fancier page engine designs from the last decade, which is a good tradeoff for workload flexibility even if technically suboptimal for read and write exclusively. However, I would argue that page layout doesn't matter that much in terms of system performance in very large scale-out systems. Other factors often dominate. I've seen row stores crush columnar stores for query performance, not because the row store was efficient (it wasn't) but because the inefficiency due to relative organization was offset by even greater efficiencies elsewhere in the architecture relative to the columnar store. Hence why using one of the "pretty good but not ideal for read and write" page engine designs and focusing elsewhere for query performance is a reasonable design choice. I would agree that a generic KV-store is not a particularly efficient basis for a general-purpose database. You need to get much more query selectivity out of the content-addressability than a KV-store typically gives you for non-trivial data models.

manigandham 9y ago

CockroachDB is also built on a key/value layer using a Spanner type design. MemSQL is OLAP that does OLTP well, and Citus can make Postgres handle OLAP pretty well too (as used at Heap). The article you're referring to is about Foundation DB written by John Hugg of VoltDB: https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-... The database world is rapidly changing and we're seeing new technologies that are enabling a lot more than we could do before. Just looking at some of the SQL on X engines alone shows massive improvements.