4 ms·

If you are considering TiDB for OLAP workloads, you should first evaluate an OLAP-only data warehouse like Snowflake, BigQuery, or Redshift. A "hybrid" database

If you are considering TiDB for OLAP workloads, you should first evaluate an OLAP-only data warehouse like Snowflake, BigQuery, or Redshift. A "hybrid" database sounds like a good idea, until you realize that you probably don't want ad-hoc analytical queries running against your production database, so you're going to end up running two systems anyway. Furthermore, column-stores are so different than row-stores in their ideal implementation that they are effectively completely different systems with the same outer interface (SQL). Daniel Abadi wrote a fantastic paper [1] in which he showed that fast column stores must use completely different strategies at every layer of query processing. You should be very skeptical of any database claiming to be good at both transaction processing and analytical workloads. Such as system would effectively be two different databases hiding behind the same SQL interface, and would not be as useful as you might expect, for devops reasons. [1] http://db.csail.mit.edu/pubs/abadi-column-stores.pdf http://db.csail.mit.edu/pubs/abadi-column-stores.pdf

morgo 8y ago
Morgan from the TiDB team here. I agree that column and row store have very different characteristics, but what I think is worth mentioning is that some hybrid solutions actually store as both row and columnar and have a query optimizer that can pick between them. For example: Oracle DB In-Memory, SQL Server Columnstore index. At the same event as this announcement, we also announced that we are working on TiFlash which will do similar. Stay tuned for a blog post with more details :-)
- georgewfraser 8y ago
  As described in that paper, it’s not sufficient to simply store a second, columnar projection of the data to get good performance. You also need a block-oriented execution engine, which means you effectively have two separate databases operating side-by-side. This is a huge challenge and it’s not clear if it’s really worth it, since for logistical reasons you will nearly always operate a separate data warehouse doing mostly OLAP and production database doing mostly OLTP.
manigandham 8y ago
The only products that support HTAP (hybrid OLTP/OLAP) well so far are MemSQL and SQL Server. It's definitely possible but youre right, it takes a lot of engineering and requires column stores and advanced execution engines to make it happen. I think the OLAP scenario is pretty well covered these days while scalable relational OLTP is still new and would likely be the better focus.

morgo 8y ago

Morgan from the TiDB team here. I agree that column and row store have very different characteristics, but what I think is worth mentioning is that some hybrid solutions actually store as both row and columnar and have a query optimizer that can pick between them. For example: Oracle DB In-Memory, SQL Server Columnstore index. At the same event as this announcement, we also announced that we are working on TiFlash which will do similar. Stay tuned for a blog post with more details :-)

georgewfraser 8y ago
As described in that paper, it’s not sufficient to simply store a second, columnar projection of the data to get good performance. You also need a block-oriented execution engine, which means you effectively have two separate databases operating side-by-side. This is a huge challenge and it’s not clear if it’s really worth it, since for logistical reasons you will nearly always operate a separate data warehouse doing mostly OLAP and production database doing mostly OLTP.

manigandham 8y ago

The only products that support HTAP (hybrid OLTP/OLAP) well so far are MemSQL and SQL Server. It's definitely possible but youre right, it takes a lot of engineering and requires column stores and advanced execution engines to make it happen. I think the OLAP scenario is pretty well covered these days while scalable relational OLTP is still new and would likely be the better focus.