3 ms·

Relational databases with joins and full SQL support are still unmatched in flexibility, and functionality like materialized views, aggregation pipelines (and t

Relational databases with joins and full SQL support are still unmatched in flexibility, and functionality like materialized views, aggregation pipelines (and table engines for Clickhouse) allows you to do everything that Druid does with aggregated summaries while still having all the other querying abilities. Druid has a slight edge in data scale-out and indexed seeks but modern data warehouses are adding similar tiering features, along with with field indexing, full-text search, nested data records, and even in-memory rowstores for OLTP support. They're all converging on the same feature set and eventually Druid will just become another data warehouse option, although I'd still recommend Clickhouse or MemSQL at that point.

sanderjd 7y ago
Question for everyone in this thread: what resources do you use to follow this stuff? Are there blogs or conferences or ... that I should follow if I'm interested in this topic?
- manigandham 7y ago
  Distributed SQL data warehouses? Start with the papers: - C-Store (column-oriented database storage and processing): http://db.lcs.mit.edu/projects/cstore/vldb.pdf http://db.lcs.mit.edu/projects/cstore/vldb.pdf and https://sookocheff.com/post/databases/c-store/ https://sookocheff.com/post/databases/c-store/ - BigQuery (the original Dremel-based query engine): https://cloud.google.com/files/BigQueryTechnicalWP.pdf https://cloud.google.com/files/BigQueryTechnicalWP.pdf - Snowflake paper: https://www.snowflake.com/resource/sigmod-2016-paper-snowflake-elastic-data-warehouse/ https://www.snowflake.com/resource/sigmod-2016-paper-snowfla... - Andy Pavlo's CMU distributed OLAP databases lecture: https://www.youtube.com/watch?v=dPMc7FZ3Gqo&t=1s https://www.youtube.com/watch?v=dPMc7FZ3Gqo&t=1s - MemSQL architecture: https://www.youtube.com/watch?v=HoompXCG5Mk https://www.youtube.com/watch?v=HoompXCG5Mk Start there and just read more about architecture of the various products. You'll learn the basics of distributed nodes that store data in column-oriented tables partitioned by a key and how to run queries over them using various performance techniques. Then you can dive into the unique differences of the various databases.
  - polskibus 7y ago
    I recommend watching Andy Pavlo's courses: https://15445.courses.cs.cmu.edu/fall2019/ https://15445.courses.cs.cmu.edu/fall2019/ https://15721.courses.cs.cmu.edu/spring2020/ https://15721.courses.cs.cmu.edu/spring2020/
  - sanderjd 7y ago
    No sorry, I didn't mean where to read about how these things are designed (though these are useful and interesting links for that too, thanks!). What I'm asking about is what to follow (blogs, conferences, etc.) to keep up with new developments in the space: new products, experiences with them, reviews of their trade offs in practical use, that kind of thing. But I do appreciate the response!
    - manigandham 7y ago
      HN is a great resource with plenty of important news posted, along with the HighScalability blog [1] and SoftwareEngineeringDaily podcast [2]. 1. http://highscalability.com/ http://highscalability.com/ 2. https://softwareengineeringdaily.com/ https://softwareengineeringdaily.com/ Other than that, reading the blogs of the various vendors is how I keep up (using Feedly with RSS). The modern projects are Redshift, BigQuery, Snowflake, Azure DW, MemSQL, Clickhouse, YellowBrick with older projects being Vertica, Teradata, Greenplum. It's also useful to follow the "new" distributed SQL projects like CockroachDB, Citus, TiDB, Vitess, Yugabyte.
      - sanderjd 7y ago
        Thank you this is super useful to me!
  - qiuyesuifeng 7y ago
    An architecture of having both row-based database and column-based data might also be interesting to explore. This post explains how they work together. https://pingcap.com/blog/delivering-real-time-analytics-and-true-htap-by-combining-columnstore-and-rowstore/ https://pingcap.com/blog/delivering-real-time-analytics-and-...

sanderjd 7y ago

Question for everyone in this thread: what resources do you use to follow this stuff? Are there blogs or conferences or ... that I should follow if I'm interested in this topic?

manigandham 7y ago
Distributed SQL data warehouses? Start with the papers: - C-Store (column-oriented database storage and processing): http://db.lcs.mit.edu/projects/cstore/vldb.pdf http://db.lcs.mit.edu/projects/cstore/vldb.pdf and https://sookocheff.com/post/databases/c-store/ https://sookocheff.com/post/databases/c-store/ - BigQuery (the original Dremel-based query engine): https://cloud.google.com/files/BigQueryTechnicalWP.pdf https://cloud.google.com/files/BigQueryTechnicalWP.pdf - Snowflake paper: https://www.snowflake.com/resource/sigmod-2016-paper-snowflake-elastic-data-warehouse/ https://www.snowflake.com/resource/sigmod-2016-paper-snowfla... - Andy Pavlo's CMU distributed OLAP databases lecture: https://www.youtube.com/watch?v=dPMc7FZ3Gqo&t=1s https://www.youtube.com/watch?v=dPMc7FZ3Gqo&t=1s - MemSQL architecture: https://www.youtube.com/watch?v=HoompXCG5Mk https://www.youtube.com/watch?v=HoompXCG5Mk Start there and just read more about architecture of the various products. You'll learn the basics of distributed nodes that store data in column-oriented tables partitioned by a key and how to run queries over them using various performance techniques. Then you can dive into the unique differences of the various databases.
- polskibus 7y ago
  I recommend watching Andy Pavlo's courses: https://15445.courses.cs.cmu.edu/fall2019/ https://15445.courses.cs.cmu.edu/fall2019/ https://15721.courses.cs.cmu.edu/spring2020/ https://15721.courses.cs.cmu.edu/spring2020/
- sanderjd 7y ago
  No sorry, I didn't mean where to read about how these things are designed (though these are useful and interesting links for that too, thanks!). What I'm asking about is what to follow (blogs, conferences, etc.) to keep up with new developments in the space: new products, experiences with them, reviews of their trade offs in practical use, that kind of thing. But I do appreciate the response!
  - manigandham 7y ago
    HN is a great resource with plenty of important news posted, along with the HighScalability blog [1] and SoftwareEngineeringDaily podcast [2]. 1. http://highscalability.com/ http://highscalability.com/ 2. https://softwareengineeringdaily.com/ https://softwareengineeringdaily.com/ Other than that, reading the blogs of the various vendors is how I keep up (using Feedly with RSS). The modern projects are Redshift, BigQuery, Snowflake, Azure DW, MemSQL, Clickhouse, YellowBrick with older projects being Vertica, Teradata, Greenplum. It's also useful to follow the "new" distributed SQL projects like CockroachDB, Citus, TiDB, Vitess, Yugabyte.
    - sanderjd 7y ago
      Thank you this is super useful to me!
- qiuyesuifeng 7y ago
  An architecture of having both row-based database and column-based data might also be interesting to explore. This post explains how they work together. https://pingcap.com/blog/delivering-real-time-analytics-and-true-htap-by-combining-columnstore-and-rowstore/ https://pingcap.com/blog/delivering-real-time-analytics-and-...