5 ms·

Citus seems really cool (this is not a critique of the product), but probably not something I’d use. They seem to be very close in feature set to bigquery, and

Citus seems really cool (this is not a critique of the product), but probably not something I’d use. They seem to be very close in feature set to bigquery, and if I have to choose between the two bigquery wins for me. There are definitely situations where citus is a better solution, but I think the scenarios where I could need a scale out analytics platform, I’d choose BQ. Still, very supportive of what they are doing at Citus. It’s great.

manigandham 9y ago
MemSQL would be the on-premise alternative to BigQuery, RedShift, Snowflake and the others. Citus is more focused on scalable OLTP rather than OLAP scenarios, giving you essentially an automatically sharding postgres database. Heap does use it for analytics but that's more so because of the JSON support and indexing in postgres itself.
- wjossey 9y ago
  I’m not sure I agree with this statement. BigQuery is not in memory, nor is redshift (not super familiar with snowflake). Whereas memsql is and would be prohibitively expensive to operate on datasets that are analytics based, because it is in memory based. It’s my understanding that citus is best suited to analytics workloads and parallel queries, and really isn’t comparable to memsql as it is attempting to provide scale out support with disk, which memsql doesn’t natively prefer. Can you share some docs with me around how citus is more OLTP based? Looking to be educated.
  - manigandham 9y ago
    MemSQL has in-memory rowstores and disk-based columnstores. Every columnstore table has a rowstore to aid in fast data ingest. Rowstores are meant for access of individual rows, lots of data updates, and smaller reference tables while columnstores are for the bulk of data requiring large scans and analysis. We have tables with 20 billion rows that take up < 10gb of memory. BigQuery streaming ingest also uses in-memory buffering into BigTable until it's written to disk, just like Cassandra with memtables flushed into on-disk SSTables. Citus is more manual work than MemSQL and doesnt support columnstores native, although you can use it with their cstore extension to get both. It's just a different design giving you a postgres database that automatically shards vs MemSQL's many optimizations around HA, replication, and very fast analytics performance. Try the features page: http://www.memsql.com/features/ http://www.memsql.com/features/
    - wjossey 9y ago
      Thanks for the education on MemSQL. I wasn’t aware of the analytics use case (had never come up in discussions I was a part of for that tech), so it was news to me. This article had a nice overview. Somewhat outdated but mostly still up to date. https://lnkd.in/g4anCf5 https://lnkd.in/g4anCf5 I’ll say it seems precarious to me to mix and match row store in memory data with a columnar store. Is it never a performance issue to have these two systems potentially in contention with one another for resources?
      - manigandham 9y ago
        It's just data in memory. Rowstores are read from memory and columnstores are read from disk. The architecture uses leaf nodes and aggregator nodes so aggregrator memory is used for assembling the final results of a query. Version 6 is even faster, you can try out the beta: http://www1.memsql.com/beta-6.3-features.html http://www1.memsql.com/beta-6.3-features.html
  - ddorian43 9y ago
    Every BigQuery query, actually loads all the data (for that query) in-memory until it finishes.

manigandham 9y ago

MemSQL would be the on-premise alternative to BigQuery, RedShift, Snowflake and the others. Citus is more focused on scalable OLTP rather than OLAP scenarios, giving you essentially an automatically sharding postgres database. Heap does use it for analytics but that's more so because of the JSON support and indexing in postgres itself.

wjossey 9y ago
I’m not sure I agree with this statement. BigQuery is not in memory, nor is redshift (not super familiar with snowflake). Whereas memsql is and would be prohibitively expensive to operate on datasets that are analytics based, because it is in memory based. It’s my understanding that citus is best suited to analytics workloads and parallel queries, and really isn’t comparable to memsql as it is attempting to provide scale out support with disk, which memsql doesn’t natively prefer. Can you share some docs with me around how citus is more OLTP based? Looking to be educated.
- manigandham 9y ago
  MemSQL has in-memory rowstores and disk-based columnstores. Every columnstore table has a rowstore to aid in fast data ingest. Rowstores are meant for access of individual rows, lots of data updates, and smaller reference tables while columnstores are for the bulk of data requiring large scans and analysis. We have tables with 20 billion rows that take up < 10gb of memory. BigQuery streaming ingest also uses in-memory buffering into BigTable until it's written to disk, just like Cassandra with memtables flushed into on-disk SSTables. Citus is more manual work than MemSQL and doesnt support columnstores native, although you can use it with their cstore extension to get both. It's just a different design giving you a postgres database that automatically shards vs MemSQL's many optimizations around HA, replication, and very fast analytics performance. Try the features page: http://www.memsql.com/features/ http://www.memsql.com/features/
  - wjossey 9y ago
    Thanks for the education on MemSQL. I wasn’t aware of the analytics use case (had never come up in discussions I was a part of for that tech), so it was news to me. This article had a nice overview. Somewhat outdated but mostly still up to date. https://lnkd.in/g4anCf5 https://lnkd.in/g4anCf5 I’ll say it seems precarious to me to mix and match row store in memory data with a columnar store. Is it never a performance issue to have these two systems potentially in contention with one another for resources?
    - manigandham 9y ago
      It's just data in memory. Rowstores are read from memory and columnstores are read from disk. The architecture uses leaf nodes and aggregator nodes so aggregrator memory is used for assembling the final results of a query. Version 6 is even faster, you can try out the beta: http://www1.memsql.com/beta-6.3-features.html http://www1.memsql.com/beta-6.3-features.html
- ddorian43 9y ago
  Every BigQuery query, actually loads all the data (for that query) in-memory until it finishes.