4 ms·

I’m not sure I agree with this statement. BigQuery is not in memory, nor is redshift (not super familiar with snowflake). Whereas memsql is and would be prohibi

I’m not sure I agree with this statement. BigQuery is not in memory, nor is redshift (not super familiar with snowflake). Whereas memsql is and would be prohibitively expensive to operate on datasets that are analytics based, because it is in memory based. It’s my understanding that citus is best suited to analytics workloads and parallel queries, and really isn’t comparable to memsql as it is attempting to provide scale out support with disk, which memsql doesn’t natively prefer. Can you share some docs with me around how citus is more OLTP based? Looking to be educated.

manigandham 9y ago
MemSQL has in-memory rowstores and disk-based columnstores. Every columnstore table has a rowstore to aid in fast data ingest. Rowstores are meant for access of individual rows, lots of data updates, and smaller reference tables while columnstores are for the bulk of data requiring large scans and analysis. We have tables with 20 billion rows that take up < 10gb of memory. BigQuery streaming ingest also uses in-memory buffering into BigTable until it's written to disk, just like Cassandra with memtables flushed into on-disk SSTables. Citus is more manual work than MemSQL and doesnt support columnstores native, although you can use it with their cstore extension to get both. It's just a different design giving you a postgres database that automatically shards vs MemSQL's many optimizations around HA, replication, and very fast analytics performance. Try the features page: http://www.memsql.com/features/ http://www.memsql.com/features/
- wjossey 9y ago
  Thanks for the education on MemSQL. I wasn’t aware of the analytics use case (had never come up in discussions I was a part of for that tech), so it was news to me. This article had a nice overview. Somewhat outdated but mostly still up to date. https://lnkd.in/g4anCf5 https://lnkd.in/g4anCf5 I’ll say it seems precarious to me to mix and match row store in memory data with a columnar store. Is it never a performance issue to have these two systems potentially in contention with one another for resources?
  - manigandham 9y ago
    It's just data in memory. Rowstores are read from memory and columnstores are read from disk. The architecture uses leaf nodes and aggregator nodes so aggregrator memory is used for assembling the final results of a query. Version 6 is even faster, you can try out the beta: http://www1.memsql.com/beta-6.3-features.html http://www1.memsql.com/beta-6.3-features.html
ddorian43 9y ago
Every BigQuery query, actually loads all the data (for that query) in-memory until it finishes.

manigandham 9y ago

MemSQL has in-memory rowstores and disk-based columnstores. Every columnstore table has a rowstore to aid in fast data ingest. Rowstores are meant for access of individual rows, lots of data updates, and smaller reference tables while columnstores are for the bulk of data requiring large scans and analysis. We have tables with 20 billion rows that take up < 10gb of memory. BigQuery streaming ingest also uses in-memory buffering into BigTable until it's written to disk, just like Cassandra with memtables flushed into on-disk SSTables. Citus is more manual work than MemSQL and doesnt support columnstores native, although you can use it with their cstore extension to get both. It's just a different design giving you a postgres database that automatically shards vs MemSQL's many optimizations around HA, replication, and very fast analytics performance. Try the features page: http://www.memsql.com/features/ http://www.memsql.com/features/

wjossey 9y ago
Thanks for the education on MemSQL. I wasn’t aware of the analytics use case (had never come up in discussions I was a part of for that tech), so it was news to me. This article had a nice overview. Somewhat outdated but mostly still up to date. https://lnkd.in/g4anCf5 https://lnkd.in/g4anCf5 I’ll say it seems precarious to me to mix and match row store in memory data with a columnar store. Is it never a performance issue to have these two systems potentially in contention with one another for resources?
- manigandham 9y ago
  It's just data in memory. Rowstores are read from memory and columnstores are read from disk. The architecture uses leaf nodes and aggregator nodes so aggregrator memory is used for assembling the final results of a query. Version 6 is even faster, you can try out the beta: http://www1.memsql.com/beta-6.3-features.html http://www1.memsql.com/beta-6.3-features.html

ddorian43 9y ago

Every BigQuery query, actually loads all the data (for that query) in-memory until it finishes.