16 ms·

I don't think we disagree, but I want to be cautious about saying "whatever we start with is always good enough." I'm co-founder of a company, doing all of the

I don't think we disagree, but I want to be cautious about saying "whatever we start with is always good enough." I'm co-founder of a company, doing all of the technical work, and I too am using a very vanilla stack: Postgres, Elasticsearch, Redis, Memcache, Rails. However, having worked on massive scale-out projects before, I recognize where my pain points are going to be with my tech choices. [1] I'm storing calendar data in postgres. This data is primarily key/value based, needs to be persistent, and on a per user basis potentially is tens of thousands of records. This is a medium term problem, and I'll likely move this data out of postgres and into a scale out solution sooner rather than later. I need relatively fast lookup times, but this is primarily a "high write, low read" scenario, that Cassandra is well suited for. [2] I'm storing a bunch of text from blog posts, news articles, etc., in Postgres. I really dont' need the data to be in postgres, it's just a handy place to store it to synchronize with Algolia & Elasticsearch, where the true work happens. However, I dont' want to hold onto TBs & TBs of news articles in a postgres system. As that system scales out, it'll be unnecessarily expensive, as I don't need all the bells and whistles I'm getting from Postgres, for what is effectively a form of cold storage. This is a long term problem, and I likely won't tackle it for well over a year. So, you're right, I'm not contemplating what DB to use, but I'm definitely aware of why I won't just leveraging the same database in perpetuity. Being focused on delivering customer value is vital, and that's my intent; however, scalability and cost structure do matter over the long term, so an understanding of where one will go is good as well.

noncoml 9y ago
Agreed
simonw 9y ago
Have you considered Citus as your scale-out solution? https://www.citusdata.com https://www.citusdata.com - maybe you can have your PostgreSQL and eat it too.
- wjossey 9y ago
  Citus seems really cool (this is not a critique of the product), but probably not something I’d use. They seem to be very close in feature set to bigquery, and if I have to choose between the two bigquery wins for me. There are definitely situations where citus is a better solution, but I think the scenarios where I could need a scale out analytics platform, I’d choose BQ. Still, very supportive of what they are doing at Citus. It’s great.
  - manigandham 9y ago
    MemSQL would be the on-premise alternative to BigQuery, RedShift, Snowflake and the others. Citus is more focused on scalable OLTP rather than OLAP scenarios, giving you essentially an automatically sharding postgres database. Heap does use it for analytics but that's more so because of the JSON support and indexing in postgres itself.
    - wjossey 9y ago
      I’m not sure I agree with this statement. BigQuery is not in memory, nor is redshift (not super familiar with snowflake). Whereas memsql is and would be prohibitively expensive to operate on datasets that are analytics based, because it is in memory based. It’s my understanding that citus is best suited to analytics workloads and parallel queries, and really isn’t comparable to memsql as it is attempting to provide scale out support with disk, which memsql doesn’t natively prefer. Can you share some docs with me around how citus is more OLTP based? Looking to be educated.
      - manigandham 9y ago
        MemSQL has in-memory rowstores and disk-based columnstores. Every columnstore table has a rowstore to aid in fast data ingest. Rowstores are meant for access of individual rows, lots of data updates, and smaller reference tables while columnstores are for the bulk of data requiring large scans and analysis. We have tables with 20 billion rows that take up < 10gb of memory. BigQuery streaming ingest also uses in-memory buffering into BigTable until it's written to disk, just like Cassandra with memtables flushed into on-disk SSTables. Citus is more manual work than MemSQL and doesnt support columnstores native, although you can use it with their cstore extension to get both. It's just a different design giving you a postgres database that automatically shards vs MemSQL's many optimizations around HA, replication, and very fast analytics performance. Try the features page: http://www.memsql.com/features/ http://www.memsql.com/features/

noncoml 9y ago

Agreed

simonw 9y ago

Have you considered Citus as your scale-out solution? https://www.citusdata.com https://www.citusdata.com - maybe you can have your PostgreSQL and eat it too.

wjossey 9y ago
Citus seems really cool (this is not a critique of the product), but probably not something I’d use. They seem to be very close in feature set to bigquery, and if I have to choose between the two bigquery wins for me. There are definitely situations where citus is a better solution, but I think the scenarios where I could need a scale out analytics platform, I’d choose BQ. Still, very supportive of what they are doing at Citus. It’s great.
- manigandham 9y ago
  MemSQL would be the on-premise alternative to BigQuery, RedShift, Snowflake and the others. Citus is more focused on scalable OLTP rather than OLAP scenarios, giving you essentially an automatically sharding postgres database. Heap does use it for analytics but that's more so because of the JSON support and indexing in postgres itself.
  - wjossey 9y ago
    I’m not sure I agree with this statement. BigQuery is not in memory, nor is redshift (not super familiar with snowflake). Whereas memsql is and would be prohibitively expensive to operate on datasets that are analytics based, because it is in memory based. It’s my understanding that citus is best suited to analytics workloads and parallel queries, and really isn’t comparable to memsql as it is attempting to provide scale out support with disk, which memsql doesn’t natively prefer. Can you share some docs with me around how citus is more OLTP based? Looking to be educated.
    - manigandham 9y ago
      MemSQL has in-memory rowstores and disk-based columnstores. Every columnstore table has a rowstore to aid in fast data ingest. Rowstores are meant for access of individual rows, lots of data updates, and smaller reference tables while columnstores are for the bulk of data requiring large scans and analysis. We have tables with 20 billion rows that take up < 10gb of memory. BigQuery streaming ingest also uses in-memory buffering into BigTable until it's written to disk, just like Cassandra with memtables flushed into on-disk SSTables. Citus is more manual work than MemSQL and doesnt support columnstores native, although you can use it with their cstore extension to get both. It's just a different design giving you a postgres database that automatically shards vs MemSQL's many optimizations around HA, replication, and very fast analytics performance. Try the features page: http://www.memsql.com/features/ http://www.memsql.com/features/