8 ms·

Author here. Thanks for reading! As well as these DBs, we also evaluated Kafka and a document database (CouchDB, it seemed awful). For messaging, the Commit Lo

Author here. Thanks for reading! As well as these DBs, we also evaluated Kafka and a document database (CouchDB, it seemed awful). For messaging, the Commit Log and Log-Structured Merge Tree storage in Cassandra should have been near ideal. Commit Log gives durability, LSMT gives automatic compaction -- important for the pending work store, where you delete entries once work is completed. The domain here is healthcare; due to legal requirements & existing product architecture, deployments and databases are usually per-customer rather than trying to scale the world on a single cluster. For these scenarios Postgres and Cassandra should both be (have been) highly suitable. The Postgres ecosystem is very active these days, and offer options for further scaling. Aligning with Postgres already looks like a 5x scalability improvement over the prior datastore, and AWS Aurora and Citus Data both offer avenues for further performance if we ever needed.

hardwaresofton 5y ago
Thanks for writing the blog post -- loved the deep dive and investigation bits. > (CouchDB, it seemed awful). Yeah I just... am not sure about picking document stores these days! I was the biggest fan of RethinkDB (over Mongo) and I always get CouchDB and CouchBase mixed up. At the end of the day, I almost always just choose to store some JSONB in Postgres. > For messaging, the Commit Log and Log-Structured Merge Tree storage in Cassandra should have been near ideal. Commit Log gives durability, LSMT gives automatic compaction -- important for the pending work store, where you delete entries once work is completed. > The domain here is healthcare; due to legal requirements & existing product architecture, deployments and databases are usually per-customer rather than trying to scale the world on a single cluster. For these scenarios Postgres and Cassandra should both be (have been) highly suitable. Ahhh this makes a lot more sense now. So hard multi-tenancy + needs to be pre-approved. > The Postgres ecosystem is very active these days, and offer options for further scaling. Aligning with Postgres already looks like a 5x scalability improvement over the prior datastore, and AWS Aurora and Citus Data both offer avenues for further performance if we ever needed. I'm an unabashed Postgres shill I agree -- Postgres is the absolutely the most flexible database to build on, you can do OLAP, OLTP, partitioning and sharding (Partman/Timescale/Citus), and zheap is in the pipe (so MySQL's usual raison d'être is gone). Enough shilling though, one thing I wonder about in this space -- are you planning on using ZFS at the bottom of any of these systems? With the recent spate of data ransoming attacks and hacks I wonder why people "don't just" (famous first words) put ZFS down there, add some replication + offsite incremental backups and sleep a bit easier at night?
PeterCorless 5y ago
I was going to ask: was running in the Windows OS a requirement? I was curious because of the prevalence of Linux in back-end server environments. Did you try the same setup as you had on Windows with a Linux distro?

hardwaresofton 5y ago

Thanks for writing the blog post -- loved the deep dive and investigation bits. > (CouchDB, it seemed awful). Yeah I just... am not sure about picking document stores these days! I was the biggest fan of RethinkDB (over Mongo) and I always get CouchDB and CouchBase mixed up. At the end of the day, I almost always just choose to store some JSONB in Postgres. > For messaging, the Commit Log and Log-Structured Merge Tree storage in Cassandra should have been near ideal. Commit Log gives durability, LSMT gives automatic compaction -- important for the pending work store, where you delete entries once work is completed. > The domain here is healthcare; due to legal requirements & existing product architecture, deployments and databases are usually per-customer rather than trying to scale the world on a single cluster. For these scenarios Postgres and Cassandra should both be (have been) highly suitable. Ahhh this makes a lot more sense now. So hard multi-tenancy + needs to be pre-approved. > The Postgres ecosystem is very active these days, and offer options for further scaling. Aligning with Postgres already looks like a 5x scalability improvement over the prior datastore, and AWS Aurora and Citus Data both offer avenues for further performance if we ever needed. I'm an unabashed Postgres shill I agree -- Postgres is the absolutely the most flexible database to build on, you can do OLAP, OLTP, partitioning and sharding (Partman/Timescale/Citus), and zheap is in the pipe (so MySQL's usual raison d'être is gone). Enough shilling though, one thing I wonder about in this space -- are you planning on using ZFS at the bottom of any of these systems? With the recent spate of data ransoming attacks and hacks I wonder why people "don't just" (famous first words) put ZFS down there, add some replication + offsite incremental backups and sleep a bit easier at night?

PeterCorless 5y ago

I was going to ask: was running in the Windows OS a requirement? I was curious because of the prevalence of Linux in back-end server environments. Did you try the same setup as you had on Windows with a Linux distro?