3 ms·

More options dilute talent and stifle innovation - people spend time writing new tools instead of advancing state of the art But by all means, it's not my mone

by jjirsa 10y ago

More options dilute talent and stifle innovation - people spend time writing new tools instead of advancing state of the art But by all means, it's not my money being burned, so I'm not losing sleep over it. I'll just sit here and giggle as people talk about scaling Postgres and in my mind all I hear is single point of failure masters and vacuum to prevent wraparound in high write workflows

enordstr 10y ago
Sounds like you’re main concern is high availability (HA) and scaling, things Cassandra certainly does well. But if you are also interested in data exploration and complex queries, Cassandra might not be the best choice. I think there is certainly a part of the spectrum here that is not well addressed by something like Cassandra. A future clustering version of TimescaleDB will provide horizontal scale out and solutions for HA, btw. Also, regarding vacuuming, TimescaleDB does not suffer from it much under the normal workloads it targets because time-series data is largely immutable and deletions typically happen at the chunk level (i.e., dropping complete tables), avoiding any vacuuming during deletes and making wraparound vacuuming more efficient.
- jjirsa 10y ago
  A future clustering version of TimescaleDB will reinvent problems already solved by Cassandra, Riak, and similar databases, and the level of effort in doing so will be far greater than the level of effort needed to add data exploration and complex queries on top of cassandra in the first place (that is: clustering is the hard part, and had this team spent time to bolt the analytics logic onto cassandra similar to what FiloDB did, they'd have the best of both worlds already). Wraparound vacuuming still sucks for high write workloads. I've been there. I've fought that problem in a high-write-throughput-no-delete-immutable-workload. I've seen it in person. You're still writing a lot of frozen txids to disk. Your slaves are still going to get the WAL command in a single-threaded WAL sender and fall behind in replication as that vacuum runs. You're still going to have pain trying to create an HA setup.
  - cevian 10y ago
    Vacuuming has traditionally been a problem with large table sizes. In TimescaleDB we break up the tables so that they are smaller. That, combined with the new freeze map feature in Postgres (since PG 9.6: http://rhaas.blogspot.jp/2016/03/no-more-full-table-vacuums.html http://rhaas.blogspot.jp/2016/03/no-more-full-table-vacuums....) make vacuums not an issue for us. Certainly we've never seen this issue (even on default autovacuum settings) and we've tested some huge datasets.
manigandham 10y ago
Yes, but people work on what they choose to and it's not possible to say putting them all on the same team/project would move that forward - or that they would even want to do that. Does it waste a certain amount of effort in duplicated work and products? Sure. But we also get new projects that may try new approaches, gain different experiences, and train new developers which all helps the market expand. Ultimately it's better to have yet another version than nothing at all.

by jjirsa 10y ago

enordstr 10y ago

Sounds like you’re main concern is high availability (HA) and scaling, things Cassandra certainly does well. But if you are also interested in data exploration and complex queries, Cassandra might not be the best choice. I think there is certainly a part of the spectrum here that is not well addressed by something like Cassandra. A future clustering version of TimescaleDB will provide horizontal scale out and solutions for HA, btw. Also, regarding vacuuming, TimescaleDB does not suffer from it much under the normal workloads it targets because time-series data is largely immutable and deletions typically happen at the chunk level (i.e., dropping complete tables), avoiding any vacuuming during deletes and making wraparound vacuuming more efficient.

jjirsa 10y ago
A future clustering version of TimescaleDB will reinvent problems already solved by Cassandra, Riak, and similar databases, and the level of effort in doing so will be far greater than the level of effort needed to add data exploration and complex queries on top of cassandra in the first place (that is: clustering is the hard part, and had this team spent time to bolt the analytics logic onto cassandra similar to what FiloDB did, they'd have the best of both worlds already). Wraparound vacuuming still sucks for high write workloads. I've been there. I've fought that problem in a high-write-throughput-no-delete-immutable-workload. I've seen it in person. You're still writing a lot of frozen txids to disk. Your slaves are still going to get the WAL command in a single-threaded WAL sender and fall behind in replication as that vacuum runs. You're still going to have pain trying to create an HA setup.
- cevian 10y ago
  Vacuuming has traditionally been a problem with large table sizes. In TimescaleDB we break up the tables so that they are smaller. That, combined with the new freeze map feature in Postgres (since PG 9.6: http://rhaas.blogspot.jp/2016/03/no-more-full-table-vacuums.html http://rhaas.blogspot.jp/2016/03/no-more-full-table-vacuums....) make vacuums not an issue for us. Certainly we've never seen this issue (even on default autovacuum settings) and we've tested some huge datasets.

manigandham 10y ago

Yes, but people work on what they choose to and it's not possible to say putting them all on the same team/project would move that forward - or that they would even want to do that. Does it waste a certain amount of effort in duplicated work and products? Sure. But we also get new projects that may try new approaches, gain different experiences, and train new developers which all helps the market expand. Ultimately it's better to have yet another version than nothing at all.