4 ms·

At my last company we did tens of billions of sensor events per day into Cassandra and I thank god our engineering team was smart enough to spend manpower on pr

by jjirsa 10y ago

At my last company we did tens of billions of sensor events per day into Cassandra and I thank god our engineering team was smart enough to spend manpower on product instead of writing yet another database

manigandham 10y ago
So, good for your past company? What happens when your primary product is the database? More options are never a bad thing. If there's a downside here, it's that this database is still a little too immature to have a great use-case.
- jjirsa 10y ago
  More options dilute talent and stifle innovation - people spend time writing new tools instead of advancing state of the art But by all means, it's not my money being burned, so I'm not losing sleep over it. I'll just sit here and giggle as people talk about scaling Postgres and in my mind all I hear is single point of failure masters and vacuum to prevent wraparound in high write workflows
  - enordstr 10y ago
    Sounds like you’re main concern is high availability (HA) and scaling, things Cassandra certainly does well. But if you are also interested in data exploration and complex queries, Cassandra might not be the best choice. I think there is certainly a part of the spectrum here that is not well addressed by something like Cassandra. A future clustering version of TimescaleDB will provide horizontal scale out and solutions for HA, btw. Also, regarding vacuuming, TimescaleDB does not suffer from it much under the normal workloads it targets because time-series data is largely immutable and deletions typically happen at the chunk level (i.e., dropping complete tables), avoiding any vacuuming during deletes and making wraparound vacuuming more efficient.
    - jjirsa 10y ago
      A future clustering version of TimescaleDB will reinvent problems already solved by Cassandra, Riak, and similar databases, and the level of effort in doing so will be far greater than the level of effort needed to add data exploration and complex queries on top of cassandra in the first place (that is: clustering is the hard part, and had this team spent time to bolt the analytics logic onto cassandra similar to what FiloDB did, they'd have the best of both worlds already). Wraparound vacuuming still sucks for high write workloads. I've been there. I've fought that problem in a high-write-throughput-no-delete-immutable-workload. I've seen it in person. You're still writing a lot of frozen txids to disk. Your slaves are still going to get the WAL command in a single-threaded WAL sender and fall behind in replication as that vacuum runs. You're still going to have pain trying to create an HA setup.
      - cevian 10y ago
        Vacuuming has traditionally been a problem with large table sizes. In TimescaleDB we break up the tables so that they are smaller. That, combined with the new freeze map feature in Postgres (since PG 9.6: http://rhaas.blogspot.jp/2016/03/no-more-full-table-vacuums.html http://rhaas.blogspot.jp/2016/03/no-more-full-table-vacuums....) make vacuums not an issue for us. Certainly we've never seen this issue (even on default autovacuum settings) and we've tested some huge datasets.
  - manigandham 10y ago
    Yes, but people work on what they choose to and it's not possible to say putting them all on the same team/project would move that forward - or that they would even want to do that. Does it waste a certain amount of effort in duplicated work and products? Sure. But we also get new projects that may try new approaches, gain different experiences, and train new developers which all helps the market expand. Ultimately it's better to have yet another version than nothing at all.
bigger_cheese 10y ago
One thing I'm curious about is what are the differences between an operational historian (like ABB, PI, Honeywell etc) and a time series database like Cassandra or the one in this article? I work a step upward in the stack (analyzing the sensor data and building models based off of it - usually using SQL) so I'm not an expert on underlying technology. Over the last 15 years or so my work (industrial plant) has been big on historians. We've moved from in house bespoke stuff from the 90's which as I understood it was basically a custom written ring buffer that held 1-2 days of live data and would get down sampled into oracle tables as 1min/ 6min/hourly averaged data every hour or so. Prior to this in the 80's we used a database technology called "IMS" but I'm not as familiar with how this system worked. Nowadays in all our newer systems all I hear from the low level guys is "OPC" (which I gather is a messaging format) and historians. Time series databases sound like they should fit in somewhere but I'm not sure where - given other people use them for logging high frequency sensor data sounds like they could be a solution.
- jjirsa 10y ago
  To be clear - Cassandra is great for time series but it's not a "time series database" - it just solves all of the same problems (and then some), because it gives you natural replication, real HA, cross-wan, let's you cluster/order data on disk using big table like sorting, and scales over a thousand nodes per cluster. For sensor data collection, it's very hard to beat. Lay something like spark or filodb on top for more flexibility and it's virtually unmatched.
happyweasel 10y ago
How many machines did you use ?? Not everyone can afford a cluster - and I do not think a single cassandra node could store 150000 entries per sec. Why didn't you use Scylla, which is 3x to 10x faster than Cassandra?
- jjirsa 10y ago
  A single cassandra node can probably do 100k writes/second. We ran hundreds of machines (per cluster), and multiple clusters. We don't use Scylla for dozens of reasons, including (but not limited to): - The product was written before Scylla existed - Our benchmarks didn't show Scylla as being 3x faster, let alone 10x faster - Our machine count wasn't based on Cassandra limits, it was based on the amount of data we were comfortable storing on a given machine - We don't throw shiny new tech into a critical stack just because someone says it's faster. I'd rather run code that's tested at scale. ONE of my Cassandra clusters there would have been 10x larger than any known Scylla deployment. At the same time, that deployment is at least 2 orders of magnitude smaller than other published Cassandra deployments (meaning largest known Cassandra deployments are 3 orders of magnitude larger than the largest known Scylla deployment) The one thing I agree with the author: databases aren't where you want to have hipster tech. You want boring things that work. For me, Cassandra is the boring thing that works.
  - ADefenestrator 10y ago
    I absolutely agree about hipster tech and databases, but I'm not so sure that Cassandra fits entirely into the "just works" category. Don't get me wrong, it's pretty neat and definitely more mature than stuff like Mongo or whatever other NoSQL-of-the-week, but it's not exactly at the level of PostgreSQL or even InnoDB/MySQL. It works, but it needs quite a bit of care and feeding by comparison.
  - sharadov 10y ago
    Really Cassandra? No way does it compare to mature RDBMS's like Postgres, MySQL or SQL server.

by jjirsa 10y ago

manigandham 10y ago

So, good for your past company? What happens when your primary product is the database? More options are never a bad thing. If there's a downside here, it's that this database is still a little too immature to have a great use-case.

jjirsa 10y ago
More options dilute talent and stifle innovation - people spend time writing new tools instead of advancing state of the art But by all means, it's not my money being burned, so I'm not losing sleep over it. I'll just sit here and giggle as people talk about scaling Postgres and in my mind all I hear is single point of failure masters and vacuum to prevent wraparound in high write workflows
- enordstr 10y ago
  Sounds like you’re main concern is high availability (HA) and scaling, things Cassandra certainly does well. But if you are also interested in data exploration and complex queries, Cassandra might not be the best choice. I think there is certainly a part of the spectrum here that is not well addressed by something like Cassandra. A future clustering version of TimescaleDB will provide horizontal scale out and solutions for HA, btw. Also, regarding vacuuming, TimescaleDB does not suffer from it much under the normal workloads it targets because time-series data is largely immutable and deletions typically happen at the chunk level (i.e., dropping complete tables), avoiding any vacuuming during deletes and making wraparound vacuuming more efficient.
  - jjirsa 10y ago
    A future clustering version of TimescaleDB will reinvent problems already solved by Cassandra, Riak, and similar databases, and the level of effort in doing so will be far greater than the level of effort needed to add data exploration and complex queries on top of cassandra in the first place (that is: clustering is the hard part, and had this team spent time to bolt the analytics logic onto cassandra similar to what FiloDB did, they'd have the best of both worlds already). Wraparound vacuuming still sucks for high write workloads. I've been there. I've fought that problem in a high-write-throughput-no-delete-immutable-workload. I've seen it in person. You're still writing a lot of frozen txids to disk. Your slaves are still going to get the WAL command in a single-threaded WAL sender and fall behind in replication as that vacuum runs. You're still going to have pain trying to create an HA setup.
    - cevian 10y ago
      Vacuuming has traditionally been a problem with large table sizes. In TimescaleDB we break up the tables so that they are smaller. That, combined with the new freeze map feature in Postgres (since PG 9.6: http://rhaas.blogspot.jp/2016/03/no-more-full-table-vacuums.html http://rhaas.blogspot.jp/2016/03/no-more-full-table-vacuums....) make vacuums not an issue for us. Certainly we've never seen this issue (even on default autovacuum settings) and we've tested some huge datasets.
- manigandham 10y ago
  Yes, but people work on what they choose to and it's not possible to say putting them all on the same team/project would move that forward - or that they would even want to do that. Does it waste a certain amount of effort in duplicated work and products? Sure. But we also get new projects that may try new approaches, gain different experiences, and train new developers which all helps the market expand. Ultimately it's better to have yet another version than nothing at all.

bigger_cheese 10y ago

One thing I'm curious about is what are the differences between an operational historian (like ABB, PI, Honeywell etc) and a time series database like Cassandra or the one in this article? I work a step upward in the stack (analyzing the sensor data and building models based off of it - usually using SQL) so I'm not an expert on underlying technology. Over the last 15 years or so my work (industrial plant) has been big on historians. We've moved from in house bespoke stuff from the 90's which as I understood it was basically a custom written ring buffer that held 1-2 days of live data and would get down sampled into oracle tables as 1min/ 6min/hourly averaged data every hour or so. Prior to this in the 80's we used a database technology called "IMS" but I'm not as familiar with how this system worked. Nowadays in all our newer systems all I hear from the low level guys is "OPC" (which I gather is a messaging format) and historians. Time series databases sound like they should fit in somewhere but I'm not sure where - given other people use them for logging high frequency sensor data sounds like they could be a solution.

jjirsa 10y ago
To be clear - Cassandra is great for time series but it's not a "time series database" - it just solves all of the same problems (and then some), because it gives you natural replication, real HA, cross-wan, let's you cluster/order data on disk using big table like sorting, and scales over a thousand nodes per cluster. For sensor data collection, it's very hard to beat. Lay something like spark or filodb on top for more flexibility and it's virtually unmatched.

happyweasel 10y ago

How many machines did you use ?? Not everyone can afford a cluster - and I do not think a single cassandra node could store 150000 entries per sec. Why didn't you use Scylla, which is 3x to 10x faster than Cassandra?

jjirsa 10y ago
A single cassandra node can probably do 100k writes/second. We ran hundreds of machines (per cluster), and multiple clusters. We don't use Scylla for dozens of reasons, including (but not limited to): - The product was written before Scylla existed - Our benchmarks didn't show Scylla as being 3x faster, let alone 10x faster - Our machine count wasn't based on Cassandra limits, it was based on the amount of data we were comfortable storing on a given machine - We don't throw shiny new tech into a critical stack just because someone says it's faster. I'd rather run code that's tested at scale. ONE of my Cassandra clusters there would have been 10x larger than any known Scylla deployment. At the same time, that deployment is at least 2 orders of magnitude smaller than other published Cassandra deployments (meaning largest known Cassandra deployments are 3 orders of magnitude larger than the largest known Scylla deployment) The one thing I agree with the author: databases aren't where you want to have hipster tech. You want boring things that work. For me, Cassandra is the boring thing that works.
- ADefenestrator 10y ago
  I absolutely agree about hipster tech and databases, but I'm not so sure that Cassandra fits entirely into the "just works" category. Don't get me wrong, it's pretty neat and definitely more mature than stuff like Mongo or whatever other NoSQL-of-the-week, but it's not exactly at the level of PostgreSQL or even InnoDB/MySQL. It works, but it needs quite a bit of care and feeding by comparison.
- sharadov 10y ago
  Really Cassandra? No way does it compare to mature RDBMS's like Postgres, MySQL or SQL server.