6 ms·

Always-On Time-Series Database: Keeping Up Where There's No Way to Catch Up

57 points by MindGods 6y ago

amenod 6y ago
> We've witnessed more than exponential growth in the volume and breadth of that data. In fact, by our estimate, we've seen an increase by a factor of about 1x10^12 over the past decade. If 10 years ago we had X data collected, now it is 1,000,000,000,000 * X data??? I find this highly surprising and wonder if it is a typo, and if so, what the real numbers are.
- jandrewrogers 6y ago
  I can't speak for Theo's number (it does seem high) but in my own experience working on similar data models, the typical machine-generated data model size roughly doubles every 6 months on average and has been growing at this rate for two decades. This implies that average data models today are a million times larger than data models a decade ago. Exabyte scale working data models have been something I've needed to consider in designs for at least a few years. We are surprisingly close to overflowing 64-bit integers in real systems.
  - amenod 6y ago
    Interesting - thank you for the insight!
maxmcd 6y ago
> But the real advantage of the CRDT approach is that, if you can limit your entire operation set to CRDTs, you can forego consensus algorithms such as Paxos, Multi-Paxos, Fast Paxos, Raft, Extended Virtual Synchrony, and anything else along those lines. Things like this are said a lot, but I don't believe CRDTs provide the same consistency guarantees as Paxos/Raft. Anyone have thoughts on why CRDTs might be favored over the internals of something like a well written eventually consistent datastore like Cassandra that doesn't use Paxos/Raft? Is he just saying that your consensus algorithm then just becomes that math of CRDTs and not the careful complexity of a distributed consensus protocol? (the interviewee then goes on to describe that CRDTs fit their use case well, so maybe that is all he is saying...)
- tracnar 6y ago
  From the article: > TS It's certainly the case that a lot of CRDTs are really complicated, especially in those instances where they represent some sort of complex interrelated state. A classic example would be the CRDTs used for document editing, string interjection, and that sort of thing. But the vast majority of machine-generated data is of the write-once, delete-never, update-never, append-only variety. That's the type of data yielded by the idempotent transactions that occur when a device measures what something looked like at one particular point in time. It's this element of idempotency in machine-generated data that really lends itself to the use of simplistic CRDTs. So for this particular use case, the CRDT is simple and they want to favor availability over consistency. I'm not sure if they provide further guarantees, e.g. from the description in the article it seems that temporary holes in the time series would be allowed if updates would arrive out-of-order. The article also mentions Cassandra, it seems they went for a different design so progress is more quickly visible in the DB (among other reasons I suppose). > What we wanted was a topology that looked similar to consistent hashing databases like DynamoDB or Riak or Cassandra, but we also wanted to make some minor adjustments, and we wanted all of the data types to be CRDTs [conflict-free replicated data types]. We ended up building a CRDT-exclusive database. That radically changes what is possible, specifically around how you make progress writing to the database.
- jandrewrogers 6y ago
  CRDTs work well for the specific case of machine-generated time-series because your records are samples. There are no complex relationships between records even for the same source. Gaps in the time series are like missing pixels in a picture, there is no specific pixel that is critical as long as you have enough of them to figure out what you are looking at in the moment. This also implies that there are no semantically meaningful update operations on specific records beyond converging values. Platforms like Cassandra can make some of the design choices they do because they don't support high write throughput (relatively). With sensor data models throughput and efficient use of bandwidth is everything, so minimizing the chattiness of ensuring consistency is critical.
sradman 6y ago
ACM Queue interview of Theo Schlossnagle, founder and CTO of Circonus, about IRONdb [1], an alternative time-series platform to Prometheus/InfluxDB. Some high-level architecture choices: - CRDT to avoid coordination service - LMDB (B-Tree) for read optimized requests - RocksDB (LSM-tree) for write optimized requests - Modified consistent hashing ring split across two Availability Zones - OpenZFS on Linux - Flatbuffers for Serialization-Deserialization - Information Lifecycle Management for historical data - circllhist; HDR [high dynamic range] log-linear quantized histograms [1] https://docs.circonus.com/irondb/ https://docs.circonus.com/irondb/

57 points by MindGods 6y ago

> We've witnessed more than exponential growth in the volume and breadth of that data. In fact, by our estimate, we've seen an increase by a factor of about 1x10^12 over the past decade. If 10 years ago we had X data collected, now it is 1,000,000,000,000 * X data??? I find this highly surprising and wonder if it is a typo, and if so, what the real numbers are.

jandrewrogers 6y ago
I can't speak for Theo's number (it does seem high) but in my own experience working on similar data models, the typical machine-generated data model size roughly doubles every 6 months on average and has been growing at this rate for two decades. This implies that average data models today are a million times larger than data models a decade ago. Exabyte scale working data models have been something I've needed to consider in designs for at least a few years. We are surprisingly close to overflowing 64-bit integers in real systems.
- amenod 6y ago
  Interesting - thank you for the insight!

maxmcd 6y ago

> But the real advantage of the CRDT approach is that, if you can limit your entire operation set to CRDTs, you can forego consensus algorithms such as Paxos, Multi-Paxos, Fast Paxos, Raft, Extended Virtual Synchrony, and anything else along those lines. Things like this are said a lot, but I don't believe CRDTs provide the same consistency guarantees as Paxos/Raft. Anyone have thoughts on why CRDTs might be favored over the internals of something like a well written eventually consistent datastore like Cassandra that doesn't use Paxos/Raft? Is he just saying that your consensus algorithm then just becomes that math of CRDTs and not the careful complexity of a distributed consensus protocol? (the interviewee then goes on to describe that CRDTs fit their use case well, so maybe that is all he is saying...)

tracnar 6y ago
From the article: > TS It's certainly the case that a lot of CRDTs are really complicated, especially in those instances where they represent some sort of complex interrelated state. A classic example would be the CRDTs used for document editing, string interjection, and that sort of thing. But the vast majority of machine-generated data is of the write-once, delete-never, update-never, append-only variety. That's the type of data yielded by the idempotent transactions that occur when a device measures what something looked like at one particular point in time. It's this element of idempotency in machine-generated data that really lends itself to the use of simplistic CRDTs. So for this particular use case, the CRDT is simple and they want to favor availability over consistency. I'm not sure if they provide further guarantees, e.g. from the description in the article it seems that temporary holes in the time series would be allowed if updates would arrive out-of-order. The article also mentions Cassandra, it seems they went for a different design so progress is more quickly visible in the DB (among other reasons I suppose). > What we wanted was a topology that looked similar to consistent hashing databases like DynamoDB or Riak or Cassandra, but we also wanted to make some minor adjustments, and we wanted all of the data types to be CRDTs [conflict-free replicated data types]. We ended up building a CRDT-exclusive database. That radically changes what is possible, specifically around how you make progress writing to the database.
jandrewrogers 6y ago
CRDTs work well for the specific case of machine-generated time-series because your records are samples. There are no complex relationships between records even for the same source. Gaps in the time series are like missing pixels in a picture, there is no specific pixel that is critical as long as you have enough of them to figure out what you are looking at in the moment. This also implies that there are no semantically meaningful update operations on specific records beyond converging values. Platforms like Cassandra can make some of the design choices they do because they don't support high write throughput (relatively). With sensor data models throughput and efficient use of bandwidth is everything, so minimizing the chattiness of ensuring consistency is critical.

sradman 6y ago

ACM Queue interview of Theo Schlossnagle, founder and CTO of Circonus, about IRONdb [1], an alternative time-series platform to Prometheus/InfluxDB. Some high-level architecture choices: - CRDT to avoid coordination service - LMDB (B-Tree) for read optimized requests - RocksDB (LSM-tree) for write optimized requests - Modified consistent hashing ring split across two Availability Zones - OpenZFS on Linux - Flatbuffers for Serialization-Deserialization - Information Lifecycle Management for historical data - circllhist; HDR [high dynamic range] log-linear quantized histograms [1] https://docs.circonus.com/irondb/ https://docs.circonus.com/irondb/