11 ms·

Poses the question So what’s the big deal? People have been recording temporally oriented data since we could chisel on tablets. Never answers it, but in

Poses the question So what’s the big deal? People have been recording temporally oriented data since we could chisel on tablets. Never answers it, but instead explains how Riak handles large time series. Certainly interesting, but I would like an answer to this question, as I don't understand the big deal.

deleted 10y ago
[deleted]
ttctciyf 10y ago
Immediately following the part you quote (my emphasis) Well, as it turns out, thanks to the software-eating- the-world thing and the Internet of Things we happen to be amassing *vast quantities* of all sorts of data [...] The demand for systems that are capable of storing and retrieving temporal data on an *ever increasing scale* necessitates systems that are specifically designed for this purpose. Strongly implying the big deal is the need to scale?
- siculars 10y ago
  Thanks, ya I thought I answered my question there ;)
- Confusion 10y ago
  That's an obstacle that needs to be overcome because you want to get somewhere. A challenging obstacle that has spawned an entire industry, but nevertheless not the goal. It is implied we can get somewhere new and exciting with these vastly larger timeseries. My question is: where?
jholman 10y ago
I think the big deal is allegedly "we now have so much data that we have a use for distributed databases with high performance", which allows them to go on and say "and our big deal is that we've built a high-performance distributed database tuned especially for time series". I only noticed it because I was shocked that there was an actual comprehensible value proposition in a blogpost about a NoSQL product.
- Twisell 10y ago
  However being system admin of a PostgreSQL database managing a lot of timeseries I was just like "meh use timestamp and good index, what's the deal?". Also for performance sake just cluster the data using the most used index... But yeah I guess if you really need NoSQL this should be nice. However sharding based on time will probably be one of the easiest approach for horizontal PostgreSQL deployment.
  - sitkack 10y ago
    One of the most compelling features of Riak, both KV and TS, is that it is masterless and replicated. When nodes fail for whatever reason, the cluster handles it transparently.
deleted 10y ago
[deleted]
adamneilson 10y ago
I think Datomic which has been doing time series data for years now answers the "what’s the big deal" question for me: "Given a value of the database, one can obtain another value of the database as-of, or since, a point in time in the past, or both (creating a windowed view of activity). These database values can be queried ordinarily, without having to make special queries parameterized by time." http://www.datomic.com/rationale.html http://www.datomic.com/rationale.html So in other words you get to see exactly how the whole database looked at any given moment through history (certain parallels with the blockchain I suppose).
- zcam 10y ago
  Until you reach 10B datoms and then you're off for a fun ride sharding with multiple databases. Also in an IoT context or more general timeseries usage you very likely don't care about tx, but very much do about write throughput and not having a spof, which Datomic isn't good at at all. Datomic is an ok choice in some contexts, but the one detailed here is not one of them, you're likely to reach Datomic limits quickly and be in a world of hurt when you do.
- assface 10y ago
  > I think Datomic which has been doing time series data for years now answers the "what’s the big deal" question for me: This is known as "time travel queries". Postgres supported this back in the original version from the 1980s but then they took it out in 1997 because it takes up too much disk space. It's trivial to do in an MVCC system. You just turn off garbage collection (vacuuming in PSQL parlance).

deleted 10y ago

[deleted]

ttctciyf 10y ago

Immediately following the part you quote (my emphasis) Well, as it turns out, thanks to the software-eating- the-world thing and the Internet of Things we happen to be amassing *vast quantities* of all sorts of data [...] The demand for systems that are capable of storing and retrieving temporal data on an *ever increasing scale* necessitates systems that are specifically designed for this purpose. Strongly implying the big deal is the need to scale?

siculars 10y ago
Thanks, ya I thought I answered my question there ;)
Confusion 10y ago
That's an obstacle that needs to be overcome because you want to get somewhere. A challenging obstacle that has spawned an entire industry, but nevertheless not the goal. It is implied we can get somewhere new and exciting with these vastly larger timeseries. My question is: where?

jholman 10y ago

I think the big deal is allegedly "we now have so much data that we have a use for distributed databases with high performance", which allows them to go on and say "and our big deal is that we've built a high-performance distributed database tuned especially for time series". I only noticed it because I was shocked that there was an actual comprehensible value proposition in a blogpost about a NoSQL product.

Twisell 10y ago
However being system admin of a PostgreSQL database managing a lot of timeseries I was just like "meh use timestamp and good index, what's the deal?". Also for performance sake just cluster the data using the most used index... But yeah I guess if you really need NoSQL this should be nice. However sharding based on time will probably be one of the easiest approach for horizontal PostgreSQL deployment.
- sitkack 10y ago
  One of the most compelling features of Riak, both KV and TS, is that it is masterless and replicated. When nodes fail for whatever reason, the cluster handles it transparently.

deleted 10y ago

[deleted]

adamneilson 10y ago

I think Datomic which has been doing time series data for years now answers the "what’s the big deal" question for me: "Given a value of the database, one can obtain another value of the database as-of, or since, a point in time in the past, or both (creating a windowed view of activity). These database values can be queried ordinarily, without having to make special queries parameterized by time." http://www.datomic.com/rationale.html http://www.datomic.com/rationale.html So in other words you get to see exactly how the whole database looked at any given moment through history (certain parallels with the blockchain I suppose).

zcam 10y ago
Until you reach 10B datoms and then you're off for a fun ride sharding with multiple databases. Also in an IoT context or more general timeseries usage you very likely don't care about tx, but very much do about write throughput and not having a spof, which Datomic isn't good at at all. Datomic is an ok choice in some contexts, but the one detailed here is not one of them, you're likely to reach Datomic limits quickly and be in a world of hurt when you do.
assface 10y ago
> I think Datomic which has been doing time series data for years now answers the "what’s the big deal" question for me: This is known as "time travel queries". Postgres supported this back in the original version from the 1980s but then they took it out in 1997 because it takes up too much disk space. It's trivial to do in an MVCC system. You just turn off garbage collection (vacuuming in PSQL parlance).