5 ms·

Not to bash Gorilla, because it looks like excellent stuff, but I do not understand the general obsession with specialized TS stores. If you're operating on FB

Not to bash Gorilla, because it looks like excellent stuff, but I do not understand the general obsession with specialized TS stores. If you're operating on FB-scale, then sure that's what you have to do, but in most cases your database (especially Postgres) is a far superior option. Time series is "unusual" in the sense that most people don't know first thing about it, event folks with degrees in math/statistics. I think this is why there is the prevailing misconception that it requires a specialized database. Incidently I've just written a blog about storing TS in PostgreSQL: https://grisha.org/blog/2016/12/16/storing-time-series-in-postgresql-part-ii/ https://grisha.org/blog/2016/12/16/storing-time-series-in-po...

mej10 10y ago
You don't have to be FB-scale to need something capable of handling a larger amount of timeseries data than postgresql can handle efficiently. 100 servers each producing 100 stats tagged 15 different ways aggregated into avg/50th/95th/min/max every 5 seconds is 540 million data points per hour (double check my math but it seems about right).
- gtrubetskoy 10y ago
  100 servers producing 100 stats is 10,000 series - the number of data points is a function of the resolution of the series, for example at 5s they'll accumulate 17,280 per day per series for a total of 172,800,000 data points / day, which is not a big number (relatively speaking). You probably also don't want to keep a 5s resolution forever, and after a day you may want to store 1m resolutions (reducing points 12x), then perhaps hourly after week, daily after a month, etc.
  - CptMauli 10y ago
    And this is exactly the misconception. In my use cases I want to have the original data for at least a couple of weeks, maybe even a year. I also need millisecond precision because some of the sources produce several values per second (others might not change for years). I also need to store the information if the value had an alarm, or an error or if it is an replacement value. So a simple round robin scheme just doesn't work.
    - gtrubetskoy 10y ago
      I wonder if you're confusing time series (which is a sequence of numbers which lends itself to stuff like regressions, etc, for predictions and other insights) with quite simply _logging_ so that you can go back a year later and figure out "what the hell happened then".
      - chaotic-good 10y ago
        https://en.wikipedia.org/wiki/Unevenly_spaced_time_series https://en.wikipedia.org/wiki/Unevenly_spaced_time_series You can call that _logging_ if you want, but people will ask for it anyway.
        gtrubetskoy 10y ago
        Interesting, but I'm not convinced. Everything is evenly spaced given correct resolution. If your timestamp is at nanosecond precision, then you simply have a 1ns resolution series which is very sparse, I suppose another name for it might be "unevenly spaced".
      - bbrazil 10y ago
        While metrics are often evenly spaced, that's not always the case. Good resolution on the timestamps helps avoid graph artifacts and other weirdness, even if you're only collecting them once a minute. Millisecond accuracy seems to be sufficient in my experience, there's other races of that magnitude anyway (network jitter, kernel scheduling etc.).
  - feisuzhu 10y ago
    You probably also don't want to keep a 5s resolution forever. Yeah, that's a feature provided by specialized TSDB, you don't need to implement yourself.
  - user5994461 10y ago
    Need 5 seconds for at least one week. One day is too short, I regularly have to look at the metrics from yesterday to debug an issue that we noticed today and they're worthless because they've been averaged.
thu 10y ago
Did you compare the disk usage vs. whisper ?
- gtrubetskoy 10y ago
  My guess is that whisper would be more efficient on disk, not by much though. But whisper won't let you SELECT ... FROM tv JOIN customers ON ... JOIN purchases ON ...
- SEJeff 10y ago
  whisper is 12 bytes per data point. The Gorilla style time series saves gobs of data over the whisper style, there is simply little to no comparison. Gorilla is better than whisper when it comes to storing the same data in less storage space. Disclaimer: I am a co-maintainer of graphite and try to write code for it when I have free time (so not much right now or for the past year or so) https://github.com/graphite-project/whisper/commits?author=SEJeff https://github.com/graphite-project/whisper/commits?author=S...
  - gtrubetskoy 10y ago
    In this case Tgres beats Whisper by quite a bit - it uses 8 bytes per data point. In fact looking at the numbers for a small test db I have 2,057,275 points taking up 16,687,104 bytes, which is exactly 8.11 bytes per point. BTW getting the total point count is as easy as: select sum(array_length(dp,1)) from ts; If you adjust the width (i.e. points per table row) to a higher number than the 768 default, as soon as you get above the PG pages size, TOAST (https://www.postgresql.org/docs/current/static/storage-toast.html https://www.postgresql.org/docs/current/static/storage-toast...) kicks in, which PG compresses. Not sure how well this works, but at least in theory it should make it even more compact (though not sure about it being as performant).
DrPhish 10y ago
Interesting. It'd be very nice to use my preferred tool set (postgres) as an alternative to RRDTool when doing datalogging. Do you any comparisons for speed/accuracy/compactness vs using RRDTool?
bbrazil 10y ago
> If you're operating on FB-scale, then sure that's what you have to do, but in most cases your database (especially Postgres) is a far superior option. > Time series is "unusual" in the sense that most people don't know first thing about it, event folks with degrees in math/statistics. I think this is why there is the prevailing misconception that it requires a specialized database. I'm one of the developers of Prometheus, and while I'm going to point appropriate use cases towards Postgres (and regularly do), even relatively small monitoring loads require careful handling such that a traditional database isn't suitable. An example from previous company is that we had only ~50 machines in one datacenter and yet were up to 30,000 samples per second. This is not considered to be a large setup, which would be in the hundreds of thousands of samples per second. All the buffering/batching required to make such loads practical requires special design, as naively making each new data point into a disk write (with fdatasync) will not work out - even with SSDs. And that's just writes. You also need to be able to efficiently read and process back the data efficiently when it is queried. I'd suggest https://www.youtube.com/watch?v=HbnGSNEjhUc https://www.youtube.com/watch?v=HbnGSNEjhUc to give a look into how Prometheus does it (which also covers our use of Gorilla). There was also a good post on the InfluxDB blog about the evolution of their design and why the problem is hard that I can't seem find right now.
- gtrubetskoy 10y ago
  I think that this is an argument over apples and oranges. I'm well familiar with the write problem you're speaking of and even wrote a blog a couple of years ago detailing influx's storage code. The "apples" use case is when you want for lack of a better description a Grafana chart out of it. (E.g. the cluster cpu load, "we're getting X page views per second", etc). The "oranges" case is when every data point matters, and I think that this particular case is not so much about time series, but logging, where a log is a "time series" technically speaking. Here of course you're talking massive amounts of data which you want to do your best to optimize and it's a hard problem. It's not a _time series_ problem, it's a problem of storage. Splunk (sorry, can't think of a better example ATM) doesn't describe itself as a "times series database", and yet it's addressing that very problem of horizontally scalable write-intensive storage. So is hbase, cassandra, accumulo, bigquery and all their friends. In my experience the best solution is to use two separate systems one for each case. Because there is absolutely no value in a "disk used" measurement every millisecond - aggregated to once a minute (or few) is perfectly fine, and retaining the original data is not needed and you'd be a fool to build a Cassndra cluster to handle it. On the other hand if I'm recording equity bids and asks, well then I better record every point forever.
scottlocklin 10y ago
It's not a misconception. If you need to store timeseries and actually do something substantive with them (time oriented joins; calculating bid-ask spreads, order book math, correlation matrices, calculating betas), Postgres won't cut it, even on a relatively small data set, like all the OCHL daily stock market data for the last 30 years. Neither will any of the other open source solutions.
chaotic-good 10y ago
> I do not understand the general obsession with specialized TS stores Specialized TS stores exist for decades (process historians, tickerplants, and so on).

mej10 10y ago

You don't have to be FB-scale to need something capable of handling a larger amount of timeseries data than postgresql can handle efficiently. 100 servers each producing 100 stats tagged 15 different ways aggregated into avg/50th/95th/min/max every 5 seconds is 540 million data points per hour (double check my math but it seems about right).

gtrubetskoy 10y ago
100 servers producing 100 stats is 10,000 series - the number of data points is a function of the resolution of the series, for example at 5s they'll accumulate 17,280 per day per series for a total of 172,800,000 data points / day, which is not a big number (relatively speaking). You probably also don't want to keep a 5s resolution forever, and after a day you may want to store 1m resolutions (reducing points 12x), then perhaps hourly after week, daily after a month, etc.
- CptMauli 10y ago
  And this is exactly the misconception. In my use cases I want to have the original data for at least a couple of weeks, maybe even a year. I also need millisecond precision because some of the sources produce several values per second (others might not change for years). I also need to store the information if the value had an alarm, or an error or if it is an replacement value. So a simple round robin scheme just doesn't work.
  - gtrubetskoy 10y ago
    I wonder if you're confusing time series (which is a sequence of numbers which lends itself to stuff like regressions, etc, for predictions and other insights) with quite simply _logging_ so that you can go back a year later and figure out "what the hell happened then".
    - chaotic-good 10y ago
      https://en.wikipedia.org/wiki/Unevenly_spaced_time_series https://en.wikipedia.org/wiki/Unevenly_spaced_time_series You can call that _logging_ if you want, but people will ask for it anyway.
      - gtrubetskoy 10y ago
        Interesting, but I'm not convinced. Everything is evenly spaced given correct resolution. If your timestamp is at nanosecond precision, then you simply have a 1ns resolution series which is very sparse, I suppose another name for it might be "unevenly spaced".
    - bbrazil 10y ago
      While metrics are often evenly spaced, that's not always the case. Good resolution on the timestamps helps avoid graph artifacts and other weirdness, even if you're only collecting them once a minute. Millisecond accuracy seems to be sufficient in my experience, there's other races of that magnitude anyway (network jitter, kernel scheduling etc.).
- feisuzhu 10y ago
  You probably also don't want to keep a 5s resolution forever. Yeah, that's a feature provided by specialized TSDB, you don't need to implement yourself.
- user5994461 10y ago
  Need 5 seconds for at least one week. One day is too short, I regularly have to look at the metrics from yesterday to debug an issue that we noticed today and they're worthless because they've been averaged.

thu 10y ago

Did you compare the disk usage vs. whisper ?

gtrubetskoy 10y ago
My guess is that whisper would be more efficient on disk, not by much though. But whisper won't let you SELECT ... FROM tv JOIN customers ON ... JOIN purchases ON ...
SEJeff 10y ago
whisper is 12 bytes per data point. The Gorilla style time series saves gobs of data over the whisper style, there is simply little to no comparison. Gorilla is better than whisper when it comes to storing the same data in less storage space. Disclaimer: I am a co-maintainer of graphite and try to write code for it when I have free time (so not much right now or for the past year or so) https://github.com/graphite-project/whisper/commits?author=SEJeff https://github.com/graphite-project/whisper/commits?author=S...
- gtrubetskoy 10y ago
  In this case Tgres beats Whisper by quite a bit - it uses 8 bytes per data point. In fact looking at the numbers for a small test db I have 2,057,275 points taking up 16,687,104 bytes, which is exactly 8.11 bytes per point. BTW getting the total point count is as easy as: select sum(array_length(dp,1)) from ts; If you adjust the width (i.e. points per table row) to a higher number than the 768 default, as soon as you get above the PG pages size, TOAST (https://www.postgresql.org/docs/current/static/storage-toast.html https://www.postgresql.org/docs/current/static/storage-toast...) kicks in, which PG compresses. Not sure how well this works, but at least in theory it should make it even more compact (though not sure about it being as performant).

DrPhish 10y ago

Interesting. It'd be very nice to use my preferred tool set (postgres) as an alternative to RRDTool when doing datalogging. Do you any comparisons for speed/accuracy/compactness vs using RRDTool?

bbrazil 10y ago

> If you're operating on FB-scale, then sure that's what you have to do, but in most cases your database (especially Postgres) is a far superior option. > Time series is "unusual" in the sense that most people don't know first thing about it, event folks with degrees in math/statistics. I think this is why there is the prevailing misconception that it requires a specialized database. I'm one of the developers of Prometheus, and while I'm going to point appropriate use cases towards Postgres (and regularly do), even relatively small monitoring loads require careful handling such that a traditional database isn't suitable. An example from previous company is that we had only ~50 machines in one datacenter and yet were up to 30,000 samples per second. This is not considered to be a large setup, which would be in the hundreds of thousands of samples per second. All the buffering/batching required to make such loads practical requires special design, as naively making each new data point into a disk write (with fdatasync) will not work out - even with SSDs. And that's just writes. You also need to be able to efficiently read and process back the data efficiently when it is queried. I'd suggest https://www.youtube.com/watch?v=HbnGSNEjhUc https://www.youtube.com/watch?v=HbnGSNEjhUc to give a look into how Prometheus does it (which also covers our use of Gorilla). There was also a good post on the InfluxDB blog about the evolution of their design and why the problem is hard that I can't seem find right now.

gtrubetskoy 10y ago
I think that this is an argument over apples and oranges. I'm well familiar with the write problem you're speaking of and even wrote a blog a couple of years ago detailing influx's storage code. The "apples" use case is when you want for lack of a better description a Grafana chart out of it. (E.g. the cluster cpu load, "we're getting X page views per second", etc). The "oranges" case is when every data point matters, and I think that this particular case is not so much about time series, but logging, where a log is a "time series" technically speaking. Here of course you're talking massive amounts of data which you want to do your best to optimize and it's a hard problem. It's not a _time series_ problem, it's a problem of storage. Splunk (sorry, can't think of a better example ATM) doesn't describe itself as a "times series database", and yet it's addressing that very problem of horizontally scalable write-intensive storage. So is hbase, cassandra, accumulo, bigquery and all their friends. In my experience the best solution is to use two separate systems one for each case. Because there is absolutely no value in a "disk used" measurement every millisecond - aggregated to once a minute (or few) is perfectly fine, and retaining the original data is not needed and you'd be a fool to build a Cassndra cluster to handle it. On the other hand if I'm recording equity bids and asks, well then I better record every point forever.

scottlocklin 10y ago

It's not a misconception. If you need to store timeseries and actually do something substantive with them (time oriented joins; calculating bid-ask spreads, order book math, correlation matrices, calculating betas), Postgres won't cut it, even on a relatively small data set, like all the OCHL daily stock market data for the last 30 years. Neither will any of the other open source solutions.

chaotic-good 10y ago

> I do not understand the general obsession with specialized TS stores Specialized TS stores exist for decades (process historians, tickerplants, and so on).