4 ms·

You don't have to be FB-scale to need something capable of handling a larger amount of timeseries data than postgresql can handle efficiently. 100 servers each

by mej10 10y ago

You don't have to be FB-scale to need something capable of handling a larger amount of timeseries data than postgresql can handle efficiently. 100 servers each producing 100 stats tagged 15 different ways aggregated into avg/50th/95th/min/max every 5 seconds is 540 million data points per hour (double check my math but it seems about right).

gtrubetskoy 10y ago
100 servers producing 100 stats is 10,000 series - the number of data points is a function of the resolution of the series, for example at 5s they'll accumulate 17,280 per day per series for a total of 172,800,000 data points / day, which is not a big number (relatively speaking). You probably also don't want to keep a 5s resolution forever, and after a day you may want to store 1m resolutions (reducing points 12x), then perhaps hourly after week, daily after a month, etc.
- CptMauli 10y ago
  And this is exactly the misconception. In my use cases I want to have the original data for at least a couple of weeks, maybe even a year. I also need millisecond precision because some of the sources produce several values per second (others might not change for years). I also need to store the information if the value had an alarm, or an error or if it is an replacement value. So a simple round robin scheme just doesn't work.
  - gtrubetskoy 10y ago
    I wonder if you're confusing time series (which is a sequence of numbers which lends itself to stuff like regressions, etc, for predictions and other insights) with quite simply _logging_ so that you can go back a year later and figure out "what the hell happened then".
    - chaotic-good 10y ago
      https://en.wikipedia.org/wiki/Unevenly_spaced_time_series https://en.wikipedia.org/wiki/Unevenly_spaced_time_series You can call that _logging_ if you want, but people will ask for it anyway.
      - gtrubetskoy 10y ago
        Interesting, but I'm not convinced. Everything is evenly spaced given correct resolution. If your timestamp is at nanosecond precision, then you simply have a 1ns resolution series which is very sparse, I suppose another name for it might be "unevenly spaced".
    - bbrazil 10y ago
      While metrics are often evenly spaced, that's not always the case. Good resolution on the timestamps helps avoid graph artifacts and other weirdness, even if you're only collecting them once a minute. Millisecond accuracy seems to be sufficient in my experience, there's other races of that magnitude anyway (network jitter, kernel scheduling etc.).
- feisuzhu 10y ago
  You probably also don't want to keep a 5s resolution forever. Yeah, that's a feature provided by specialized TSDB, you don't need to implement yourself.
- user5994461 10y ago
  Need 5 seconds for at least one week. One day is too short, I regularly have to look at the metrics from yesterday to debug an issue that we noticed today and they're worthless because they've been averaged.

by mej10 10y ago

gtrubetskoy 10y ago

100 servers producing 100 stats is 10,000 series - the number of data points is a function of the resolution of the series, for example at 5s they'll accumulate 17,280 per day per series for a total of 172,800,000 data points / day, which is not a big number (relatively speaking). You probably also don't want to keep a 5s resolution forever, and after a day you may want to store 1m resolutions (reducing points 12x), then perhaps hourly after week, daily after a month, etc.

CptMauli 10y ago
And this is exactly the misconception. In my use cases I want to have the original data for at least a couple of weeks, maybe even a year. I also need millisecond precision because some of the sources produce several values per second (others might not change for years). I also need to store the information if the value had an alarm, or an error or if it is an replacement value. So a simple round robin scheme just doesn't work.
- gtrubetskoy 10y ago
  I wonder if you're confusing time series (which is a sequence of numbers which lends itself to stuff like regressions, etc, for predictions and other insights) with quite simply _logging_ so that you can go back a year later and figure out "what the hell happened then".
  - chaotic-good 10y ago
    https://en.wikipedia.org/wiki/Unevenly_spaced_time_series https://en.wikipedia.org/wiki/Unevenly_spaced_time_series You can call that _logging_ if you want, but people will ask for it anyway.
    - gtrubetskoy 10y ago
      Interesting, but I'm not convinced. Everything is evenly spaced given correct resolution. If your timestamp is at nanosecond precision, then you simply have a 1ns resolution series which is very sparse, I suppose another name for it might be "unevenly spaced".
  - bbrazil 10y ago
    While metrics are often evenly spaced, that's not always the case. Good resolution on the timestamps helps avoid graph artifacts and other weirdness, even if you're only collecting them once a minute. Millisecond accuracy seems to be sufficient in my experience, there's other races of that magnitude anyway (network jitter, kernel scheduling etc.).
feisuzhu 10y ago
You probably also don't want to keep a 5s resolution forever. Yeah, that's a feature provided by specialized TSDB, you don't need to implement yourself.
user5994461 10y ago
Need 5 seconds for at least one week. One day is too short, I regularly have to look at the metrics from yesterday to debug an issue that we noticed today and they're worthless because they've been averaged.