7 ms·

Gorilla: A Fast, Scalable, In-Memory Time Series Database (2015) [pdf]

125 points by kiril-me 10y ago

hefeweizen 10y ago
VLDB '15. Previously discussed at: https://news.ycombinator.com/item?id=10207863 https://news.ycombinator.com/item?id=10207863
- kiril-me 10y ago
  Yes, here is open source representation of the ideas presented in this paper: https://github.com/facebookincubator/beringei https://github.com/facebookincubator/beringei
DonHopkins 10y ago
"You will get a better Gorilla effect if you use as big a piece of paper as possible." -Kunihiko Kasahara, Creative Origami.
gtrubetskoy 10y ago
Not to bash Gorilla, because it looks like excellent stuff, but I do not understand the general obsession with specialized TS stores. If you're operating on FB-scale, then sure that's what you have to do, but in most cases your database (especially Postgres) is a far superior option. Time series is "unusual" in the sense that most people don't know first thing about it, event folks with degrees in math/statistics. I think this is why there is the prevailing misconception that it requires a specialized database. Incidently I've just written a blog about storing TS in PostgreSQL: https://grisha.org/blog/2016/12/16/storing-time-series-in-postgresql-part-ii/ https://grisha.org/blog/2016/12/16/storing-time-series-in-po...
- mej10 10y ago
  You don't have to be FB-scale to need something capable of handling a larger amount of timeseries data than postgresql can handle efficiently. 100 servers each producing 100 stats tagged 15 different ways aggregated into avg/50th/95th/min/max every 5 seconds is 540 million data points per hour (double check my math but it seems about right).
  - gtrubetskoy 10y ago
    100 servers producing 100 stats is 10,000 series - the number of data points is a function of the resolution of the series, for example at 5s they'll accumulate 17,280 per day per series for a total of 172,800,000 data points / day, which is not a big number (relatively speaking). You probably also don't want to keep a 5s resolution forever, and after a day you may want to store 1m resolutions (reducing points 12x), then perhaps hourly after week, daily after a month, etc.
    - CptMauli 10y ago
      And this is exactly the misconception. In my use cases I want to have the original data for at least a couple of weeks, maybe even a year. I also need millisecond precision because some of the sources produce several values per second (others might not change for years). I also need to store the information if the value had an alarm, or an error or if it is an replacement value. So a simple round robin scheme just doesn't work.
      - gtrubetskoy 10y ago
        I wonder if you're confusing time series (which is a sequence of numbers which lends itself to stuff like regressions, etc, for predictions and other insights) with quite simply _logging_ so that you can go back a year later and figure out "what the hell happened then".
        chaotic-good 10y ago
        https://en.wikipedia.org/wiki/Unevenly_spaced_time_series https://en.wikipedia.org/wiki/Unevenly_spaced_time_series You can call that _logging_ if you want, but people will ask for it anyway.
        gtrubetskoy 10y ago
        Interesting, but I'm not convinced. Everything is evenly spaced given correct resolution. If your timestamp is at nanosecond precision, then you simply have a 1ns resolution series which is very sparse, I suppose another name for it might be "unevenly spaced".
- thu 10y ago
  Did you compare the disk usage vs. whisper ?
  - gtrubetskoy 10y ago
    My guess is that whisper would be more efficient on disk, not by much though. But whisper won't let you SELECT ... FROM tv JOIN customers ON ... JOIN purchases ON ...
  - SEJeff 10y ago
    whisper is 12 bytes per data point. The Gorilla style time series saves gobs of data over the whisper style, there is simply little to no comparison. Gorilla is better than whisper when it comes to storing the same data in less storage space. Disclaimer: I am a co-maintainer of graphite and try to write code for it when I have free time (so not much right now or for the past year or so) https://github.com/graphite-project/whisper/commits?author=SEJeff https://github.com/graphite-project/whisper/commits?author=S...
    - gtrubetskoy 10y ago
      In this case Tgres beats Whisper by quite a bit - it uses 8 bytes per data point. In fact looking at the numbers for a small test db I have 2,057,275 points taking up 16,687,104 bytes, which is exactly 8.11 bytes per point. BTW getting the total point count is as easy as: select sum(array_length(dp,1)) from ts; If you adjust the width (i.e. points per table row) to a higher number than the 768 default, as soon as you get above the PG pages size, TOAST (https://www.postgresql.org/docs/current/static/storage-toast.html https://www.postgresql.org/docs/current/static/storage-toast...) kicks in, which PG compresses. Not sure how well this works, but at least in theory it should make it even more compact (though not sure about it being as performant).
- DrPhish 10y ago
  Interesting. It'd be very nice to use my preferred tool set (postgres) as an alternative to RRDTool when doing datalogging. Do you any comparisons for speed/accuracy/compactness vs using RRDTool?
- bbrazil 10y ago
  > If you're operating on FB-scale, then sure that's what you have to do, but in most cases your database (especially Postgres) is a far superior option. > Time series is "unusual" in the sense that most people don't know first thing about it, event folks with degrees in math/statistics. I think this is why there is the prevailing misconception that it requires a specialized database. I'm one of the developers of Prometheus, and while I'm going to point appropriate use cases towards Postgres (and regularly do), even relatively small monitoring loads require careful handling such that a traditional database isn't suitable. An example from previous company is that we had only ~50 machines in one datacenter and yet were up to 30,000 samples per second. This is not considered to be a large setup, which would be in the hundreds of thousands of samples per second. All the buffering/batching required to make such loads practical requires special design, as naively making each new data point into a disk write (with fdatasync) will not work out - even with SSDs. And that's just writes. You also need to be able to efficiently read and process back the data efficiently when it is queried. I'd suggest https://www.youtube.com/watch?v=HbnGSNEjhUc https://www.youtube.com/watch?v=HbnGSNEjhUc to give a look into how Prometheus does it (which also covers our use of Gorilla). There was also a good post on the InfluxDB blog about the evolution of their design and why the problem is hard that I can't seem find right now.
  - gtrubetskoy 10y ago
    I think that this is an argument over apples and oranges. I'm well familiar with the write problem you're speaking of and even wrote a blog a couple of years ago detailing influx's storage code. The "apples" use case is when you want for lack of a better description a Grafana chart out of it. (E.g. the cluster cpu load, "we're getting X page views per second", etc). The "oranges" case is when every data point matters, and I think that this particular case is not so much about time series, but logging, where a log is a "time series" technically speaking. Here of course you're talking massive amounts of data which you want to do your best to optimize and it's a hard problem. It's not a _time series_ problem, it's a problem of storage. Splunk (sorry, can't think of a better example ATM) doesn't describe itself as a "times series database", and yet it's addressing that very problem of horizontally scalable write-intensive storage. So is hbase, cassandra, accumulo, bigquery and all their friends. In my experience the best solution is to use two separate systems one for each case. Because there is absolutely no value in a "disk used" measurement every millisecond - aggregated to once a minute (or few) is perfectly fine, and retaining the original data is not needed and you'd be a fool to build a Cassndra cluster to handle it. On the other hand if I'm recording equity bids and asks, well then I better record every point forever.
- scottlocklin 10y ago
  It's not a misconception. If you need to store timeseries and actually do something substantive with them (time oriented joins; calculating bid-ask spreads, order book math, correlation matrices, calculating betas), Postgres won't cut it, even on a relatively small data set, like all the OCHL daily stock market data for the last 30 years. Neither will any of the other open source solutions.
- chaotic-good 10y ago
  > I do not understand the general obsession with specialized TS stores Specialized TS stores exist for decades (process historians, tickerplants, and so on).
tschellenbach 10y ago
There still isn't a leading time series database. Think the use case definitely deserves it though. I've seen so many projects spend months in building custom data rollup solutions, it's a waste of time. A good time series database would be pretty amazing.
- invertedohm 10y ago
  While it's a relatively young product, I've been running InfluxDB at scale for over a year in production and it's been a joy to use. Hundreds of hosts reporting stats every 10 seconds and the query language is straightforward and powerful once you grok the data structures a bit. It fits the bill, I think.
  - pwernersbach 10y ago
    Can you describe your workload and your experience with InfluxDB? We evaluated InfluxDB in 2015 and found that it fell apart due to the large amount of data we produce. I'd be interested in hearing your experiences to see if we need to reevaluate.
    - invertedohm 10y ago
      Workload is mostly writing incoming metrics from many dynamic instances (AWS). In 2015 Influx was at pre-1.0 and definitely had a tendency to "fall apart" under stress - for one, there were no memory limitations implemented, so an accidental large query would eat up all your memory and kill the database. To be fair, no one recommended putting it in production at that point. It's just that I needed something to run a Graphite-like metrics platform at scale and it had to be future-proof. Bit of a gamble but it turned out well! :)
  - lsh 10y ago
    +1 for InfluxDB and the TICK stack. We evaluated it for our monitoring needs, decided not to go with it but were damned impressed with how easy it was to setup and use, the features (including the query language) and the overall quality of the stack. Less impressed with the Influx org and their SaaS, but I definitely want to find a usecase for getting stuck into Influx for time series collection
pixelmonkey 10y ago
I read this paper a few months ago and summarized it for my team. Sharing the notes here in case it's helpful for others: - when storing time series keys, you can save a lot of space by encoding them as a first timestamp followed by timestamp offsets (deltas) - when storing time series values, you can save a lot of space by realizing sequential data points tend not to be volatile... e.g. a "writes per minute" series is more likely to be 100, 99, 101, etc. than it is to be 100, 999999, 1, etc. This means you can encode the current value by using XOR tricks with prior values to save space - the authors suggest using in-memory caching for recent data, but doing eventual persistence to HBase or similar distributed filesystems; thanks to this, you can get real-time operational metrics that are fast and space efficient, yet still have a system that scales out horizontally for historical storage The Morning Paper also did a good analysis here: https://blog.acolyer.org/2016/05/03/gorilla-a-fast-scalable-in-memory-time-series-database/ https://blog.acolyer.org/2016/05/03/gorilla-a-fast-scalable-...
- marknadal 10y ago
  Good tips, on that note you really don't have to use HBase or some always-on managed filesystem. You can actually use S3! I know this sounds weird as you'd rack up API costs way faster than storage cost, but we implemented a simple batching approach for timeseries data that could handle 100M+ messages for $10/day (over 100GB+ of data, includes all costs, processing, storage, backup). Did a 2 min screen cast here: https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8 . This hopefully can be a lifesaver on costs for some people, and keeps your system overall simpler.
  - StreamBright 10y ago
    This is neat! S3 is always my first choice when it comes to scalable cheap storage. The price and scalability beats almost everything in this category. Reliability is unprecedented as well. I am wondering what was you file format for storing the TS data.
- mrits 10y ago
  Picking an encoding or compression on any column is now available in several popular datastores. Vertica by HP and Druid.IO are two that come to mine.
pauldix 10y ago
This is a great paper. On InfluxDB we used a similar compression technique for float64s in our new storage engine. A key difference we have is that we separated timestamps from values so we can use run length encoding for regular time series.
crudbug 10y ago
Does the architecture allow persistent storage plugins ?
manigandham 10y ago
Submitted this a while back, the Morning Paper did a good write up of BTrDB which is another very fast TSDB written in Go, storing 2.1 trillion data points, and supporting 119M queries per second (53M inserts per second) in a four-node cluster https://blog.acolyer.org/2016/05/04/btrdb-optimizing-storage-system-design-for-timeseries-processing/ https://blog.acolyer.org/2016/05/04/btrdb-optimizing-storage...

125 points by kiril-me 10y ago

VLDB '15. Previously discussed at: https://news.ycombinator.com/item?id=10207863 https://news.ycombinator.com/item?id=10207863

kiril-me 10y ago
Yes, here is open source representation of the ideas presented in this paper: https://github.com/facebookincubator/beringei https://github.com/facebookincubator/beringei

DonHopkins 10y ago

"You will get a better Gorilla effect if you use as big a piece of paper as possible." -Kunihiko Kasahara, Creative Origami.

gtrubetskoy 10y ago

Not to bash Gorilla, because it looks like excellent stuff, but I do not understand the general obsession with specialized TS stores. If you're operating on FB-scale, then sure that's what you have to do, but in most cases your database (especially Postgres) is a far superior option. Time series is "unusual" in the sense that most people don't know first thing about it, event folks with degrees in math/statistics. I think this is why there is the prevailing misconception that it requires a specialized database. Incidently I've just written a blog about storing TS in PostgreSQL: https://grisha.org/blog/2016/12/16/storing-time-series-in-postgresql-part-ii/ https://grisha.org/blog/2016/12/16/storing-time-series-in-po...

mej10 10y ago
You don't have to be FB-scale to need something capable of handling a larger amount of timeseries data than postgresql can handle efficiently. 100 servers each producing 100 stats tagged 15 different ways aggregated into avg/50th/95th/min/max every 5 seconds is 540 million data points per hour (double check my math but it seems about right).
- gtrubetskoy 10y ago
  100 servers producing 100 stats is 10,000 series - the number of data points is a function of the resolution of the series, for example at 5s they'll accumulate 17,280 per day per series for a total of 172,800,000 data points / day, which is not a big number (relatively speaking). You probably also don't want to keep a 5s resolution forever, and after a day you may want to store 1m resolutions (reducing points 12x), then perhaps hourly after week, daily after a month, etc.
  - CptMauli 10y ago
    And this is exactly the misconception. In my use cases I want to have the original data for at least a couple of weeks, maybe even a year. I also need millisecond precision because some of the sources produce several values per second (others might not change for years). I also need to store the information if the value had an alarm, or an error or if it is an replacement value. So a simple round robin scheme just doesn't work.
    - gtrubetskoy 10y ago
      I wonder if you're confusing time series (which is a sequence of numbers which lends itself to stuff like regressions, etc, for predictions and other insights) with quite simply _logging_ so that you can go back a year later and figure out "what the hell happened then".
      - chaotic-good 10y ago
        https://en.wikipedia.org/wiki/Unevenly_spaced_time_series https://en.wikipedia.org/wiki/Unevenly_spaced_time_series You can call that _logging_ if you want, but people will ask for it anyway.
        gtrubetskoy 10y ago
        Interesting, but I'm not convinced. Everything is evenly spaced given correct resolution. If your timestamp is at nanosecond precision, then you simply have a 1ns resolution series which is very sparse, I suppose another name for it might be "unevenly spaced".
thu 10y ago
Did you compare the disk usage vs. whisper ?
- gtrubetskoy 10y ago
  My guess is that whisper would be more efficient on disk, not by much though. But whisper won't let you SELECT ... FROM tv JOIN customers ON ... JOIN purchases ON ...
- SEJeff 10y ago
  whisper is 12 bytes per data point. The Gorilla style time series saves gobs of data over the whisper style, there is simply little to no comparison. Gorilla is better than whisper when it comes to storing the same data in less storage space. Disclaimer: I am a co-maintainer of graphite and try to write code for it when I have free time (so not much right now or for the past year or so) https://github.com/graphite-project/whisper/commits?author=SEJeff https://github.com/graphite-project/whisper/commits?author=S...
  - gtrubetskoy 10y ago
    In this case Tgres beats Whisper by quite a bit - it uses 8 bytes per data point. In fact looking at the numbers for a small test db I have 2,057,275 points taking up 16,687,104 bytes, which is exactly 8.11 bytes per point. BTW getting the total point count is as easy as: select sum(array_length(dp,1)) from ts; If you adjust the width (i.e. points per table row) to a higher number than the 768 default, as soon as you get above the PG pages size, TOAST (https://www.postgresql.org/docs/current/static/storage-toast.html https://www.postgresql.org/docs/current/static/storage-toast...) kicks in, which PG compresses. Not sure how well this works, but at least in theory it should make it even more compact (though not sure about it being as performant).
DrPhish 10y ago
Interesting. It'd be very nice to use my preferred tool set (postgres) as an alternative to RRDTool when doing datalogging. Do you any comparisons for speed/accuracy/compactness vs using RRDTool?
bbrazil 10y ago
> If you're operating on FB-scale, then sure that's what you have to do, but in most cases your database (especially Postgres) is a far superior option. > Time series is "unusual" in the sense that most people don't know first thing about it, event folks with degrees in math/statistics. I think this is why there is the prevailing misconception that it requires a specialized database. I'm one of the developers of Prometheus, and while I'm going to point appropriate use cases towards Postgres (and regularly do), even relatively small monitoring loads require careful handling such that a traditional database isn't suitable. An example from previous company is that we had only ~50 machines in one datacenter and yet were up to 30,000 samples per second. This is not considered to be a large setup, which would be in the hundreds of thousands of samples per second. All the buffering/batching required to make such loads practical requires special design, as naively making each new data point into a disk write (with fdatasync) will not work out - even with SSDs. And that's just writes. You also need to be able to efficiently read and process back the data efficiently when it is queried. I'd suggest https://www.youtube.com/watch?v=HbnGSNEjhUc https://www.youtube.com/watch?v=HbnGSNEjhUc to give a look into how Prometheus does it (which also covers our use of Gorilla). There was also a good post on the InfluxDB blog about the evolution of their design and why the problem is hard that I can't seem find right now.
- gtrubetskoy 10y ago
  I think that this is an argument over apples and oranges. I'm well familiar with the write problem you're speaking of and even wrote a blog a couple of years ago detailing influx's storage code. The "apples" use case is when you want for lack of a better description a Grafana chart out of it. (E.g. the cluster cpu load, "we're getting X page views per second", etc). The "oranges" case is when every data point matters, and I think that this particular case is not so much about time series, but logging, where a log is a "time series" technically speaking. Here of course you're talking massive amounts of data which you want to do your best to optimize and it's a hard problem. It's not a _time series_ problem, it's a problem of storage. Splunk (sorry, can't think of a better example ATM) doesn't describe itself as a "times series database", and yet it's addressing that very problem of horizontally scalable write-intensive storage. So is hbase, cassandra, accumulo, bigquery and all their friends. In my experience the best solution is to use two separate systems one for each case. Because there is absolutely no value in a "disk used" measurement every millisecond - aggregated to once a minute (or few) is perfectly fine, and retaining the original data is not needed and you'd be a fool to build a Cassndra cluster to handle it. On the other hand if I'm recording equity bids and asks, well then I better record every point forever.
scottlocklin 10y ago
It's not a misconception. If you need to store timeseries and actually do something substantive with them (time oriented joins; calculating bid-ask spreads, order book math, correlation matrices, calculating betas), Postgres won't cut it, even on a relatively small data set, like all the OCHL daily stock market data for the last 30 years. Neither will any of the other open source solutions.
chaotic-good 10y ago
> I do not understand the general obsession with specialized TS stores Specialized TS stores exist for decades (process historians, tickerplants, and so on).

tschellenbach 10y ago

There still isn't a leading time series database. Think the use case definitely deserves it though. I've seen so many projects spend months in building custom data rollup solutions, it's a waste of time. A good time series database would be pretty amazing.

invertedohm 10y ago
While it's a relatively young product, I've been running InfluxDB at scale for over a year in production and it's been a joy to use. Hundreds of hosts reporting stats every 10 seconds and the query language is straightforward and powerful once you grok the data structures a bit. It fits the bill, I think.
- pwernersbach 10y ago
  Can you describe your workload and your experience with InfluxDB? We evaluated InfluxDB in 2015 and found that it fell apart due to the large amount of data we produce. I'd be interested in hearing your experiences to see if we need to reevaluate.
  - invertedohm 10y ago
    Workload is mostly writing incoming metrics from many dynamic instances (AWS). In 2015 Influx was at pre-1.0 and definitely had a tendency to "fall apart" under stress - for one, there were no memory limitations implemented, so an accidental large query would eat up all your memory and kill the database. To be fair, no one recommended putting it in production at that point. It's just that I needed something to run a Graphite-like metrics platform at scale and it had to be future-proof. Bit of a gamble but it turned out well! :)
- lsh 10y ago
  +1 for InfluxDB and the TICK stack. We evaluated it for our monitoring needs, decided not to go with it but were damned impressed with how easy it was to setup and use, the features (including the query language) and the overall quality of the stack. Less impressed with the Influx org and their SaaS, but I definitely want to find a usecase for getting stuck into Influx for time series collection

pixelmonkey 10y ago

I read this paper a few months ago and summarized it for my team. Sharing the notes here in case it's helpful for others: - when storing time series keys, you can save a lot of space by encoding them as a first timestamp followed by timestamp offsets (deltas) - when storing time series values, you can save a lot of space by realizing sequential data points tend not to be volatile... e.g. a "writes per minute" series is more likely to be 100, 99, 101, etc. than it is to be 100, 999999, 1, etc. This means you can encode the current value by using XOR tricks with prior values to save space - the authors suggest using in-memory caching for recent data, but doing eventual persistence to HBase or similar distributed filesystems; thanks to this, you can get real-time operational metrics that are fast and space efficient, yet still have a system that scales out horizontally for historical storage The Morning Paper also did a good analysis here: https://blog.acolyer.org/2016/05/03/gorilla-a-fast-scalable-in-memory-time-series-database/ https://blog.acolyer.org/2016/05/03/gorilla-a-fast-scalable-...

marknadal 10y ago
Good tips, on that note you really don't have to use HBase or some always-on managed filesystem. You can actually use S3! I know this sounds weird as you'd rack up API costs way faster than storage cost, but we implemented a simple batching approach for timeseries data that could handle 100M+ messages for $10/day (over 100GB+ of data, includes all costs, processing, storage, backup). Did a 2 min screen cast here: https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8 . This hopefully can be a lifesaver on costs for some people, and keeps your system overall simpler.
- StreamBright 10y ago
  This is neat! S3 is always my first choice when it comes to scalable cheap storage. The price and scalability beats almost everything in this category. Reliability is unprecedented as well. I am wondering what was you file format for storing the TS data.
mrits 10y ago
Picking an encoding or compression on any column is now available in several popular datastores. Vertica by HP and Druid.IO are two that come to mine.

pauldix 10y ago

This is a great paper. On InfluxDB we used a similar compression technique for float64s in our new storage engine. A key difference we have is that we separated timestamps from values so we can use run length encoding for regular time series.

crudbug 10y ago

Does the architecture allow persistent storage plugins ?

manigandham 10y ago

Submitted this a while back, the Morning Paper did a good write up of BTrDB which is another very fast TSDB written in Go, storing 2.1 trillion data points, and supporting 119M queries per second (53M inserts per second) in a four-node cluster https://blog.acolyer.org/2016/05/04/btrdb-optimizing-storage-system-design-for-timeseries-processing/ https://blog.acolyer.org/2016/05/04/btrdb-optimizing-storage...