3 ms·

I'd be curious to see a couple of organizations that have taken different approaches in this area discuss how they made decisions around investing in observabil

by abeppu 6y ago

I'd be curious to see a couple of organizations that have taken different approaches in this area discuss how they made decisions around investing in observability systems. When metrics gets to be a meaningful fraction of your total data size, and when it gets to be an operational burden on its own, I think it's tempting to ask "Do we really need to emit and store all this?" especially if you acknowledge that only ~2% of writes ever get read. In some sense, the organization is making an expected value of information estimate, where there's a complex interaction between the data you have and actions you can take, especially to preempt or resolve issues.

hodgesrm 6y ago
I've seen two approaches. (We have a lot of customers who deal with this exact problem.) 1.) Down-sample data to aggregates. Keep the aggregated but drop the source data after a certain period of time. You can see historical trends down to the sampling interval but cannot drill down to individual observations. 2.) Use tiered storage. Keep recent data on NVMe SSD and migrate to high density HDD or object storage over time. This allows you to keep more data. Not all DBMS support this out of the box.