2 ms·
I'm just guessing but observability is often looking at small slices of hot data and then there's a vast set of cold data that is occasionally needed. Sounds li
by sethev 2mo ago
I'm just guessing but observability is often looking at small slices of hot data and then there's a vast set of cold data that is occasionally needed. Sounds like NVMe is for the hot data cache and object storage is for cold data.
- encoderer 2mo agoYes, you nailed it. Old data is important sometimes, like when a problem has been identified and investigated, but most workloads are looking at the current state of the system. So we keep the hot data cached locally (not nas/ebs) and s3 is always the source of truth. DuckDB over parquet files on a local ssd is fast enough you don’t need a traditional database. We are using DuckLake with a “lakehouse” architecture for the observability agent.