3 ms·
My main question is why observability data needs (or benefits from) a tailor-made database instead of a general purpose one. In 2025, anyone working on observab
by etaioinshrdlu 2y ago
My main question is why observability data needs (or benefits from) a tailor-made database instead of a general purpose one. In 2025, anyone working on observability who told me they have to build their own database, I would be very suspicious!
- tison 2y agoDatadog always builds their own event store: https://www.datadoghq.com/blog/engineering/introducing-husky/ https://www.datadoghq.com/blog/engineering/introducing-husky... It may not be named "database" but actually take the place of a database. Observability vendors will try to store logs with ElasticSearch and later find it over expensive and has weak support for archiving cold data. Data Warehouse solution requires a complex ETL pipeline and can be awkward when handling log data (semi-structured data). That said, if you're building an observability solution for a single company, I'd totally agree to start with single node PG with backup, and only consider other solution when data and query workload grow.
- jcgrillo 2y agoIn 2025 I'd consider starting with clickhouse instead, if you're going the DIY route
- Jolter 2y agoNot even limited to general purpose ones, there are existing tailor made databases for observability. Maybe somewhere on that page, they explain why this one is better.