This totally makes sense. At Spacelift[0] we're using Datadog for most of our observability. It's great overall, but the way traces and retention work there, de

This totally makes sense. At Spacelift[0] we're using Datadog for most of our observability. It's great overall, but the way traces and retention work there, debugging issues you didn't predict beforehand is nigh impossible. Retained spans don't include the whole trace with them and you can't make them do that. You can't filter traces by fields in multiple spans. You have to set facets beforehand on fields you want to filter on, otherwise you can't. So we created a small fork of the dd tracer repo which sends all spans to firehose on the side and stores them as compressed JSON files on S3. There we query them using Athena. It's cheap, fast, and has 100% retention for an extended period of time. If you get the partitioning right and create a temporary table with just the time period you're interested in before starting a debugging session, then Athena will also be incredibly cheap. Datadog APM is still nice for exploration and initial analysis, but Athena is great for any required deep dives. PS: We'll probably write a blog post about the specifics of that setup at some point. [0]: https://spacelift.io https://spacelift.io