4 ms·
I get “designed by Java developers” vibes, which is a kind of overengineering to be sure, but a particularly nasty one, and one none of us should have to deal w
by hinkley 3y ago
I get “designed by Java developers” vibes, which is a kind of overengineering to be sure, but a particularly nasty one, and one none of us should have to deal with in 2024.
We moved a project from statsd to otel last year, and I really wish we had spent that time on something else. I really wish I had gotten to spend my time in something else. Statsd makes it the aggregator’s problem to deal with stats, so application lifetime (you mention lambda) is not a problem. They can fire and forget, and at a much higher data rate than we achieved with OTEL.
The main feature of OTEL is the tagging, but Amazon charges you for using it. So much so that we had to impoverish our tags until they provided only a small factor of improvement in observability that was ultimately not worth the cost of migration. If I had straight ported, we would have generated about 40x as much traffic as we got to in the end by cutting corners.
OTEL is actively hostile to programming languages with a global interpreter lock. Languages where you run processes proportional to the number of cores on the machine need to tag each process separately. If you don’t, then the stats interfere with each other. And that brings us back to pricing by tag, because now you have 16-128x times as many combinations of tags because each process has a unique tag per machine.
We ended up having to put our own aggregator sidecar in place that could merge the counters and histograms from the same machine. If you cross your eyes it just looks like one of the statsd forks that adds tags. Which would have been so much easier for us to do.
And each process remembers every stat it has ever seen, otherwise it will self clobber. So again, 16-128x as much memory for bookkeeping unless you send them to a bookkeeping process.
They had weird memory leaks that only got sorted out in the summer. There’s a workaround, but it nearly caused a production outage for us and we violated our SLAs, which costs us money.
We also had total stats loss because the JavaScript implementation is Typescript and their typescript implementation did not assert that numeric inputs were numbers instead of numeric strings. That lead to number + string arithmetic bugs, which lead to giant numbers that OTRL choked on and dropped.
It also took a lot of work to get our sidecar to consume input from 30+ processes without dropping any. Most of our boxes run about cpucount + 1-2. We don’t have an obscene amount of telemetry, but it’s a mature app with years of “we need to track X” conversations. It wasn’t until September that I was confident we could ingest from 64 cores at once, and I have no idea how we’d handle 128. Because again, OTEL does not like more than one process that thinks it’s the “same” app, so you have to tag or centralize to disambiguate.
And the thing I hate the most about OpenTelemetry: it has One Bad Apple Syndrome. Because the stats are accumulated and sent in bulk, if it does not like one value in the update, it drops the entire message. One poison pill stat causes 100% loss of telemetry from that machine. That is a stupid fucking design decision and I want the author of that particular level of hell to feel my anger about this jackass decision with every fiber of my being. Fuck you, sir. You have no business working in standards track software. Get out.
- phillipcarter 3y ago> The main feature of OTEL is the tagging, but Amazon charges you for using it. So much so that we had to impoverish our tags until they provided only a small factor of improvement in observability that was ultimately not worth the cost of migration. If I had straight ported, we would have generated about 40x as much traffic as we got to in the end by cutting corners. This bit is really unfortunate. Hopefully not too unhelpful, but there are several other vendors out there that don't limit you like this. Other tradeoffs to be sure (no such thing as a perfect o11y tool), but I think most dedicated observability backends have moved away from this kind of pricing structure.
- hinkley 3y agoWe are also collecting on 30 second intervals instead of 10, which makes evidence of incidents a little more ambiguous. Not impossible, just slightly more stressful to handle a production incident. Like that guy you don’t hate, but are bummed he showed up to your party.
- arccy 3y agoThe collector or protocol doesn't care that you have duplicate entries (from different processes). It's your telemetry/storage backend that drops the data, like we have the same issue with datadog, fuck them for silently dropping data.
- hinkley 3y agoI’m pretty sure otelcollector is where it’s getting dropped. It sees data with the same tags coming from a few dozen processes, doesn’t know how to aggregate them, and so just does last writer wins about once a second. Makes for some very peculiar rate charts.
- arccy 3y agomight depend on your exporter but for all the ones I cared about I was inspecting the network traffic for it and it definitely sent everything