4 ms·
The collector or protocol doesn't care that you have duplicate entries (from different processes). It's your telemetry/storage backend that drops the data, like
by arccy 3y ago
The collector or protocol doesn't care that you have duplicate entries (from different processes). It's your telemetry/storage backend that drops the data, like we have the same issue with datadog, fuck them for silently dropping data.
- hinkley 3y agoI’m pretty sure otelcollector is where it’s getting dropped. It sees data with the same tags coming from a few dozen processes, doesn’t know how to aggregate them, and so just does last writer wins about once a second. Makes for some very peculiar rate charts.
- arccy 3y agomight depend on your exporter but for all the ones I cared about I was inspecting the network traffic for it and it definitely sent everything
- hinkley 3y agoIt wasn't dropped or not dropped that was the problem, it was clobbering. If one connection said it had seen 200 events, and another 190, it ping ponged between them instead of deciding 200+190 = 390. What keeps them from clobbering is distinct tags per connection. If you're running Ruby or Python or Node, that's one connection per thread, and that's ridiculously expensive.