4 ms·
For this: > There is only one person who this benefits and it's the person who charges you for the pipeline, which is exactly why people who sell a pipeline ar
by phillipcarter 3y ago
For this:
> There is only one person who this benefits and it's the person who charges you for the pipeline, which is exactly why people who sell a pipeline are incentivized to tell you that sampling is bad
I largely agree, and I'll say that at least with Honeycomb (since it's mentioned by the author) we make sampling a key component of pretty much any deal before anything gets signed. For small stuff this clearly doesn't matter so much, but it basically boils down to:
- Most of your data is probably uninteresting because it's uniform and represents success cases, so you just need a statistically significant sampling to get a sense of what "okay" means for comparisons
- Whenever there's an error or high latency, there's almost always something interesting in there, and you probably want all of it
And so this typically works out to generating an order of magnitude or two more data than you actually need to get an accurate view of what's going on any any point in time.
And so when you do this, you can (and probably should?) pack your events/logs/traces/whatever-you-call-it with a bunch of data.
There's some examples where you can't do this, though. Some people want to be able to do something like plug in a customer ID and dig up the exact trace that represents something they complained about. Or there's some compliance to adhere to, legal or non-legal, where it's still less money to pay for everything unsampled than it is to deal with the consequences of non-compliance. But for most organizations I'd say what I mentioned above holds true.
...but that's just one of the several ways that some folks will frame up Observability. The term has been kleenexed a bit so it now means whatever any vendor says it means, and they all say varyingly different things.