3 ms·

> what do you do when you can not record everything you want to The thesis of the article is that you should use events as the source of truth, and derive logs

> what do you do when you can not record everything you want to The thesis of the article is that you should use events as the source of truth, and derive logs, metrics, and traces from them. I see nothing logically or spiritually wrong with that fundamental approach and it's been Honeycomb's approach since Day 1. I do feel like Charity is ignoring the elephant in the room of transit and storage cost: "Logs are infinitely more powerful, useful and cost-effective than metrics." Weeeelllllll.... Honeycomb has been around a while. If transit and storage were free, using Honeycomb and similar solutions would be a no-brainer. Storage is cheap, but probably not as cheap as it ought to be in the cloud.[1] And certain kinds of transit are still pretty pricey in the cloud. Even if you get transit for free by keeping it local, using your primary network interface for shipping events reduces the amount of bandwidth remaining for the primary purpose of doing real work (i.e., handling requests). Plus, I think people are aware--even if they don't specifically say so--that data that is processed, stored, and never used again is waste. Since we can't have perfect prior knowledge whether some data will be valuable later, the logical thing course of action is to retain everything. But since doing so has a cost, people will naturally tend towards trying to capture as little as they can get away with yet still be able to do their job. [1] I work for AWS, but any opinions stated herein are mine and not necessarily those of my colleagues or the company I work for.

Veserv 2y ago
Of course everything is events. The distinction is how do and should they degrade when you can not record everything (and secondarily how we should interpret and correlate them, though that is more in the structured/schema aspect). I personally believe you should deploy systems with robust, comprehensive, automatic instrumentation that you analyze offline, which appears to match what the author is advocating for (except maybe the automatic part). But, that is orthogonal to the author’s claim that it is some sort of groundbreaking revolution in observability. It is not, it is just a scheme that assumes you only need to tread the happy path. However, if you are able to stay on the happy path, then you should absolutely prefer a scheme in the same general vicinity as what the article proposes. Personally, the things the author links to leads me to question the efficiency of the implementations (compute, bandwidth, storage bottlenecks become worse with inefficient implementations) which makes it harder to stay on the happy path, but the core idea is very effective on the happy path.
rustybolt 2y ago
> The thesis of the article is that you should use events as the source of truth, and derive logs, metrics, and traces from them. That was the main point? Man, the article goes on and on about definitions and semantic versioning and how we should call observability "olly", but glad to hear there's actually a coherent thought behind it. Also, I don't think the "it's too expensive to log everything" argument should be true. Storage is insanely cheap and you can fit a whole lot of information inside a single MB (even more so when using compression).

Veserv 2y ago

Of course everything is events. The distinction is how do and should they degrade when you can not record everything (and secondarily how we should interpret and correlate them, though that is more in the structured/schema aspect). I personally believe you should deploy systems with robust, comprehensive, automatic instrumentation that you analyze offline, which appears to match what the author is advocating for (except maybe the automatic part). But, that is orthogonal to the author’s claim that it is some sort of groundbreaking revolution in observability. It is not, it is just a scheme that assumes you only need to tread the happy path. However, if you are able to stay on the happy path, then you should absolutely prefer a scheme in the same general vicinity as what the article proposes. Personally, the things the author links to leads me to question the efficiency of the implementations (compute, bandwidth, storage bottlenecks become worse with inefficient implementations) which makes it harder to stay on the happy path, but the core idea is very effective on the happy path.

rustybolt 2y ago

> The thesis of the article is that you should use events as the source of truth, and derive logs, metrics, and traces from them. That was the main point? Man, the article goes on and on about definitions and semantic versioning and how we should call observability "olly", but glad to hear there's actually a coherent thought behind it. Also, I don't think the "it's too expensive to log everything" argument should be true. Storage is insanely cheap and you can fit a whole lot of information inside a single MB (even more so when using compression).