5 ms·

First paragraph https://github.com/hyperdxio/hyperdx https://github.com/hyperdxio/hyperdx "HyperDX helps engineers figure out why production is broken faster b

First paragraph https://github.com/hyperdxio/hyperdx https://github.com/hyperdxio/hyperdx "HyperDX helps engineers figure out why production is broken faster by centralizing and correlating logs, metrics, traces, exceptions and session replays in one place. An open source and developer-friendly alternative to Datadog and New Relic." Just perfect. Bravo. -- As a merc, I never understood the why of Datadog (or equiv). The teams and projects I rotated thru each embraced the "LOG ALL THE THINGS!" strategy. No guiding purpose, no esthetics. General agreement about need to improve signal to noise ratio. But little courage or gumption to act. And any such efforts would be easily rebuffed by citing the parable of Chesterfordstorm's Fences of Doom and something something about velocity. Late last century, IT projects, like CRMs and ERPs, were plagued by over collection of data. Opaque provenance, dubious (data) quality, unclear ownership, subtractive value propositions (where the whole is worth less than the parts). No, no, don't remove that field. We might need it some day. Today's "analytics" projects are the same, right? Every drive-by stakeholder tosses in a few tags, some misc fields, a little extra meta. And before anyone can say "kanban", the stone soup accreted enough mass to become a gravity well threatening implosion dragging the entire org-chart into the gapping maw of our universe's newest black hole. Am I wrong? But logging is useful, right? Or at least has that potential. The last time I designed a system end-to-end, that's kinda what we did. Listed all the kinds of things we wanted to log. Sorta settled on formats and content (never really ever done). Did regular log bashs to explain and clear anomalies. Scripts for grooming and archiving. (For one team I rotated thru, most of their spend was on just cloudwatch. Hysterical.) But my stuff wasn't B2C, so wasn't tainted by the attention economy, manufactured outrage, or recommenders. No tags, referrers, campaigns, etc. It was just about keeping the system up and true. And resolving customer support incidents asap. Does any one talk or write about this? (Those SRE themed novels are now buried deep in my to read pile.) I'd like some cookbooks or blue prints which show some idealized logging strategies, with depictions of common enough troubleshooting scenarios. Having something authoritative to cite could reduce my semblance to an Eeyore. "Hey, team mates, you know what'd be really great?! Correlation IDs! So we can see how an action percolates thru our system!" Just curious. PS- Datadog's server hexagon map/chart thingie is something else. The kind of innovation that wins prizes.

deleted 3y ago
[deleted]
mikeshi42 3y ago
Yes! It should definitely be thoughtful about what you log and how you expect to use it. My biggest gripe with logs is often people writing them never think about "how would I use this when things are on fire?" and tend to log useless information or fail to tag them in ways that are actually searchable. Tagging the right IDs are a huge thing - customer X is saying their instance is really slow, but if none of your logs let you link service performance to customer X, your telemetry you're paying for is absolutely useless! You have an ally in me on this one :) I'm hoping given a bit more time we get to write things like this - practical observability from the perspective of a dev, as opposed to the SRE angle that I think is well covered. Feel free to join us on discord btw if you want to chat more - I (for better/worse) love musing about these things :)
- specialist 3y ago
  PS- I just skimmed https://opentelemetry.io https://opentelemetry.io, which your readme.md links to. Good stuff. Much industry progress since I was last in the arena. Their site has words about manual and automatic instrumentation. I'd have to dig a bit to see what they mean. -- So. Remembering a bit more... Will try to keep this brief; you're a busy person. > tend to log useless information or fail to tag them in ways that are actually searchable #1 - I don't know know to manage lifecycle of meta. Who needs what? When is it safe to remove stuff? We logged a lot of URLs. So many URL params. And when that wasn't crazy enough, over flow into HTTP headers. Plus partially duplicate, incorrectly, info in the payloads, a la SOAP. ("A person with two watches has no idea what time it is.") When individual teams were uncertain, they'd just forward everything they received (copypasta), and add their own stuff. Just replace all that context with correlation IDs, right? Ah, but there's "legacy". And unsupported protocols, like Redis and JDBC. And brain dead 3rd party services, with their own brain dead CSRs and engrs. This is really bad, and just propagates badness, but a few times, in a pinch, I've created Q&D "logging proxy". Just to get some visibility. So dumb. And yet... Why stop there? Just have "the fabric" record stuff. Repurpose Wireguard into an Omniscient Logger. (Like the NSA does. Probably.) That'd eliminate most I/O trace style logging, right? Image all these "webservices" and serverless apps without any need for instrumentation. Just have old school app level logging. #2 - So much text processing. An egregious example is logging HTTP headers. Serialize them as JSON and send that payload to a logging service. Which then rehydrate and store it some where. My radical idea, which exactly no one has bought into, is to just pipe HTTP (Requests and Responses) as-is to log files. Then rotate, groom, archive, forward, ingest, compress, whatever as desired. That's what I did on the system I mentioned. All I/O was just streamed to files. And in the case of the HL7 (medical records stuff), it was super easy to extract the good bits, use that for Lucene's metadata, and store the whole message as the Lucene document. I know such a radical idea is out of scope for your work. Just something fun to think about. #3 > if none of your logs let you link service performance to customer X Yup. Just keep adding servers. Kick the can down the road. One team I helped had stuff randomly peg P95. And then sometimes a seemingly unrelated server would tip over. Between timeouts, retries, and load balancers, it really seemed like the ankle bone was connected to the shoulder bone. It just made no sense. Fortunately, I had some prior experience. Being new to nodejs, maybe 5 years ago, I was shocked to learn there was no notion of back pressure. It was a challenging concept to explain to those teammates. But the omission of backpressure, and a hunch, was a good place for me start. (I'm no Dan Luu or Bryan Cantrill.) I'd like to think that proper end-to-end logging, and the ability to find signal in the noise, diagnosis would have been more mundane.

deleted 3y ago

[deleted]

mikeshi42 3y ago

Yes! It should definitely be thoughtful about what you log and how you expect to use it. My biggest gripe with logs is often people writing them never think about "how would I use this when things are on fire?" and tend to log useless information or fail to tag them in ways that are actually searchable. Tagging the right IDs are a huge thing - customer X is saying their instance is really slow, but if none of your logs let you link service performance to customer X, your telemetry you're paying for is absolutely useless! You have an ally in me on this one :) I'm hoping given a bit more time we get to write things like this - practical observability from the perspective of a dev, as opposed to the SRE angle that I think is well covered. Feel free to join us on discord btw if you want to chat more - I (for better/worse) love musing about these things :)

specialist 3y ago
PS- I just skimmed https://opentelemetry.io https://opentelemetry.io, which your readme.md links to. Good stuff. Much industry progress since I was last in the arena. Their site has words about manual and automatic instrumentation. I'd have to dig a bit to see what they mean. -- So. Remembering a bit more... Will try to keep this brief; you're a busy person. > tend to log useless information or fail to tag them in ways that are actually searchable #1 - I don't know know to manage lifecycle of meta. Who needs what? When is it safe to remove stuff? We logged a lot of URLs. So many URL params. And when that wasn't crazy enough, over flow into HTTP headers. Plus partially duplicate, incorrectly, info in the payloads, a la SOAP. ("A person with two watches has no idea what time it is.") When individual teams were uncertain, they'd just forward everything they received (copypasta), and add their own stuff. Just replace all that context with correlation IDs, right? Ah, but there's "legacy". And unsupported protocols, like Redis and JDBC. And brain dead 3rd party services, with their own brain dead CSRs and engrs. This is really bad, and just propagates badness, but a few times, in a pinch, I've created Q&D "logging proxy". Just to get some visibility. So dumb. And yet... Why stop there? Just have "the fabric" record stuff. Repurpose Wireguard into an Omniscient Logger. (Like the NSA does. Probably.) That'd eliminate most I/O trace style logging, right? Image all these "webservices" and serverless apps without any need for instrumentation. Just have old school app level logging. #2 - So much text processing. An egregious example is logging HTTP headers. Serialize them as JSON and send that payload to a logging service. Which then rehydrate and store it some where. My radical idea, which exactly no one has bought into, is to just pipe HTTP (Requests and Responses) as-is to log files. Then rotate, groom, archive, forward, ingest, compress, whatever as desired. That's what I did on the system I mentioned. All I/O was just streamed to files. And in the case of the HL7 (medical records stuff), it was super easy to extract the good bits, use that for Lucene's metadata, and store the whole message as the Lucene document. I know such a radical idea is out of scope for your work. Just something fun to think about. #3 > if none of your logs let you link service performance to customer X Yup. Just keep adding servers. Kick the can down the road. One team I helped had stuff randomly peg P95. And then sometimes a seemingly unrelated server would tip over. Between timeouts, retries, and load balancers, it really seemed like the ankle bone was connected to the shoulder bone. It just made no sense. Fortunately, I had some prior experience. Being new to nodejs, maybe 5 years ago, I was shocked to learn there was no notion of back pressure. It was a challenging concept to explain to those teammates. But the omission of backpressure, and a hunch, was a good place for me start. (I'm no Dan Luu or Bryan Cantrill.) I'd like to think that proper end-to-end logging, and the ability to find signal in the noise, diagnosis would have been more mundane.