3 ms·
We built a centralized tracing/logging infrastructure that all of our environments talk to - including local developer environments. This is a 100% in-house cre
by bob1029 5y ago
We built a centralized tracing/logging infrastructure that all of our environments talk to - including local developer environments. This is a 100% in-house creation which is purpose-built to convey important facts specific to our product.
Anyone from developer to end user is able to submit a trace to this system simply by selecting a menu option and entering a comment. These traces contain sufficient information to perfectly reconstruct all user interactions and review all communications with external systems.
There is a dashboard that updates in real-time containing all of these submissions, so everyone on the team typically has this off to the side for a quick glance throughout the day.
The amount of time this sort of tooling has bought us is hard to quantify, but I can't imagine this stuff hasn't paid for itself yet. I honestly don't see how you could what we did by just taping together 3rd party solutions. Certainly, not when also considering the security & deployment model we have to cope with due to banking industry regulations.
- lawrjone 5y agoThat's really interesting! In terms of incident.io, we're a team of 3 engineers (looking for more!) and a company of 6 in total, so our priority was on buy and not build. But in my most recent experience at a larger company (fintech unicorn, ~1k employees) we: - Used Sentry to capture exceptions from all over the app. Our language was Ruby, which meant the stacktraces were usually explanatory, and attached debugging information to the event whenever we sent it - We didn't use tracing much, though when we did it would be at a request level. Our Sentries would link to the trace, and you'd see what happened during your request - All logs were associated with the trace So our error report would be a Sentry, which could be joined against the logs and trace information. In the situation where error reporting (probably better phrased as crash report?) was triggered by a user, it would normally come from a frontend site and trigger a Sentry. The Sentry plugin will have tracked the console.log/debug calls up until that moment, and any requests that took place. So it was quite easy to piece together what had happened, and why it went wrong. Not ever seen anything so custom as what you describe, though. Interesting to know that those systems are out there, and that you feel it worth building over buying!
- wdb 5y agoThe Opentelemetry community is currently working on defining a specification for how to use OTLP in such manner that user interaction (RUG) can be 'traced too. See more details: https://docs.google.com/document/d/16Vsdh-DM72AfMg_FIt9yT9ExEWF4A_vRbQ3jRNBe09w/edit https://docs.google.com/document/d/16Vsdh-DM72AfMg_FIt9yT9Ex...