10 ms·

You can't debug distributed systems with dashboards

suff 8y ago
You can if your loggung structure is uniform, and includes a correlation ID. This happens every day... pretty simple actually.
- ryanworl 8y ago
  This is literally the point of the article. Charity is saying to structure your logs, add useful information to them from all portions of the transaction, and save the logs somewhere to query them across all fields in any combination. That's what her product does. This is in opposition to poor quality text logs, which typically contain almost no useful information, and time series aggregates, which either throw away all the context, or explode traditional time series databases when you try to add high cardinality fields to them. Edit: I should add that "dashboards" in this context means the typical wall of time series charts you can buy from a bunch of different vendors.
  - fizx 8y ago
    Title is a bit clickbait, really.
- icedchai 8y ago
  Sounds simple. Unfortunately, having uniform logging structure in a non-trivial project is about as likely to happen as winning the lottery.
  - drb91 8y ago
    ...only if you view software development like gambling. All problems can be methodically addressed; fixing log formatting is actually rather easy to track down.
    - icedchai 8y ago
      Yes, in your own code, it is easily fixed... With third party code, like dependencies, it becomes more of a problem...
      - drb91 8y ago
        That's true. But you only need to re-format the events you care about from that dependency.
        Fellshard 8y ago
        That's still a non-trivial problem, depending on how easy it is to adapt logs from a library; maybe with a proper logging facade, it's easy to attach those adapters to those logs, but I suspect most libraries don't have strong contracts around the formats of their logs, as it is. Maybe that's something to improve.
  - scarface74 8y ago
    I use .Net and Serilog for structured logging. Each log entry can be a structure with name/value pairs instead of just a stream of text. Most of the time it’s JSON. You have various outputs (sinks) where you can send your logs to almost anywhere - AWS CloudWatch to its own group/stream where you can trigger alerts, a NoSql database like ElasticSearch or Mongo where you can do queries against properties. You can also have every log entry tagged with metadata like the application name/EC2 instance, AMI being used either globally or within a “context” (everything that’s called inside a using block). Why anyone would do unstructured logging in 2018 is beyond me.
  - lallysingh 8y ago
    No it's not. You can start with something that snarfs data from your log messages and normalizes them. Then you just have to log enough data that you can extract it after. The normalization can connect IDs together to make the log data more accessible.
    - icedchai 8y ago
      Now I have two problems: my app and a logging thing.
k__ 8y ago
I'm still wating for the day when these products use serverless-pricing too.
- scarface74 8y ago
  With CloudWatch you can literally do everything the article proposes just by sending your logs there - you get easy to create dashboards and alerts.
  - k__ 8y ago
    Nice, thanks for the info :) Maybe I should to a course on CloudWatch. Seems like a powerful tool, but a bit convulted at times...
drb91 8y ago
What’s the pricing for the pitched product?
- bboreham 8y ago
  $70 per month and up, based on GB/day transmitted. https://www.honeycomb.io/pricing/ https://www.honeycomb.io/pricing/
draw_down 8y ago
I don’t think anyone tries to do that? Tools like Splunk and Sentry are a lot better to stare at if you’re trying to debug a problem.
kozak 8y ago
Such a massive unscrollable top bar that I had to open my browser's Dev Tools and remove it from the DOM it before reading the article.
hinkley 8y ago
Console logging is the easiest but least effective method of debugging. Some people refuse to call it debugging because they believe a debugger or diagnostic tools are involved with the act of “debugging” (vs troubleshooting?) Distributed systems mean you have to fall back to the most basic form of debugging. You had better be getting something really good for that bargain. You’ve nearly permanently compromised your ability to reason about obscure bugs in the system in order to have it, so be sure it was worth it.
- daotoad 8y ago
  You need to take that basic form of "debugging" and turn it into instrumentation. Collect structured data and build tools to process it. Once you have a nice collection of data you can build profilers and execution traces, things classical debugging tools provide. I've done it and the tools can work quite well. The same consistent instrumentation of your code also makes it easy to build really great dashboard monitoring. All this time building tools costs money and adds complexity to your stack, so, yes, distribution had better be worth it for your business needs before you take it on. Pretty much like any engineering trade-off.
  - orweis 8y ago
    Do you mean something like this? (Rookout.com) https://youtu.be/bAxNWg6QYL8?t=62 https://youtu.be/bAxNWg6QYL8?t=62
peterwwillis 8y ago
Yeah, I'm sorry, but this is ridiculous clickbait written by hipster sysengs using fringe architectures to generalize about a gigantic range of complex systems and saying "all you need to do is sample some logs, guys, why are you still using dashboards". You know what solves ops issues? System visibility connected to automation designed to diagnose root causes and provide simple solutions to common problems. Serverless adds extra complexity and points of failure, but that doesn't mean "you can't debug distributed systems with dashboards". Give me a break.
- fhood 8y ago
  I clicked on that link ready to experience a revelation that was going to change how I debugged infrastructure problems. I came away with nothing. I got nothing out of this article. I have re-read it twice now hoping that I was missing something, or had failed to understand some point.
  - peterwwillis 8y ago
    The only point was that distributed tracing apparently solves all problems.
- dsfyu404ed 8y ago
  >System visibility connected to automation designed to diagnose root causes and provide simple solutions to common problems. This. At the end of the say you gotta put the right stack trace in front of the right eyeballs so someone can connect the dots and figure out why your crap is burning down. Automation helps people who are not intimately familiar with every aspect of each system do that.
- tomatotomato37 8y ago
  >but this is ridiculous clickbait written by hipster sysengs using fringe architectures to generalize about a gigantic range of complex systems To be honest that can describe a lot of articles here

suff 8y ago

You can if your loggung structure is uniform, and includes a correlation ID. This happens every day... pretty simple actually.

ryanworl 8y ago
This is literally the point of the article. Charity is saying to structure your logs, add useful information to them from all portions of the transaction, and save the logs somewhere to query them across all fields in any combination. That's what her product does. This is in opposition to poor quality text logs, which typically contain almost no useful information, and time series aggregates, which either throw away all the context, or explode traditional time series databases when you try to add high cardinality fields to them. Edit: I should add that "dashboards" in this context means the typical wall of time series charts you can buy from a bunch of different vendors.
- fizx 8y ago
  Title is a bit clickbait, really.
icedchai 8y ago
Sounds simple. Unfortunately, having uniform logging structure in a non-trivial project is about as likely to happen as winning the lottery.
- drb91 8y ago
  ...only if you view software development like gambling. All problems can be methodically addressed; fixing log formatting is actually rather easy to track down.
  - icedchai 8y ago
    Yes, in your own code, it is easily fixed... With third party code, like dependencies, it becomes more of a problem...
    - drb91 8y ago
      That's true. But you only need to re-format the events you care about from that dependency.
      - Fellshard 8y ago
        That's still a non-trivial problem, depending on how easy it is to adapt logs from a library; maybe with a proper logging facade, it's easy to attach those adapters to those logs, but I suspect most libraries don't have strong contracts around the formats of their logs, as it is. Maybe that's something to improve.
- scarface74 8y ago
  I use .Net and Serilog for structured logging. Each log entry can be a structure with name/value pairs instead of just a stream of text. Most of the time it’s JSON. You have various outputs (sinks) where you can send your logs to almost anywhere - AWS CloudWatch to its own group/stream where you can trigger alerts, a NoSql database like ElasticSearch or Mongo where you can do queries against properties. You can also have every log entry tagged with metadata like the application name/EC2 instance, AMI being used either globally or within a “context” (everything that’s called inside a using block). Why anyone would do unstructured logging in 2018 is beyond me.
- lallysingh 8y ago
  No it's not. You can start with something that snarfs data from your log messages and normalizes them. Then you just have to log enough data that you can extract it after. The normalization can connect IDs together to make the log data more accessible.
  - icedchai 8y ago
    Now I have two problems: my app and a logging thing.

k__ 8y ago

I'm still wating for the day when these products use serverless-pricing too.

scarface74 8y ago
With CloudWatch you can literally do everything the article proposes just by sending your logs there - you get easy to create dashboards and alerts.
- k__ 8y ago
  Nice, thanks for the info :) Maybe I should to a course on CloudWatch. Seems like a powerful tool, but a bit convulted at times...

drb91 8y ago

What’s the pricing for the pitched product?

bboreham 8y ago
$70 per month and up, based on GB/day transmitted. https://www.honeycomb.io/pricing/ https://www.honeycomb.io/pricing/

draw_down 8y ago

I don’t think anyone tries to do that? Tools like Splunk and Sentry are a lot better to stare at if you’re trying to debug a problem.

kozak 8y ago

Such a massive unscrollable top bar that I had to open my browser's Dev Tools and remove it from the DOM it before reading the article.

hinkley 8y ago

Console logging is the easiest but least effective method of debugging. Some people refuse to call it debugging because they believe a debugger or diagnostic tools are involved with the act of “debugging” (vs troubleshooting?) Distributed systems mean you have to fall back to the most basic form of debugging. You had better be getting something really good for that bargain. You’ve nearly permanently compromised your ability to reason about obscure bugs in the system in order to have it, so be sure it was worth it.

daotoad 8y ago
You need to take that basic form of "debugging" and turn it into instrumentation. Collect structured data and build tools to process it. Once you have a nice collection of data you can build profilers and execution traces, things classical debugging tools provide. I've done it and the tools can work quite well. The same consistent instrumentation of your code also makes it easy to build really great dashboard monitoring. All this time building tools costs money and adds complexity to your stack, so, yes, distribution had better be worth it for your business needs before you take it on. Pretty much like any engineering trade-off.
- orweis 8y ago
  Do you mean something like this? (Rookout.com) https://youtu.be/bAxNWg6QYL8?t=62 https://youtu.be/bAxNWg6QYL8?t=62

peterwwillis 8y ago

Yeah, I'm sorry, but this is ridiculous clickbait written by hipster sysengs using fringe architectures to generalize about a gigantic range of complex systems and saying "all you need to do is sample some logs, guys, why are you still using dashboards". You know what solves ops issues? System visibility connected to automation designed to diagnose root causes and provide simple solutions to common problems. Serverless adds extra complexity and points of failure, but that doesn't mean "you can't debug distributed systems with dashboards". Give me a break.

fhood 8y ago
I clicked on that link ready to experience a revelation that was going to change how I debugged infrastructure problems. I came away with nothing. I got nothing out of this article. I have re-read it twice now hoping that I was missing something, or had failed to understand some point.
- peterwwillis 8y ago
  The only point was that distributed tracing apparently solves all problems.
dsfyu404ed 8y ago
>System visibility connected to automation designed to diagnose root causes and provide simple solutions to common problems. This. At the end of the say you gotta put the right stack trace in front of the right eyeballs so someone can connect the dots and figure out why your crap is burning down. Automation helps people who are not intimately familiar with every aspect of each system do that.
tomatotomato37 8y ago
>but this is ridiculous clickbait written by hipster sysengs using fringe architectures to generalize about a gigantic range of complex systems To be honest that can describe a lot of articles here