4 ms·

This has been my experience. The pieces are simple, but being to able to understand what is going on in the whole system is not. Detailed, quality structured lo

by gbuk2013 7y ago

This has been my experience. The pieces are simple, but being to able to understand what is going on in the whole system is not. Detailed, quality structured logging is very important, sent to a central location that can be queried (e.g Elastic Search / Kibana). I have come across several experienced developers who really don’t like he idea of verbose logging but in my experience it has been critically important for dealing with a mesh of micro services.

stingraycharles 7y ago
Yep, you never know what logs you need until you need it. If there’s an economical factor coming into play (because proper telemetry will cost a noticeable amount of money), you can always choose to sample the more verbose logs.
- ignoramous 7y ago
  Honest Q: Isn't that's where tracers like AWS XRay and Google Dapper come in? And somewhere down the line, solutions like Envoy/istio try to make the whole ordeal manageable?
  - gbuk2013 7y ago
    Nothing cloud is relevant when your production network has no outbound access to the internet due to PCI. ;) Not sure about the other two (will check it out) but we have have apps in 4 programming languages across several teams. Structured (JSON) logging is the easiest way of sending data to one place from literally everywhere.
  - dropofwill 7y ago
    You can implement distributed tracing with logs if every player in the system is propagating the headers appropriately. Tools like NewRelic APM and XRay will take care of that bit mostly for you. Service mesh data planes have a wider scope, but they do make it much easier to implement distributed tracing (I think Istio includes this out of the box). In the end though these tools can’t replace app logs because they only let you reconstruct what happens in between services, not internal to them.
- gbuk2013 7y ago
  Agree 100% - logging is expensive, in terms of performance hit and storage, but it is totally worth it.
  - QuickToBan 7y ago
    In the medium to long term, the absence of logging is always more expensive than its presence.
alexbanks 7y ago
"How does this even get invoked" and "Where would I go to find these logs" are the two questions that, if the answers are not obvious/obviously documented, there is a big problem with that particular system. My current company for whatever reason does not value logging at all, to the point where for lots of our systems there just aren't logs/nobody has ever bothered to look for them. It's pretty astonishing to me, a person who has valued logging as one of the highest-value mechanisms to increase tracibility.
- gbuk2013 7y ago
  Prior to moving to software development, I worked 5 years in Support and consequently I have a murderous hatred of anyone writing software that is hard to debug and logging is a huge part of that. I was actually greatly inspired by a product I used to support which had such detailed logs that you could troubleshoot almost any issue from the (rather huge) log dump. I try to do the same now.
  - yjftsjthsd-h 7y ago
    I really wish all developers had to do support at the beginning of their careers and so they understand how to build things that the ops team can work with. Devops/SREs are one approach to this, but I'd really like to see it be more widespread.
  - alexanderdmitri 7y ago
    I share your enthusiam for good logs, so am curious what made this product's logging great from a support perspective?
- cbetti 7y ago
  I wonder if they simply haven't seen a strong application of logging, and therefore don't know what they're missing. Is there an opportunity in your daily work to sneak some good logging into a complex subsystem to show how valuable it can be?
  - QuickToBan 7y ago

stingraycharles 7y ago

Yep, you never know what logs you need until you need it. If there’s an economical factor coming into play (because proper telemetry will cost a noticeable amount of money), you can always choose to sample the more verbose logs.

ignoramous 7y ago
Honest Q: Isn't that's where tracers like AWS XRay and Google Dapper come in? And somewhere down the line, solutions like Envoy/istio try to make the whole ordeal manageable?
- gbuk2013 7y ago
  Nothing cloud is relevant when your production network has no outbound access to the internet due to PCI. ;) Not sure about the other two (will check it out) but we have have apps in 4 programming languages across several teams. Structured (JSON) logging is the easiest way of sending data to one place from literally everywhere.
- dropofwill 7y ago
  You can implement distributed tracing with logs if every player in the system is propagating the headers appropriately. Tools like NewRelic APM and XRay will take care of that bit mostly for you. Service mesh data planes have a wider scope, but they do make it much easier to implement distributed tracing (I think Istio includes this out of the box). In the end though these tools can’t replace app logs because they only let you reconstruct what happens in between services, not internal to them.
gbuk2013 7y ago
Agree 100% - logging is expensive, in terms of performance hit and storage, but it is totally worth it.
- QuickToBan 7y ago
  In the medium to long term, the absence of logging is always more expensive than its presence.

alexbanks 7y ago

"How does this even get invoked" and "Where would I go to find these logs" are the two questions that, if the answers are not obvious/obviously documented, there is a big problem with that particular system. My current company for whatever reason does not value logging at all, to the point where for lots of our systems there just aren't logs/nobody has ever bothered to look for them. It's pretty astonishing to me, a person who has valued logging as one of the highest-value mechanisms to increase tracibility.

gbuk2013 7y ago
Prior to moving to software development, I worked 5 years in Support and consequently I have a murderous hatred of anyone writing software that is hard to debug and logging is a huge part of that. I was actually greatly inspired by a product I used to support which had such detailed logs that you could troubleshoot almost any issue from the (rather huge) log dump. I try to do the same now.
- yjftsjthsd-h 7y ago
  I really wish all developers had to do support at the beginning of their careers and so they understand how to build things that the ops team can work with. Devops/SREs are one approach to this, but I'd really like to see it be more widespread.
- alexanderdmitri 7y ago
  I share your enthusiam for good logs, so am curious what made this product's logging great from a support perspective?
cbetti 7y ago
I wonder if they simply haven't seen a strong application of logging, and therefore don't know what they're missing. Is there an opportunity in your daily work to sneak some good logging into a complex subsystem to show how valuable it can be?
- QuickToBan 7y ago