3 ms·

Interesting that he considers logs to not be worth the investment. I can appreciate his argument and all, and logging seems pretty hard to get right, but still.

Interesting that he considers logs to not be worth the investment. I can appreciate his argument and all, and logging seems pretty hard to get right, but still. Thankfully, my current job is a monolith database and webserver on a box, so I'm still in the realm of pointers for now.

tristor 10y ago
You misunderstood what he's saying here. He's not saying logs aren't worth the investment, he's saying that logging is rarely done usefully/correctly, so don't have an over reliance on log output. Instead, you should track and rely on system-wide metrics. Of course you should put the time in to do logging, but at every point of control over what the contents of the logs are you should be ensuring that what you log is information about the state of the system, not picking a class of errors you think is possible at the time and logging only that. If you've worked with any "enterprise" type Java software that ends up in a distributed system, you'll know exactly what he means. Your logs end up filled with huge XML outputs that are completely irrelevant about one particular error that actually doesn't really affect system state overall, while the things that plague you in a global way are almost impossible to hunt down without building a test cluster and running captured requests through it. Logging request/response, logging process steps by request ID, etc. are much more useful. Having the IDs in your logs correlate to the IDs in your data is also helpful because it allows your Ops people to do queries across both datasets to find common factors for where requests fail and what that type of data looks like vs requests that succeed, as an example. As he mentions though, most people don't think it's important to log successes, but in fact what you care most about is correlating request types/request parameters to metrics. Whether a request failed or succeeded is just a metric itself, but the more data you have the better understanding of the system you gain. The real challenge is that distributed systems are designed to be opaque in their complexity to the users external to the system and this often leads to them being opaque in their complexity to the developers internal to the system as well.
- yumaikas 10y ago
  Thanks for the clarification on all that. You have some good "simple" (but not obvious) ideas there too, like the correlated ids in logs

tristor 10y ago

You misunderstood what he's saying here. He's not saying logs aren't worth the investment, he's saying that logging is rarely done usefully/correctly, so don't have an over reliance on log output. Instead, you should track and rely on system-wide metrics. Of course you should put the time in to do logging, but at every point of control over what the contents of the logs are you should be ensuring that what you log is information about the state of the system, not picking a class of errors you think is possible at the time and logging only that. If you've worked with any "enterprise" type Java software that ends up in a distributed system, you'll know exactly what he means. Your logs end up filled with huge XML outputs that are completely irrelevant about one particular error that actually doesn't really affect system state overall, while the things that plague you in a global way are almost impossible to hunt down without building a test cluster and running captured requests through it. Logging request/response, logging process steps by request ID, etc. are much more useful. Having the IDs in your logs correlate to the IDs in your data is also helpful because it allows your Ops people to do queries across both datasets to find common factors for where requests fail and what that type of data looks like vs requests that succeed, as an example. As he mentions though, most people don't think it's important to log successes, but in fact what you care most about is correlating request types/request parameters to metrics. Whether a request failed or succeeded is just a metric itself, but the more data you have the better understanding of the system you gain. The real challenge is that distributed systems are designed to be opaque in their complexity to the users external to the system and this often leads to them being opaque in their complexity to the developers internal to the system as well.

yumaikas 10y ago
Thanks for the clarification on all that. You have some good "simple" (but not obvious) ideas there too, like the correlated ids in logs