5 ms·

The problem is that in order to add a missing log in production you need to effectively redeploy. That isn't tenable for a polyglot production cluster. There's

The problem is that in order to add a missing log in production you need to effectively redeploy. That isn't tenable for a polyglot production cluster. There's a rising field of developer observability which is specifically designed to provide polyglot cloud "debugging" capabilities. One of the capabilities is to inject new logs dynamically.

geophile 4y ago
And if you need to fix a bug in production you need to redeploy too. Bugs and logs and testing are all related. You obviously (should) do unit testing, integration testing, system testing before production. As you do these things, you will find and fix bugs. Ideally, you would add logging to help you diagnose the bugs you are fixing. Congratulations, you now have logging in code known to have been buggy, which will serve you well in production if bugs not caught by your tests turn up. But bugs in previously reliable code do slip through, and this gives rise to the need for previously unanticipated logging. If either of these things happens a lot, the problem is your testing leading up to production. In other words, if you often find yourself needing dynamically deployed logging, then the lack of this feature is not the problem you should be paying attention to. Also, dynamically deployed logging is basically shipping untested code, no?
- invalidname 4y ago
  The difference between deploying once and deploying just to add more logs. And more logs is a huge difference. I don't know about your environment but I can't push something to production on my own schedule. I need approval, process, etc. This is painful to do just to add a log which might not include all the information I need.
  - geophile 4y ago
    The more constraints you have on pushing something to production, the less it makes sense to deploy logging dynamically! Those constraints are there to minimize risk, ensure quality and security, etc. Dynamically deployed logging seems to provide an extremely large loophole. For example, consider the recent Java logging fiasco.
    - invalidname 4y ago
      Excellent point. The tool we use (Lightun) runs all of that in a read-only sandbox and limits the classes you can access (they are in a sandbox too) so you can't print "anything". It throttles your access so you won't use too much CPU with expensive operations too. It also comes with PII reduction and block lists so you won't just log user passwords in the login code... This solves the problem of over-logging: https://www.reddit.com/r/devops/comments/udgohy/there_is_no_such_thing_as_too_much_logging_or_is/ https://www.reddit.com/r/devops/comments/udgohy/there_is_no_...
icedchai 4y ago
Not necessarily. I've been at some places where you just "do it live." We had dozens of worker nodes. An engineer would "reserve" one for debugging purposes, editing python code live, adding logs or prints as needed and restarting services. Worst case a small percentage of requests was lost, though generally they would be retried except in the most extreme cases. This risk was considered acceptable. Obviously, depending on the business and maturity level of the company, this could be a big no-no. In general, I find those "wild west" environments more fun. Waiting for an hour long build, deploys, and "approvals" to add a couple of prints is a big drag.
- invalidname 4y ago
  For that you have developer observability which doesn't carry the HUGE gaping security risks or performance problems. But still lets you debug the cluster at scale.
  - icedchai 4y ago
    I’ve worked in that space. Observability tools may not give you all you need to debug certain classes of problems. I saw your other comments about Lightrun and will check it out though!

geophile 4y ago

And if you need to fix a bug in production you need to redeploy too. Bugs and logs and testing are all related. You obviously (should) do unit testing, integration testing, system testing before production. As you do these things, you will find and fix bugs. Ideally, you would add logging to help you diagnose the bugs you are fixing. Congratulations, you now have logging in code known to have been buggy, which will serve you well in production if bugs not caught by your tests turn up. But bugs in previously reliable code do slip through, and this gives rise to the need for previously unanticipated logging. If either of these things happens a lot, the problem is your testing leading up to production. In other words, if you often find yourself needing dynamically deployed logging, then the lack of this feature is not the problem you should be paying attention to. Also, dynamically deployed logging is basically shipping untested code, no?

invalidname 4y ago
The difference between deploying once and deploying just to add more logs. And more logs is a huge difference. I don't know about your environment but I can't push something to production on my own schedule. I need approval, process, etc. This is painful to do just to add a log which might not include all the information I need.
- geophile 4y ago
  The more constraints you have on pushing something to production, the less it makes sense to deploy logging dynamically! Those constraints are there to minimize risk, ensure quality and security, etc. Dynamically deployed logging seems to provide an extremely large loophole. For example, consider the recent Java logging fiasco.
  - invalidname 4y ago
    Excellent point. The tool we use (Lightun) runs all of that in a read-only sandbox and limits the classes you can access (they are in a sandbox too) so you can't print "anything". It throttles your access so you won't use too much CPU with expensive operations too. It also comes with PII reduction and block lists so you won't just log user passwords in the login code... This solves the problem of over-logging: https://www.reddit.com/r/devops/comments/udgohy/there_is_no_such_thing_as_too_much_logging_or_is/ https://www.reddit.com/r/devops/comments/udgohy/there_is_no_...

icedchai 4y ago

Not necessarily. I've been at some places where you just "do it live." We had dozens of worker nodes. An engineer would "reserve" one for debugging purposes, editing python code live, adding logs or prints as needed and restarting services. Worst case a small percentage of requests was lost, though generally they would be retried except in the most extreme cases. This risk was considered acceptable. Obviously, depending on the business and maturity level of the company, this could be a big no-no. In general, I find those "wild west" environments more fun. Waiting for an hour long build, deploys, and "approvals" to add a couple of prints is a big drag.

invalidname 4y ago
For that you have developer observability which doesn't carry the HUGE gaping security risks or performance problems. But still lets you debug the cluster at scale.
- icedchai 4y ago
  I’ve worked in that space. Observability tools may not give you all you need to debug certain classes of problems. I saw your other comments about Lightrun and will check it out though!