3 ms·

I do not now how many hours was wasted on trying to debug Java applications that did not have this principle and produced 10000s of lines of stack traces in pro

I do not now how many hours was wasted on trying to debug Java applications that did not have this principle and produced 10000s of lines of stack traces in production with cryptic exceptions.

tsimionescu 7y ago
Which principle would that be? With Java, even if you do a bad job of error handling, you usually still get at least a stack trace that you can investigate based on. If you do a bad job with error codes, you can easily be left with no way of investigating a problem other than trying to reproduce it.
- StreamBright 7y ago
  Yes? Let me tell you just one story about this. I used to work as a systems engineer for one company where we had Hadoop as the data warehouse solution. One day my co-worker decided that he is going to get into DevOps and his first thing will be to touch DNS settings. He also thought that the best course of action is to push his changes without a review. His code was deployed to staging. I was called into to investigate an outage on staging later on. Needless to say I was unaware of the changes but this is not the point. Hadoop datanodes were failing with the following: "Cannot connect to NameNode: host.name.tld port x" I was was trying to investigate why. Unfortunately I did some quick testing as root and I could connect to the host and port combo. I went through all the changes in Hadoop configuration but there was no change that could cause this. Finally (several hours later) I wrote a simple Ruby code that did the exact same thing and I run it as hdfs user. I run the code and it failed with the following: "Cannot open /etc/resolv.conf" As you can see the Java codebase was failing with the same reason but nobody bothered to check for this problem in the library so the exception that you cannot open a file resulted in a different exception. It turns out my co-worker was very eager and security aware and he changed the resolv.conf permission from 444 to 400. I also made the mistake to investigate this as root first and not going through the unrelated changes. If Hadoop's Java was decent enough it would have printed the exact error message instead of something that is related but not exactly the error you are facing. This is what I push for regardless of the language, catch the error exactly where it happens and log it there and I think this is what the parent was referring to as well.
  - tsimionescu 7y ago
    This is a good example,and the error returned by the Java DNS library is obviously wrong. However, I don't think that simply returning the file IO exception would have been the right solution. The best solution would have been for the DNS resolution error to have carried the FileIO error as a Cause. That way, code that could recover from a DNS resolution error could handle the DNS resolution problem, and code that couldn't would still log the actual root cause. The Java networking libraries seem to be particularly poorly written for DNS error handling. I encountered an error where I was opening a socket to connect to a hostname, with an explicit timeout of a few seconds (it was an external service, but the application was supposed to work offline as well). To my surprise, the call would block for a few minutes on a customer system. It turned out that the call was first resolving the hostname, and that the timeout value I supplied was not applied to the DNS resolution, which was using a system-wide default timeout. One other point about your example: I think that one thing missing from all language runtimes that I have seen is a proper logging mechanism that is truly available language-wide. Of course there is always stdout/stderr, but these are usually far too limited and it is nkt considered good form to log to them from libraries. Even if you did, you would obviously not respect the application logging format in regards to stuff like time format, timezone etc. In contrast, a standardized configurable logging mechanism would help so much with 3rd party library debugging.
- loopz 7y ago
  Stack traces are a bad way to make sure your program has sound logic flow, though may be convenient when lacking tracing and validations.
- fxtentacle 7y ago
  Except if your framework uses bytecode patching, e.g. Hibernate ERM, in which case the Stack trace shows you classes that do not exist in source code form.

I do not now how many hours was wasted on trying to debug Java applications that did not have this principle and produced 10000s of lines of stack traces in production with cryptic exceptions.

tsimionescu 7y ago

Which principle would that be? With Java, even if you do a bad job of error handling, you usually still get at least a stack trace that you can investigate based on. If you do a bad job with error codes, you can easily be left with no way of investigating a problem other than trying to reproduce it.

StreamBright 7y ago
Yes? Let me tell you just one story about this. I used to work as a systems engineer for one company where we had Hadoop as the data warehouse solution. One day my co-worker decided that he is going to get into DevOps and his first thing will be to touch DNS settings. He also thought that the best course of action is to push his changes without a review. His code was deployed to staging. I was called into to investigate an outage on staging later on. Needless to say I was unaware of the changes but this is not the point. Hadoop datanodes were failing with the following: "Cannot connect to NameNode: host.name.tld port x" I was was trying to investigate why. Unfortunately I did some quick testing as root and I could connect to the host and port combo. I went through all the changes in Hadoop configuration but there was no change that could cause this. Finally (several hours later) I wrote a simple Ruby code that did the exact same thing and I run it as hdfs user. I run the code and it failed with the following: "Cannot open /etc/resolv.conf" As you can see the Java codebase was failing with the same reason but nobody bothered to check for this problem in the library so the exception that you cannot open a file resulted in a different exception. It turns out my co-worker was very eager and security aware and he changed the resolv.conf permission from 444 to 400. I also made the mistake to investigate this as root first and not going through the unrelated changes. If Hadoop's Java was decent enough it would have printed the exact error message instead of something that is related but not exactly the error you are facing. This is what I push for regardless of the language, catch the error exactly where it happens and log it there and I think this is what the parent was referring to as well.
- tsimionescu 7y ago
  This is a good example,and the error returned by the Java DNS library is obviously wrong. However, I don't think that simply returning the file IO exception would have been the right solution. The best solution would have been for the DNS resolution error to have carried the FileIO error as a Cause. That way, code that could recover from a DNS resolution error could handle the DNS resolution problem, and code that couldn't would still log the actual root cause. The Java networking libraries seem to be particularly poorly written for DNS error handling. I encountered an error where I was opening a socket to connect to a hostname, with an explicit timeout of a few seconds (it was an external service, but the application was supposed to work offline as well). To my surprise, the call would block for a few minutes on a customer system. It turned out that the call was first resolving the hostname, and that the timeout value I supplied was not applied to the DNS resolution, which was using a system-wide default timeout. One other point about your example: I think that one thing missing from all language runtimes that I have seen is a proper logging mechanism that is truly available language-wide. Of course there is always stdout/stderr, but these are usually far too limited and it is nkt considered good form to log to them from libraries. Even if you did, you would obviously not respect the application logging format in regards to stuff like time format, timezone etc. In contrast, a standardized configurable logging mechanism would help so much with 3rd party library debugging.
loopz 7y ago
Stack traces are a bad way to make sure your program has sound logic flow, though may be convenient when lacking tracing and validations.
fxtentacle 7y ago
Except if your framework uses bytecode patching, e.g. Hibernate ERM, in which case the Stack trace shows you classes that do not exist in source code form.