At a previous job I was asked to debug a large (inefficient) cronjob that was suddenly taking 24+ hours instead of the usual ~8 hours. (We had just migrated inf

At a previous job I was asked to debug a large (inefficient) cronjob that was suddenly taking 24+ hours instead of the usual ~8 hours. (We had just migrated infrastructures but noticed this days later) Being relatively new to that particular codebase I look at it and see nothing that stands out to me... after an unfruitful day and not wanting to get too deep in to the code without necessity I fired up a profiler. Logging (syslog) statements were taking HUGE swathes of time. Neither me nor the person supervising me could believe that was it so we put it on the back burner. The next day I take another look at the log statements, fire up a python shell and find the logging statements on that server are returning instantly 4/5 times. Every 5th (or so) time it would block for 5 seconds or more. Given the cronjob writes thousands of log statements in the course of action, this became a cause of concern. Didn't manage to look in to it deeply enough (I guess DNS caching plus crappy DNS) but the quick workaround was to toss the syslog server's address in to the hosts file, the cronjob ran 'smoothly' after that.