4 ms·
Rug Driven Development causes the majority of technical debt that I see in most of my clients, and it happens at the small scale, too. Just a couple weeks ago,
by yourapostasy 6y ago
Rug Driven Development causes the majority of technical debt that I see in most of my clients, and it happens at the small scale, too. Just a couple weeks ago, a junior engineer was faced with a key process that segfaulted about once a minute. No worries though, they reasoned; the process immediately respawned and because it managed a stateless procedure, nothing is lost. Except the 1 out ~30,000 times it didn't respawn. No worries though, they reasoned; they put in a request to the monitoring team to look for the absence of the process for longer than three minutes and auto-spawn it. Ta da! Problem solved!
They let this go on for about six years before I arrived and saw it. It took me pointing out that there is a decent chance this is caused by the process relying upon something in the OS kernel that isn't quite aligned to the kernel documentation but is good enough, and the immediate respawn behavior might rely upon that in turn. If the kernel ever is "fixed", then this audit-compliance-related process suddenly stops and doesn't respawn, or worse, won't start at all, becoming top of mind with all the senior management. It is always cheaper to fix problems in the small before they become problems no one can ignore.
The urge to sweep problems under the rug and move on is very powerful within our industry. Until you've done the same yourself and been bitten so many times, your scar tissue twitches every time you see the same pattern again. These days, I treat code problems like I treat cleaning up messes while cooking: I clean up as I go along. My scar tissue thanks me now. There is a delicate balance between addressing these problems in the small and bikeshedding, though.
BTW, I performed an strace that revealed a SIGKILL just pops in without anyone or any known process issuing it. The application developers suspect something in the OS, so we're now engaged with the application support team, OS support team, and our own internal OS support team to trace it down and beat it into submission.
- rwmj 6y agoIf it's Linux there are various ways to find out who is sending that signal, killsnoop being one (part of bcc: https://github.com/iovisor/bcc https://github.com/iovisor/bcc), another using a trivial systemtap script (https://www.ibm.com/support/pages/systemtap-kill-who-killed-my-process https://www.ibm.com/support/pages/systemtap-kill-who-killed-...).
- yourapostasy 6y agoI <3 Linux for just this kind of community. Thank you, you're righteous!