5 ms·

"That's okay!" Why is that Ok? How does the dev know that they've not undone anything else? Also, how does the dev know that the fix is complete? Or that it ca

"That's okay!" Why is that Ok? How does the dev know that they've not undone anything else? Also, how does the dev know that the fix is complete? Or that it caters to the defect? This anxiety of pushing out at fix - at the risk of undoing other working functionality - ought to be addressed first. It is better (and safer) to get into the habit of writing a test to reproduce the defect and then write the fix. After all, if the fix appears trivial, then the test for the defect ought not to take too much time either. I have been able to get to this mindset with practice.

KptMarchewa 6y ago
>How does the dev know that they've not undone anything else? No one says that rest of the test suite is not run.
kyralis 6y ago
Because being dogmatic is exactly what causes people to start ignoring this sort of methodology. Pragmatism really does have to win out sometimes. Maybe you're in a hurry because the bug is causing active downtime. Some bugs really are "obvious" once they've failed and you're looking at the code. Maybe development of a proper test involves some test infrastructure work that's a larger undertaking than you have the opportunity for at the moment. Maybe you have a solid manual/QA testing system behind you, allowing you at least temporary assurance that your fix is valid. No team does everything perfectly all the time, and that's fine. The real question is what gets done about it afterward: is the technical debt that you've incurred paid down in a reasonable time frame?
- karmelapple 6y ago
  This nicely encapsulates what I was hoping to say, but didn't take the time to write out. Thank you! > Maybe development of a proper test involves some test infrastructure work that's a larger undertaking than you have the opportunity for at the moment. This is something I've encountered many times. > Maybe you have a solid manual/QA testing system behind you, allowing you at least temporary assurance that your fix is valid. I would hope most people are doing this anyway, especially when a big production bug has been found.
magicalhippo 6y ago
I've fixed plenty of bugs which I could not reproduce and thus could not test, simply based on the source code and description of problem from customer. Based on the symptoms it was clear what the code must be doing, studying the code reveals it is clearly wrong, and so the fix was obvious. So make a patch, make a new build, send to customer and get a call "yeah that fixed it, thanks!". Sometimes the issue is I simply don't have time to reproduce it, like that time a blocking bug had to be fixed within 30 minutes, or the customer would have had to charter their own helicopter to get a few packages to an offshore oil platform, rather than piggy-back on the worker transport heli. Other times it's some combination of the customers system it's running on and configuration of our software that I can't reproduce. Not saying it's ideal, but it's quite possible to successfully fix issues without being able to reproduce and test.
- hnick 6y ago
  I'd imagine stuff like race conditions would fit into this category nicely. Obvious upon inspection but annoying to test for.
- bluGill 6y ago
  I once fixed a bug like that, then went wait a minute. Sure enough source control revealed I'd made exactly the same fix 2 years ago, and 2 years before that someone else had done exactly the same. In the odd years someone else undid that fix to fix a different bug that seemed unrelated. Once I figured out what the other problem was I was able to find the more complex fix for both situations. I wished I had heard of automated tests then
  - smileypete 6y ago
    Maybe add a dated note as a code comment breifly describing the reason for the change.
  - magicalhippo 6y ago
    Yeah I always check source control (blame/annotate), even if I wrote the code myself, just to be sure I'm not missing some context. Automated tests is pretty great, but a lot of the stuff we do is difficult to test, mostly due to a lot of legacy code that's not well confined. As we work on a piece of code we try to clean that part up, but it takes time.
pydry 6y ago
>Why is that Ok? How does the dev know that they've not undone anything else? If the bug is "I thought we were supposed to have a 10 minute timeout and we accidentally set the timeout to 10 seconds" it's pretty screamingly obvious that if you change the time from "10" to "600" the problem is now solved and you haven't broken anything else. Religiously applying this rule of thumb as you describe causes ALL sorts of problems including the problem of people writing tests for the above kind of behavior. That test will fail when it's changed from 600 to 1800 deliberately and that will create a pointless waste of time for everybody.
- Joker_vD 6y ago
  Yeah, and then you find out that that constant was also used in another piece of code as number of minutes, so you've just changed another timeout in the system from 10 minutes to 10 hours. Yes, of course that constant should never have been a naked integer in the first place, but we live in an imperfect world. One thing I like about Go's standard library is that almost everything takes time.Duration instead of a plain integer that's then interpreted to be milliseconds (or micro/nanoseconds in the most unexpected place, gotcha, developer, you should've read the docs!)
leonroy 6y ago
> This anxiety of pushing out at fix - at the risk of undoing other working functionality - ought to be addressed first. It really depends on how urgent the fix is. Naturally you want the fix to be as isolated as possible so it does not regress other functionality. But even non-urgent fixes can sometimes benefit from getting pushed out following quick manual testing. For example we use Sentry.io for capturing errors in our applications and LogDNA for logging. Occassionally we'll encounter some kind of edge case where we see a spike in logged errors which blows us past our Sentry and LogDNA quotas. Pushing a fix out before the test is written can be beneficial in cases like this although yes, it's worth avoiding if possible.
karmelapple 6y ago
Sounds like I could have been more thorough in describing why I think "that's okay." > How does the dev know that they've not undone anything else? I didn't write it, but I was assuming a scenario where a system already has a comprehensive automated test suite. If you have most functionality under test, then hopefully you're pretty confident that it won't undo anything else. > Also, how does the dev know that the fix is complete? The same way a dev knows if one single automated test that addresses the one known failure scenario is a complete fix. In other words: you don't know. You keep doing some manual testing, and watching whatever logs or status indicators, to see if things go back to normal after deployment. > Or that it caters to the defect? I also didn't write this, but I had in mind some level of manual testing before deploying the code change to ensure it caters to the defect. > Why is that Ok? Hopefully my answers above help explain why I said that's okay. I'm not advocating for flippantly shipping code without having a variety of other guard rails in place; I was primarily talking about when a bug has really bad consequences for users, your team is confident that writing a test is going to take a long time, and you do some level of manual testing to confirm all seems generally well.

KptMarchewa 6y ago

>How does the dev know that they've not undone anything else? No one says that rest of the test suite is not run.

kyralis 6y ago

Because being dogmatic is exactly what causes people to start ignoring this sort of methodology. Pragmatism really does have to win out sometimes. Maybe you're in a hurry because the bug is causing active downtime. Some bugs really are "obvious" once they've failed and you're looking at the code. Maybe development of a proper test involves some test infrastructure work that's a larger undertaking than you have the opportunity for at the moment. Maybe you have a solid manual/QA testing system behind you, allowing you at least temporary assurance that your fix is valid. No team does everything perfectly all the time, and that's fine. The real question is what gets done about it afterward: is the technical debt that you've incurred paid down in a reasonable time frame?

karmelapple 6y ago
This nicely encapsulates what I was hoping to say, but didn't take the time to write out. Thank you! > Maybe development of a proper test involves some test infrastructure work that's a larger undertaking than you have the opportunity for at the moment. This is something I've encountered many times. > Maybe you have a solid manual/QA testing system behind you, allowing you at least temporary assurance that your fix is valid. I would hope most people are doing this anyway, especially when a big production bug has been found.

magicalhippo 6y ago

I've fixed plenty of bugs which I could not reproduce and thus could not test, simply based on the source code and description of problem from customer. Based on the symptoms it was clear what the code must be doing, studying the code reveals it is clearly wrong, and so the fix was obvious. So make a patch, make a new build, send to customer and get a call "yeah that fixed it, thanks!". Sometimes the issue is I simply don't have time to reproduce it, like that time a blocking bug had to be fixed within 30 minutes, or the customer would have had to charter their own helicopter to get a few packages to an offshore oil platform, rather than piggy-back on the worker transport heli. Other times it's some combination of the customers system it's running on and configuration of our software that I can't reproduce. Not saying it's ideal, but it's quite possible to successfully fix issues without being able to reproduce and test.

hnick 6y ago
I'd imagine stuff like race conditions would fit into this category nicely. Obvious upon inspection but annoying to test for.
bluGill 6y ago
I once fixed a bug like that, then went wait a minute. Sure enough source control revealed I'd made exactly the same fix 2 years ago, and 2 years before that someone else had done exactly the same. In the odd years someone else undid that fix to fix a different bug that seemed unrelated. Once I figured out what the other problem was I was able to find the more complex fix for both situations. I wished I had heard of automated tests then
- smileypete 6y ago
  Maybe add a dated note as a code comment breifly describing the reason for the change.
- magicalhippo 6y ago
  Yeah I always check source control (blame/annotate), even if I wrote the code myself, just to be sure I'm not missing some context. Automated tests is pretty great, but a lot of the stuff we do is difficult to test, mostly due to a lot of legacy code that's not well confined. As we work on a piece of code we try to clean that part up, but it takes time.

pydry 6y ago

>Why is that Ok? How does the dev know that they've not undone anything else? If the bug is "I thought we were supposed to have a 10 minute timeout and we accidentally set the timeout to 10 seconds" it's pretty screamingly obvious that if you change the time from "10" to "600" the problem is now solved and you haven't broken anything else. Religiously applying this rule of thumb as you describe causes ALL sorts of problems including the problem of people writing tests for the above kind of behavior. That test will fail when it's changed from 600 to 1800 deliberately and that will create a pointless waste of time for everybody.

Joker_vD 6y ago
Yeah, and then you find out that that constant was also used in another piece of code as number of minutes, so you've just changed another timeout in the system from 10 minutes to 10 hours. Yes, of course that constant should never have been a naked integer in the first place, but we live in an imperfect world. One thing I like about Go's standard library is that almost everything takes time.Duration instead of a plain integer that's then interpreted to be milliseconds (or micro/nanoseconds in the most unexpected place, gotcha, developer, you should've read the docs!)

leonroy 6y ago

> This anxiety of pushing out at fix - at the risk of undoing other working functionality - ought to be addressed first. It really depends on how urgent the fix is. Naturally you want the fix to be as isolated as possible so it does not regress other functionality. But even non-urgent fixes can sometimes benefit from getting pushed out following quick manual testing. For example we use Sentry.io for capturing errors in our applications and LogDNA for logging. Occassionally we'll encounter some kind of edge case where we see a spike in logged errors which blows us past our Sentry and LogDNA quotas. Pushing a fix out before the test is written can be beneficial in cases like this although yes, it's worth avoiding if possible.

karmelapple 6y ago

Sounds like I could have been more thorough in describing why I think "that's okay." > How does the dev know that they've not undone anything else? I didn't write it, but I was assuming a scenario where a system already has a comprehensive automated test suite. If you have most functionality under test, then hopefully you're pretty confident that it won't undo anything else. > Also, how does the dev know that the fix is complete? The same way a dev knows if one single automated test that addresses the one known failure scenario is a complete fix. In other words: you don't know. You keep doing some manual testing, and watching whatever logs or status indicators, to see if things go back to normal after deployment. > Or that it caters to the defect? I also didn't write this, but I had in mind some level of manual testing before deploying the code change to ensure it caters to the defect. > Why is that Ok? Hopefully my answers above help explain why I said that's okay. I'm not advocating for flippantly shipping code without having a variety of other guard rails in place; I was primarily talking about when a bug has really bad consequences for users, your team is confident that writing a test is going to take a long time, and you do some level of manual testing to confirm all seems generally well.