3 ms·

That certainly makes sense - and that tight loop is very compelling. What if you have to solve a bug outside of the development loop though? E.g. a bug somewh

That certainly makes sense - and that tight loop is very compelling. What if you have to solve a bug outside of the development loop though? E.g. a bug somewhere in the whole system after you've done your development, where you don't have a root cause yet?

gnulinux 3y ago
I write integration tests as well as unittests, but mostly integration tests, to a reasonable extent. I also don't commit all integration tests if it's impractical e.g. makes the test suite too slow, instead have equivalent unittests. I use integration test to understand the underlying bug, then write a unittest. But, truthfully, to understand where to begin with, I religiously use logs. Normally prod logs are disabled or silenced, so the first step of debugging is enabling logs, reproduce the error in prod, get logs. Then, I inspect the logs and develop a hypothesis where the bug is. I write an integration test reproducing the exact same bug, red, I fix the bug, green. Then, I decide whether I want to commit this test, is it useful, is it fast enough. If so, we're done. Otherwise, I undo my commit, I write a unittest, red, apply the previous fix as-is, green. As you see, this is a mix of printf() debugging (logs are essentially printf statements) and TDD. One problem with this approach is, of course, if your logs aren't sufficient you may not now where to start. Then, you need to either ssh into a dev env and play around, or recreate the infra on you laptop and use a debugger to step through. I make sure my logs and tests are excellent in order to prevent these scenarios though, I also make sure the system is idempotent so that it's easy to reproduce bugs (because leadership tends not to like verbose logging enabled in prod due to storage costs, this means before reproducing a bug you need enable logs).

3 ms·

That certainly makes sense - and that tight loop is very compelling. What if you have to solve a bug outside of the development loop though? E.g. a bug somewh

by mark_undoio 3y ago

gnulinux 3y ago
I write integration tests as well as unittests, but mostly integration tests, to a reasonable extent. I also don't commit all integration tests if it's impractical e.g. makes the test suite too slow, instead have equivalent unittests. I use integration test to understand the underlying bug, then write a unittest. But, truthfully, to understand where to begin with, I religiously use logs. Normally prod logs are disabled or silenced, so the first step of debugging is enabling logs, reproduce the error in prod, get logs. Then, I inspect the logs and develop a hypothesis where the bug is. I write an integration test reproducing the exact same bug, red, I fix the bug, green. Then, I decide whether I want to commit this test, is it useful, is it fast enough. If so, we're done. Otherwise, I undo my commit, I write a unittest, red, apply the previous fix as-is, green. As you see, this is a mix of printf() debugging (logs are essentially printf statements) and TDD. One problem with this approach is, of course, if your logs aren't sufficient you may not now where to start. Then, you need to either ssh into a dev env and play around, or recreate the infra on you laptop and use a debugger to step through. I make sure my logs and tests are excellent in order to prevent these scenarios though, I also make sure the system is idempotent so that it's easy to reproduce bugs (because leadership tends not to like verbose logging enabled in prod due to storage costs, this means before reproducing a bug you need enable logs).