7 ms·

The best bug tracking system. Don’t raise bugs, write an automated test

37 points by fogus 17y ago

moe 17y ago
Yea, because all testers are capable of writing software tests and because all bugs warrant the effort to write a test... Not sure what this is doing on HN, really.
- andrewcherry 17y ago
  Well, while I certainly agree that this is perhaps a way off in terms of technical capability, that won't always be the case perhaps. I'm not talking about everyone learning to code, that's dead in the water. But systems which allow business people to define their own acceptance and test criteria exist now - tools like Fitnesse, etc. I'm not saying this is feasible today, but a future can be envisaged where writing a repeatable test which determines whether a function works as desired is within the domain of less technical users.
  - Uchikoma 17y ago
    "But systems which allow business people to define their own acceptance and test criteria exist now - tools like Fitness." Out of the mouth of a business person, wonderful. Out of the mouth of a developer, worthless.
  - Retric 17y ago
    If you let people without a development background write tests it's easy for them to say something to the effect of (X > 7) and (X < 4). More generally insuring that there are no conflicts in your requirements is a very hard problem.
jasonkester 17y ago
Here are 4 real-world bug reports, in their entirety. Anybody care to rewrite them as automated test cases for me? * "Try it Now" button sometimes drops below the "Learn More" button in FireFox * Capitalization inconsistent in top nav * Popup blockers occasionally stop the App window from opening properly * Wierd timeout errors clumped around 1:00 am every few nights * Accounting needs to be able to call up weekly income reports In my experience, it's actually pretty rare to find a bug that can be wrapped in a test case without actually discovering and fixing it in the process. Most things that end up in the bug tracker are either cosmetic, wierd "solar activity" things, or disguised feature requests.
- diN0bot 17y ago
  different programming domains have different problems... wrt this list: * if GUIs were easier to write tests for, then you wouldn't have problems writing tests for the first 3 on your list. * timing issues and edge cases are one of the reasons i value tests, assertions and strong design so much: debugging is hard. * the last one sounds like a feature that hasn't been written, not a bug.
  - joe_the_user 17y ago
    timing issues and edge cases are one of the reasons i value tests, assertions and strong design so much: debugging is hard. It always seems like when you scratch the TDD surface, "writing tests first lets you write bug-free software" becomes "tests are an important tool along with good design, assertions, etc". I agree with the later but I don't see how it's different earlier methodologies - and earlier methodologies certainly didn't let you write bug-free code.
- lallysingh 17y ago
  Check out Selenium.
- vlisivka 17y ago
  > * "Try it Now" button sometimes drops below the "Learn More" button in FireFox Use JavaScript or Selenium for that Test case. Get coordinates of buttons (getBoundingClientRect()) and compare them. Execute this test case on tons of different configurations. > * Capitalization inconsistent in top nav Easy to test. Also add spell checking along the way. > Popup blockers occasionally stop the App window from opening properly Selenium? > Wierd timeout errors clumped around 1:00 am every few nights You always can shift clock automatically (on Linux). > Accounting needs to be able to call up weekly income reports If it is missing feature, then no test case is need until you will start work on it. PS. If you want to automate your manual testing, you will look for ways to do that. If not, you will look for problems, which are prevents you from doing that. I wrote example how to execute HTMLUnit tests at build stage because I want that. Almost everybody else says that this is not possible. ( http://vlisivka.pp.ua/en/integration_testing_with_maven_testng_htmlunit_embeded_tomcat6 http://vlisivka.pp.ua/en/integration_testing_with_maven_test... )
colomon 17y ago
And of course, the testing for any bug can be easily automated. Not! Look, some things are very easy to test in unit tests: Does sin(180) return the right value? If I do operation X on supposedly const object O, does O have the same value before and after? And some things are incredibly hard: Does this MP3 sound right after compression? Does the typography for this combination of letters look right? Is the user interface responsive enough? What about the crash that only occurs in just the right hard-to-duplicate circumstances? Why is it that these hard core test advocates always seem to assume that all bugs are of the former sort?
- Silhouette 17y ago
  > Why is it that these hard core test advocates always seem to assume that all bugs are of the former sort? It's probably for the same reason that evangelists for certain programming styles will dogmatically insist that you shouldn't have more than X lines of code or Y levels of nesting in your functions, without reference to things like the nature of the programming language or the essential complexity of the algorithm to be implemented. Inexperienced people tend to overgeneralise from their own experience, and those who have only ever worked in one small part of the programming world are the most disadvantaged of all.
- generalk 17y ago
  Why is it that these hard core test advocates always seem to assume that all bugs are of the former sort? As someone who's more and more becoming a "hard core test advocate": not all bugs are the former sort! But why would that stop you from testing the ones that are?
  - colomon 17y ago
    For the record, I've written 18,000+ test cases in the last year. When you can write reasonable tests, they are an awesome tool for programming. I just object strongly to the notion that any possible bug can be reduced to an automated test. That's not my experience at all.
- daleharvey 17y ago
aw3c2 17y ago
Reporting bugs is already tedious. Sometimes more (eg when requiring an account), sometimes less. Asking for a test case would surely reduce the number of "bug reports" to abysmal figures.
- deleted 17y ago
  [deleted]
alextgordon 17y ago
This wouldn't work for: * Public bug trackers * GUI applications, where writing automated tests is difficult or infeasible. (how would you write a test case for "icon is upside down"?) * Abstract bugs, such as performance-related bugs or feature requests * Nondeterministic or otherwise unusual bugs I'd love to live in a world where all bugs were easily reproducible and even a non-programmer could write a test case. A quick glance over my issue tracker reveals this is not the case.
Tichy 17y ago
Raising the cost for reporting bugs might people want to avoid reporting bugs. I found I hardly every used a bug tracker I had installed on a slow server, because it was so slow. So I don't think making bug submission even slower would be a good thing.
- nathanh 17y ago
  It also raises the literal dollar cost for QA people. If you have a large QA operation, you're probably paying semi-skilled non-programmers. If you have to get QA people who can code enough to write a test, they'd cost a lot more.
deleted 17y ago
[deleted]
- a-priori 17y ago
  ...and how do you ensure that the unit tests are correct? Another layer of automated tests? Accept you will make mistakes, but code in a way that makes finding and fixing them easy. The first way is to make the test logic self-evident so its easy to audit. Another approach is to use something like Ruby's Heckle tool which mutates code. If the code can be modified and the tests still pass, then the tests are incomplete or incorrect. Perhaps other languages/platforms have something similar. ...I'd like to see the unit test that checks things look ok Internet Explorer While testing aesthetics is pretty much impossible, Selenium can test the functionality in multiple browsers. or that the system generally does what the client pays for... If you haven't reduced the client's requests to a useable specification you can write tests from, then that's your fault. Computers can't do your job for you.
lifeisstillgood 17y ago
Bug Trackers also play an important prioritisation role - a central location to decide what work is most important right now. If we wrote only unit tests, pretty soon someone would hack up a system to take the names of the tests, and put them in a list. It would be called testzilla.
grumpyfart 17y ago
If you are developer and doing TDD or some sort of automated testing (which you should!) and if you are not writing automated tests for bugs, you are doing it wrong anyway! If the bug is GUI inconsistency, looks crap on my mom's computer sort of bug, then you can't automate anyway. Assuming you testers' mom haven't got serial ports.
antirez 17y ago
Most synchronization issues in multi threaded applications can't be automated. Also there are bugs triggered by a specific state of the application that is complex to encode in a test.
whyme 17y ago
Communication problems: Sometimes the description of the bug is poor, or the developer misunderstands the problem. -> Maybe more effort should be placed on improving communication skills for your company? This will greatly improve many other aspects of your business - not just bug logging. Regression testing. This process requires the handover of code back and forward between QA and development, this causes versioning problems and duplicated testing effort. -> why? Maybe more effort should be placed on improving communication skills... Low robustness. The process doesn’t guarantee that the error will appear again in the future. -> Neither would buggy automated tests - or whom do you need to hire to test the tester? Bureaucracy. Traditional bug tracking systems switches the team focus to bureaucracy from quality -> If you care about your product quality and your other team members development goals it's not bureaucracy it is quality. Can't see past mistake occurrences, thus don't see the system as meaningful. Rather the focus should be to take the current system seriously and work on improving it.
mars 17y ago
my company is developing web applications that are used by companies to optimize internal processes. i recently found out about selenium and this tool is just great. i wasn't aware that it is possible to write tests for even complex and heavily ajax driven web interfaces. roxxx. don't get me wrong, it is indeed a lot of work to write those tests. but once written they'll save your time and more important, your life. give it a try.

37 points by fogus 17y ago

moe 17y ago

Yea, because all testers are capable of writing software tests and because all bugs warrant the effort to write a test... Not sure what this is doing on HN, really.

andrewcherry 17y ago
Well, while I certainly agree that this is perhaps a way off in terms of technical capability, that won't always be the case perhaps. I'm not talking about everyone learning to code, that's dead in the water. But systems which allow business people to define their own acceptance and test criteria exist now - tools like Fitnesse, etc. I'm not saying this is feasible today, but a future can be envisaged where writing a repeatable test which determines whether a function works as desired is within the domain of less technical users.
- Uchikoma 17y ago
  "But systems which allow business people to define their own acceptance and test criteria exist now - tools like Fitness." Out of the mouth of a business person, wonderful. Out of the mouth of a developer, worthless.
- Retric 17y ago
  If you let people without a development background write tests it's easy for them to say something to the effect of (X > 7) and (X < 4). More generally insuring that there are no conflicts in your requirements is a very hard problem.

jasonkester 17y ago

Here are 4 real-world bug reports, in their entirety. Anybody care to rewrite them as automated test cases for me? * "Try it Now" button sometimes drops below the "Learn More" button in FireFox * Capitalization inconsistent in top nav * Popup blockers occasionally stop the App window from opening properly * Wierd timeout errors clumped around 1:00 am every few nights * Accounting needs to be able to call up weekly income reports In my experience, it's actually pretty rare to find a bug that can be wrapped in a test case without actually discovering and fixing it in the process. Most things that end up in the bug tracker are either cosmetic, wierd "solar activity" things, or disguised feature requests.

diN0bot 17y ago
different programming domains have different problems... wrt this list: * if GUIs were easier to write tests for, then you wouldn't have problems writing tests for the first 3 on your list. * timing issues and edge cases are one of the reasons i value tests, assertions and strong design so much: debugging is hard. * the last one sounds like a feature that hasn't been written, not a bug.
- joe_the_user 17y ago
  timing issues and edge cases are one of the reasons i value tests, assertions and strong design so much: debugging is hard. It always seems like when you scratch the TDD surface, "writing tests first lets you write bug-free software" becomes "tests are an important tool along with good design, assertions, etc". I agree with the later but I don't see how it's different earlier methodologies - and earlier methodologies certainly didn't let you write bug-free code.
lallysingh 17y ago
Check out Selenium.
vlisivka 17y ago
> * "Try it Now" button sometimes drops below the "Learn More" button in FireFox Use JavaScript or Selenium for that Test case. Get coordinates of buttons (getBoundingClientRect()) and compare them. Execute this test case on tons of different configurations. > * Capitalization inconsistent in top nav Easy to test. Also add spell checking along the way. > Popup blockers occasionally stop the App window from opening properly Selenium? > Wierd timeout errors clumped around 1:00 am every few nights You always can shift clock automatically (on Linux). > Accounting needs to be able to call up weekly income reports If it is missing feature, then no test case is need until you will start work on it. PS. If you want to automate your manual testing, you will look for ways to do that. If not, you will look for problems, which are prevents you from doing that. I wrote example how to execute HTMLUnit tests at build stage because I want that. Almost everybody else says that this is not possible. ( http://vlisivka.pp.ua/en/integration_testing_with_maven_testng_htmlunit_embeded_tomcat6 http://vlisivka.pp.ua/en/integration_testing_with_maven_test... )

colomon 17y ago

And of course, the testing for any bug can be easily automated. Not! Look, some things are very easy to test in unit tests: Does sin(180) return the right value? If I do operation X on supposedly const object O, does O have the same value before and after? And some things are incredibly hard: Does this MP3 sound right after compression? Does the typography for this combination of letters look right? Is the user interface responsive enough? What about the crash that only occurs in just the right hard-to-duplicate circumstances? Why is it that these hard core test advocates always seem to assume that all bugs are of the former sort?

Silhouette 17y ago
> Why is it that these hard core test advocates always seem to assume that all bugs are of the former sort? It's probably for the same reason that evangelists for certain programming styles will dogmatically insist that you shouldn't have more than X lines of code or Y levels of nesting in your functions, without reference to things like the nature of the programming language or the essential complexity of the algorithm to be implemented. Inexperienced people tend to overgeneralise from their own experience, and those who have only ever worked in one small part of the programming world are the most disadvantaged of all.
generalk 17y ago
Why is it that these hard core test advocates always seem to assume that all bugs are of the former sort? As someone who's more and more becoming a "hard core test advocate": not all bugs are the former sort! But why would that stop you from testing the ones that are?
- colomon 17y ago
  For the record, I've written 18,000+ test cases in the last year. When you can write reasonable tests, they are an awesome tool for programming. I just object strongly to the notion that any possible bug can be reduced to an automated test. That's not my experience at all.
daleharvey 17y ago

aw3c2 17y ago

Reporting bugs is already tedious. Sometimes more (eg when requiring an account), sometimes less. Asking for a test case would surely reduce the number of "bug reports" to abysmal figures.

deleted 17y ago
[deleted]

alextgordon 17y ago

This wouldn't work for: * Public bug trackers * GUI applications, where writing automated tests is difficult or infeasible. (how would you write a test case for "icon is upside down"?) * Abstract bugs, such as performance-related bugs or feature requests * Nondeterministic or otherwise unusual bugs I'd love to live in a world where all bugs were easily reproducible and even a non-programmer could write a test case. A quick glance over my issue tracker reveals this is not the case.

Tichy 17y ago

Raising the cost for reporting bugs might people want to avoid reporting bugs. I found I hardly every used a bug tracker I had installed on a slow server, because it was so slow. So I don't think making bug submission even slower would be a good thing.

nathanh 17y ago
It also raises the literal dollar cost for QA people. If you have a large QA operation, you're probably paying semi-skilled non-programmers. If you have to get QA people who can code enough to write a test, they'd cost a lot more.

deleted 17y ago

[deleted]

a-priori 17y ago
...and how do you ensure that the unit tests are correct? Another layer of automated tests? Accept you will make mistakes, but code in a way that makes finding and fixing them easy. The first way is to make the test logic self-evident so its easy to audit. Another approach is to use something like Ruby's Heckle tool which mutates code. If the code can be modified and the tests still pass, then the tests are incomplete or incorrect. Perhaps other languages/platforms have something similar. ...I'd like to see the unit test that checks things look ok Internet Explorer While testing aesthetics is pretty much impossible, Selenium can test the functionality in multiple browsers. or that the system generally does what the client pays for... If you haven't reduced the client's requests to a useable specification you can write tests from, then that's your fault. Computers can't do your job for you.

lifeisstillgood 17y ago

Bug Trackers also play an important prioritisation role - a central location to decide what work is most important right now. If we wrote only unit tests, pretty soon someone would hack up a system to take the names of the tests, and put them in a list. It would be called testzilla.

grumpyfart 17y ago

If you are developer and doing TDD or some sort of automated testing (which you should!) and if you are not writing automated tests for bugs, you are doing it wrong anyway! If the bug is GUI inconsistency, looks crap on my mom's computer sort of bug, then you can't automate anyway. Assuming you testers' mom haven't got serial ports.

antirez 17y ago

Most synchronization issues in multi threaded applications can't be automated. Also there are bugs triggered by a specific state of the application that is complex to encode in a test.

whyme 17y ago

Communication problems: Sometimes the description of the bug is poor, or the developer misunderstands the problem. -> Maybe more effort should be placed on improving communication skills for your company? This will greatly improve many other aspects of your business - not just bug logging. Regression testing. This process requires the handover of code back and forward between QA and development, this causes versioning problems and duplicated testing effort. -> why? Maybe more effort should be placed on improving communication skills... Low robustness. The process doesn’t guarantee that the error will appear again in the future. -> Neither would buggy automated tests - or whom do you need to hire to test the tester? Bureaucracy. Traditional bug tracking systems switches the team focus to bureaucracy from quality -> If you care about your product quality and your other team members development goals it's not bureaucracy it is quality. Can't see past mistake occurrences, thus don't see the system as meaningful. Rather the focus should be to take the current system seriously and work on improving it.

mars 17y ago

my company is developing web applications that are used by companies to optimize internal processes. i recently found out about selenium and this tool is just great. i wasn't aware that it is possible to write tests for even complex and heavily ajax driven web interfaces. roxxx. don't get me wrong, it is indeed a lot of work to write those tests. but once written they'll save your time and more important, your life. give it a try.