3 ms·

Maybe i am naive, but i think this screams for a system which enforces reproducing of bugs. 1. Bug is filed 2. System checks if bug is reproducible 3. As lon

Maybe i am naive, but i think this screams for a system which enforces reproducing of bugs. 1. Bug is filed 2. System checks if bug is reproducible 3. As long as the bug is not reproducible it is hidden in the issue tracker and flagged as "needs reproducibility" and emails are sent to the creator. After a given time the bug is closed automatically. 4. When the testcase to reproduce the bug is provided, the build system checks against the current supported versions. If it passes, the bug is fixed as "needs reproducibility" again, if it fails, the bug is finally pushed to the main issue tracker. I can see how brutal this approach is, but this way, the number of bugs is kept to a minimum. Bug filers, who actually take the time will therefore have a high probability of immediate fixing. Motivated people can always play gardener additionally, but without the email spam, they can simply help in one step of the process.

pavel_lishin 13y ago
You'd need a custom DSL to tell the system how to reproduce the bug; at this point, I'm already bored, and I don't even have a bug to submit.
- steveklabnik 13y ago
  In theory it could be a pull request with a failing test.
  - pavel_lishin 13y ago
    That's still quite a bit of work - there are some projects I'd do that for, but if (for instance) an IRC bot was having an issue, I doubt I'd bother if submitting a bug required writing a new test, and submitting a PR.
    - steveklabnik 13y ago
      Oh yeah, I'm still not in favor of it as an idea, just tossing thoughts around.
steveklabnik 13y ago
> I can see how brutal this approach is, but this way, the number of bugs is kept to a minimum. Right, this is the problem I see here. Some people will just drop off entirely in this case, which may or may not be what your project wants. And some people give great reproducible steps, even if they aren't already automated: "Run 'rails new foo', then add this controller <code>, then hit the page and you'll see 1 instead of 2." The system would reject that, even though it has instructions to reproduce.
- allendoerfer 13y ago
  Bookkeeping software tries to catch all payments and decides which invoice was paid. If the identification misses, they can try to match by other indicators (Name etc.). Some payments slip through the system and a person looks after it. What i am saying is: Half-automated is better than manual.
daleharvey 13y ago
This is pretty contradictory to 1. my motivations as a maintainer, 2. how to solve (imo) the barriers in having people contribute to OSS 1. As a maintainer I want my project to work, if a reporter only wants to give a brief description then I am both happy and have the context in order to read it, attempt to understand it and ask the relevant follow up questions. Ideally someone turns up with a test case and patch and I will make that as easy as I can, but its not a requirement for being able to contribute. 2. I did a questionnaire on what prevents people contributing to open source, by far the biggest barriers I found were things that were solved by asking a quick question to a maintainer, people get stuck on what to fix, or how to set stuff up and they want to ask a person and not a mailing list or a chat room, and certainly not a robot policing their filed bugs
- reedlaw 13y ago
  That sounds like a good process for smaller projects. For larger projects, or projects where developer attention is a precious resource, not so much. For example, look at the detail provided by the issue submitter to the nouveau (open source Nvidia Linux driver) project: https://bugs.freedesktop.org/show_bug.cgi?id=76732 https://bugs.freedesktop.org/show_bug.cgi?id=76732 Not only is there a lot of log output, the submitter took the trouble to run git bisect on the Linux kernel to see at which point the breakage occurred. I'm sure the submitter spent quite a bit of time and has a high level of familiarity with the project and related tools. This will help developers to isolate the problem and provide a quick fix. Without this level of detail, the issue might never get fixed.
  - daleharvey 13y ago
    Thats the point, ultra motivated contributors will do this, semi motivated contributors can be coached into it, people that just want their stuff to work may not, but all of it is useful, putting a barrier in front of that is wasting developers time. And I am not speaking solely about small projects, I experience exactly the same thing while working on Firefox OS, at nearing a million 'issues' its definitely not small.
rcxdude 13y ago
This only works for bugs which are easy to reproduce automatically and on all hardware. Your system would completely block a lot of bugs in linux (e.g. https://bugzilla.kernel.org/show_bug.cgi?id=68761 https://bugzilla.kernel.org/show_bug.cgi?id=68761). For some projects this might not be so ridiculous but you're still raising the barrier to bug reports and so you're going to miss bugs. I don't think a manual process is all that difficult unless you have an especially large and/or vocal userbase.
kylemaxwell 13y ago
You are essentially advocating that reporters write a test script with each bug filed. During the process, of course, this makes sense, but it may not work right away because coming up with a solid repro can be a challenge. And not all users of all projects will be coders able to write test scripts. OTOH that is a fine job for a somebody taking on the role of triage / "gardener".
- allendoerfer 13y ago
  I think rails users, who are able to find a bug in rails are also able to write a test for it. A GUI system, which allows you to make the system do the stuff which causes the bug and records the steps would be great, but i fear most software is to complex and special to come up with a useful system.
  - kylemaxwell 13y ago
    For Rails, sure - but most FLOSS projects are not like that.
  - jscn 13y ago
    Tests generated by GUI systems are extremely fragile and you wouldn't them hanging around your codebase. OTOH, often a video recording is a great way to demonstrate a bug without having to (be able to) write an automated test, and pretty easy for even a non-technical user to produce.
enneff 13y ago
I don't think this would have helped us a lot on Go. We occasionally get reports like "We're seeing [issue] on our system, but only when it's loaded with production traffic." Some bugs are simply impossible to reproduce in an isolated, mechanical way. (Or at least very hard for people who aren't totally familiar with the project.) But often one of the core will take a look at those reports and say "Aha! That must be a consequence of [some change we made]." Then they investigate, poke at some of their code, check their assumptions, and eventually find the cause. Another large class of bugs pertains to docs, for which the concept of "reproducibility" is irrelevant.
lambda 13y ago
There are tons of bugs which are easy to describe in prose, and for which such an explanation can easily allow someone familiar with the codebase to be able to reproduce or even fix it right off the bat, but which would be quite difficult or even impossible to create an automated test for that could reproduce it in a way that the bug tracker could check. It might be nice for bugtrackers to offer an optional system that allows you to submit an automated test case, which could be run in some kind of sandbox, and bugs that were easily reproducible would be marked as such and could get more attention. But anything that simply discarded all other bugs would be throwing the baby out with the bathwater; you would simply not hear from your users as much, on lots of important issues.

pavel_lishin 13y ago

You'd need a custom DSL to tell the system how to reproduce the bug; at this point, I'm already bored, and I don't even have a bug to submit.

steveklabnik 13y ago
In theory it could be a pull request with a failing test.
- pavel_lishin 13y ago
  That's still quite a bit of work - there are some projects I'd do that for, but if (for instance) an IRC bot was having an issue, I doubt I'd bother if submitting a bug required writing a new test, and submitting a PR.
  - steveklabnik 13y ago
    Oh yeah, I'm still not in favor of it as an idea, just tossing thoughts around.

steveklabnik 13y ago

> I can see how brutal this approach is, but this way, the number of bugs is kept to a minimum. Right, this is the problem I see here. Some people will just drop off entirely in this case, which may or may not be what your project wants. And some people give great reproducible steps, even if they aren't already automated: "Run 'rails new foo', then add this controller <code>, then hit the page and you'll see 1 instead of 2." The system would reject that, even though it has instructions to reproduce.

allendoerfer 13y ago
Bookkeeping software tries to catch all payments and decides which invoice was paid. If the identification misses, they can try to match by other indicators (Name etc.). Some payments slip through the system and a person looks after it. What i am saying is: Half-automated is better than manual.

daleharvey 13y ago

This is pretty contradictory to 1. my motivations as a maintainer, 2. how to solve (imo) the barriers in having people contribute to OSS 1. As a maintainer I want my project to work, if a reporter only wants to give a brief description then I am both happy and have the context in order to read it, attempt to understand it and ask the relevant follow up questions. Ideally someone turns up with a test case and patch and I will make that as easy as I can, but its not a requirement for being able to contribute. 2. I did a questionnaire on what prevents people contributing to open source, by far the biggest barriers I found were things that were solved by asking a quick question to a maintainer, people get stuck on what to fix, or how to set stuff up and they want to ask a person and not a mailing list or a chat room, and certainly not a robot policing their filed bugs

reedlaw 13y ago
That sounds like a good process for smaller projects. For larger projects, or projects where developer attention is a precious resource, not so much. For example, look at the detail provided by the issue submitter to the nouveau (open source Nvidia Linux driver) project: https://bugs.freedesktop.org/show_bug.cgi?id=76732 https://bugs.freedesktop.org/show_bug.cgi?id=76732 Not only is there a lot of log output, the submitter took the trouble to run git bisect on the Linux kernel to see at which point the breakage occurred. I'm sure the submitter spent quite a bit of time and has a high level of familiarity with the project and related tools. This will help developers to isolate the problem and provide a quick fix. Without this level of detail, the issue might never get fixed.
- daleharvey 13y ago
  Thats the point, ultra motivated contributors will do this, semi motivated contributors can be coached into it, people that just want their stuff to work may not, but all of it is useful, putting a barrier in front of that is wasting developers time. And I am not speaking solely about small projects, I experience exactly the same thing while working on Firefox OS, at nearing a million 'issues' its definitely not small.

rcxdude 13y ago

This only works for bugs which are easy to reproduce automatically and on all hardware. Your system would completely block a lot of bugs in linux (e.g. https://bugzilla.kernel.org/show_bug.cgi?id=68761 https://bugzilla.kernel.org/show_bug.cgi?id=68761). For some projects this might not be so ridiculous but you're still raising the barrier to bug reports and so you're going to miss bugs. I don't think a manual process is all that difficult unless you have an especially large and/or vocal userbase.

kylemaxwell 13y ago

You are essentially advocating that reporters write a test script with each bug filed. During the process, of course, this makes sense, but it may not work right away because coming up with a solid repro can be a challenge. And not all users of all projects will be coders able to write test scripts. OTOH that is a fine job for a somebody taking on the role of triage / "gardener".

allendoerfer 13y ago
I think rails users, who are able to find a bug in rails are also able to write a test for it. A GUI system, which allows you to make the system do the stuff which causes the bug and records the steps would be great, but i fear most software is to complex and special to come up with a useful system.
- kylemaxwell 13y ago
  For Rails, sure - but most FLOSS projects are not like that.
- jscn 13y ago
  Tests generated by GUI systems are extremely fragile and you wouldn't them hanging around your codebase. OTOH, often a video recording is a great way to demonstrate a bug without having to (be able to) write an automated test, and pretty easy for even a non-technical user to produce.

enneff 13y ago

I don't think this would have helped us a lot on Go. We occasionally get reports like "We're seeing [issue] on our system, but only when it's loaded with production traffic." Some bugs are simply impossible to reproduce in an isolated, mechanical way. (Or at least very hard for people who aren't totally familiar with the project.) But often one of the core will take a look at those reports and say "Aha! That must be a consequence of [some change we made]." Then they investigate, poke at some of their code, check their assumptions, and eventually find the cause. Another large class of bugs pertains to docs, for which the concept of "reproducibility" is irrelevant.

lambda 13y ago

There are tons of bugs which are easy to describe in prose, and for which such an explanation can easily allow someone familiar with the codebase to be able to reproduce or even fix it right off the bat, but which would be quite difficult or even impossible to create an automated test for that could reproduce it in a way that the bug tracker could check. It might be nice for bugtrackers to offer an optional system that allows you to submit an automated test case, which could be run in some kind of sandbox, and bugs that were easily reproducible would be marked as such and could get more attention. But anything that simply discarded all other bugs would be throwing the baby out with the bathwater; you would simply not hear from your users as much, on lots of important issues.