4 ms·

A few related observations: Software development is not a stable process. Either a team is always building new things in which case there isn't a consistent pr

• jacques_chester 3y ago
  > Software development is not a stable process. This causes confusion in my experience. "Stable" doesn't mean what it does in casual usage. It means that you have a mean and variance that are not varying over time (and there are ways to alert if they are). A stable process can lurch around like a drunken sailor and still be considered stable for SPC purposes. I think bugs should be stable if you normalize it by a size metric. So bugs/SLOC or bugs/story points.
  • a1369209993 3y ago
    > "Stable" [...] means that you have a mean and variance And software development, as a process, does not have a (finite) mean or variance. > I think bugs should be stable if you normalize it by a size metric. Bugs approximately follow a heavy-tailed distribution, such that as the number of samples increases, the empirical average work to fix (and possibly other measures of severity) does not converge to any finite value, but instead increases without bound. (I think roughly logarithmicly, but don't quote me on that.) In particular, it doesn't satisfy the requirements of the central limit theorem, so a lot of statistical techniques work poorly or not at all on software projects that are large enough, individually, to do statistics on (as opposed to doing statistics on populations of software projects, which seems to mostly work, usually).
    • kqr 3y ago
      > Bugs approximately follow a heavy-tailed distribution What's your data on this? In my experience, bug fixing time has been surprisingly thin-tailed. (High variation, yes, but not subexponential.) My experience here is also mirrored by Avery Pennarun, who observes that "every bug is the same size": https://apenwarr.ca/log/20171213 https://apenwarr.ca/log/20171213
    • mharig 3y ago
      > And software development, as a process, does not have a (finite) mean or variance. Ever heared of something called linear regression? E.g. LOCs/week might be an interesting enough metric for a growing company. Edit: better example
  • roenxi 3y ago
    They aren't stable. Bugs, viewed as a timeseries, don't have a steady mean and standard deviation. There is a feedback loop where those measures change depending on the maintenance and development strategy of the software, and priorities are going to shift suddenly. If there is a fix-small-bugs month for example, or a fix-this-client's-bugs push, or a fix-the-bugs-in-the-new-feature campaign. The statistical properties of all these are not identical. And once you've normalised for story points, what you're measuring is stability of the story point allocations. Which is to say, different managers will get different means and standard deviations even if the devs change nothing about their work habits. It isn't measuring the devs, software quality or even real number of bugs; it is measuring the ticket creation process. As long as you know it isn't measuring the software, all is fine. But people do get confused.
    • jacques_chester 3y ago
      > If there is a fix-small-bugs month for example, or a fix-this-client's-bugs push, or a fix-the-bugs-in-the-new-feature campaign. The statistical properties of all these are not identical. For sure -- but these are surely special or assignable causes of variation. That's one of the key insights in SPC: that some variations are ordinary and some out of the ordinary. A "fix the bugs" campaign is out of the ordinary. > Which is to say, different managers will get different means and standard deviations even if the devs change nothing about their work habits. If they have a bad measurement rubric, certainly. But that in turn goes to questions of gauge reliability, a topic about which SPC has a lot to say. I should say by way of agreement that I'm not convinced that SPC can be safely and effectively applied to software development processes. But I'm also not convinced it can't. I don't have a firm, final position on the matter. I would need to apply it for a while and see for myself.
      • roenxi 3y ago
        > For sure -- but these are surely special or assignable causes of variation. Well, for starters they are process changes. The team is working in a different way to normal and, as expected, getting a different result. So it is special variance, but because management has decided that the team should adopt different processes with different statistical characteristics. These aren't one-offs, this is routine for how a well managed software team should perform. Priorities ought to change as the situation develops. And regardless, regular variations are what indicates that a process that isn't under control. If a process is under control, variances should be remarkably rare. Every month in software engineering is out of the ordinary. That is why SPC doesn't work; there isn't a repeatable process to model. There are lots of different processes chained together as vague specifications are continuously translated into logically formal specifications in an environment of continuous renegotiation. It might be possible to make a software team work in a controlled way, but it is pretty stupid - either the team won't fix some bugs to make the metrics look good, or they have to fake bugs from time to time to make the metrics look reasonable. Both are inferior to non-statistical controls of prioritising bugs with reference to how challenging the fix seems likely to be. Say we're going to implement an application. The bugs aren't going to appear as a stable process, there'll be some sort of big wave up front, then successive waves as new features are identified. Eventually the application will be more or less finished and the engineering team will clean up the long tail of bugs. The mean and standard deviation of the work aren't predictable in advance and aren't in control as far as SPC is concerned, because those tools assume steady state. Smoothing that curve from a hill-shape to a flat line is not value-additive; it is bad management in its own right. Resources should be reallocated to fix bugs after a big release and then moved back to dev work. Now there are going to be some situations where SPC isn't crazy, but it is still a bad management tool for software because the SPC teams aren't going to be agile in the face of change, because all their tools will scream blue murder at them for no reason.
• kqr 3y ago
  Things in software development that have been stable in my experience: - Weekly deploy count - Weekly growth in lines of code - Fraction of candidates hired - Weekly number of bug fixes - Weekly number of problems with a third party collaborator - Many internal metrics generated by the software, reflecting usage etc. - Weekly number of consultant hours required - Monthly growth of feature flag count - Length of standup - Time required to complete "small" tasks (i.e. those that don't involve novelty) - Length of successful build in CI - Proportion of CI builds that fail at least one test - Growth of number of tasks in backlog I could go on. The point is that while much of the value of product development comes from novelty and variation, there are many parts of the process that remain the same from week to week. Figuring out what these parts are and getting variation out of them allows the developer to (a) focus creativity on systematically solving process problems instead of doing it ad hoc, and (b) let the process recede into the background and focus creativity on creating end-user value.
  • roenxi 3y ago
    Your heartbeat and breaths/week are also quite consistent. The issue here is that you have to have a reasonable theory of why variance in the metrics you're tracking will destroy value. And that means actual value, not whinging that standup goes too long. I like a short standup as much as anyone, but if that is a material driver of value destruction then your organisation is not ready for statistical quality control. Plus it probably goes over time consistently. And once you start looking at the value, the lesson of software is that high variance activity is often the value add. It is the week(s) where someone implements pivot tables in Excel that creates probably billions of dollars in value over all of humanity. If that turns up as a statistical anomaly in the metrics because their line manager didn't bug them to fix bugs that week, that is a problem with the metrics not the programmer. This isn't a Toyota production line (if you're interested in the history of this, that is no random example) where value is uniformly created with each car and optimising the daily process down to degree n creates value. This is software. The value isn't created in the same way and these tools are not powerful in driving value add decisions. Variance is untidy but by no means an enemy. It must be managed case by case in context.
    • 3y ago