4 ms·

How does software get reliable at all, ever? In my experience there are two ways: 1. You write reliable software from the start. It feels a lot like writing a

How does software get reliable at all, ever? In my experience there are two ways: 1. You write reliable software from the start. It feels a lot like writing a mathematical proof, or at least sketching one. After a few dumb bugs are fixed the code continues to work without issue, year after year, and nobody ever looks at it again. (This isn't as hard as it sounds! It's not the right approach for everything, but it should be more widely used than it is.) 2. You write normal software, i.e. unreliable crap, but you look at the ways that it goes wrong and fix them. This requires that you put some work into detecting that things have gone wrong. A lot of software companies will trace all crashes, and aggregate stack traces and logs from them. In this kind of environment, you can improve software reliability by sticking a paranoid number of assertions in your code and obsessively checking to see what triggered them. After a while all the common problems will have been removed, and the remaining bugs will be less prevalent than hardware failures -- and then you can declare victory.

pif 7y ago
From my experience, the problem with no. 1 is that, in order to build something correct, you have to know what you need to build, and that is a can of worms (actually, a snakes nest) that too many people refuse to address. The problem with no. 2, on the other side, is that it will eventually get correct, but you cannot predict when it will be reliable. And the same people who refuse to sign requirement documents are the same people who insist on having arbitrary deadlines.
- rstuart4133 7y ago
  > The problem with no. 2, on the other side, is that it will eventually get correct Actually, it doesn't necessarily get correct. There are old programs out where the ratio of bugs fixed by a modification / bugs introduced is less than 1. For small systems this never becomes a problem, but huge complex systems it really is a thing. The only way I know of to prevent it from happening is unit tests with excellent coverage and CI.
moksly 7y ago
In my experience the majority of enterprise service applications and smaller few-purpose systems are build following 1. We’ve have an ASP Web Forms food ordering system that’s basically four forms and a simple CRUD admin system page that’s been running flawless for half a decade. It’s only needed upgrades to its .NET system as we moved up the TLS version ladder, and it has thousands of daily users. There are no tests, no real monitoring outside of “is it still up?” and no complaints. Eventually we’ll have to clean up the database, eventually. But chances are someone will have build something new by then.
- jaclaz 7y ago
  Maybe in this specific case there is also a factor of simplicity, simple designs, made with few elements, tend to be more robust, as a general rule.
  - moksly 7y ago
    We have 50 such systems, most don’t see the “thousands of users” a day though.
- bluecmd 7y ago
  I doubt that would be 1 and not developed using 2 followed by a "don't touch" policy, tbh
  - imtringued 7y ago
    We have one project under a don't touch policy. Sure it is probably the most stable project because no features are getting added but it is also becoming a maintenance nightmare the longer we accumulate interest on our technical debt. There is a version 2 planned that will add a lot more features and I'm sure we will spend the first weeks on upgrading the libraries it is using because they are a major version or two behind.
westoncb 7y ago
I think what you're describing here may be more about which domains are being modeled by software than it is about approaches to writing the software. You can't have a program that's like a mathematical proof if the concept for the program isn't itself like a mathematical structure. I agree in essence with your statement that the first approach should be more widely used. What that comes down to though is that the set of program concepts that do resemble mathematical structures is larger than most programmers realize—but this a separate question from whether that set is large at all relative to the totality of software being written. I'm pretty sure that, "It's not the right approach for everything," is putting it far too mildly. And the dichotomy designating software which isn't of this kind as "crap" is taking a shallow view of things: it's because the problem is more complex/irregular/unpredictable that corrective feedback must be relied on over a compact explicit model. The software isn't crap, the problem is just hard. I think we'd agree that whether the label "crap" is appropriate or not depends on whether it falls into that set of programs where 1) it is amenable to compact mathematical representation but 2) feedback methods were used instead. This is the set of Genuine Crap. Problem is, there is (ime) a much larger set of only apparent crap, which takes on that appearance as a direct consequence of the corrective feedback process. All of this leads to a state of affairs where through conflation of Apparent Crap with Genuine Crap, a perception has been created that normal software engineers are mostly doing it wrong in some way and should feel bad. I would encourage people to stave off that view until it's more definitively established that the sets of Apparent Crap and Genuine Crap actually have a large intersection. And maybe more importantly: it should be considered whether the crappiness truly derives from engineering decisions or business requirements. (I don't mean to rant too much at your comment specifically pjscott, it's just that this is a frequently expressed sentiment that I get tired of hearing and which seems to be possibly mistaken.)
- lincpa 7y ago
  There is a right approach for everything, That is the "Warehouse/Workshop Model" commonly used in industry. Enable software to be produced like integrated circuits. https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow
ryandrake 7y ago
The sad thing is most companies I've seen don't do either 1 or 2. They do: 3. Your write normal software, i.e. unreliable crap, and you fix only the bugs that block ship. Once you ship, you either: 3a. Don't touch it. Move on to a different project, leaving the software as perpetual crap, or 3b. Cram in more features, only fixing new bugs that prevent shipping those new features. Repeat 3b until nobody can maintain it anymore, then blame some nebulous "technical debt" for why we need a rewrite.

pif 7y ago

From my experience, the problem with no. 1 is that, in order to build something correct, you have to know what you need to build, and that is a can of worms (actually, a snakes nest) that too many people refuse to address. The problem with no. 2, on the other side, is that it will eventually get correct, but you cannot predict when it will be reliable. And the same people who refuse to sign requirement documents are the same people who insist on having arbitrary deadlines.

rstuart4133 7y ago
> The problem with no. 2, on the other side, is that it will eventually get correct Actually, it doesn't necessarily get correct. There are old programs out where the ratio of bugs fixed by a modification / bugs introduced is less than 1. For small systems this never becomes a problem, but huge complex systems it really is a thing. The only way I know of to prevent it from happening is unit tests with excellent coverage and CI.

moksly 7y ago

In my experience the majority of enterprise service applications and smaller few-purpose systems are build following 1. We’ve have an ASP Web Forms food ordering system that’s basically four forms and a simple CRUD admin system page that’s been running flawless for half a decade. It’s only needed upgrades to its .NET system as we moved up the TLS version ladder, and it has thousands of daily users. There are no tests, no real monitoring outside of “is it still up?” and no complaints. Eventually we’ll have to clean up the database, eventually. But chances are someone will have build something new by then.

jaclaz 7y ago
Maybe in this specific case there is also a factor of simplicity, simple designs, made with few elements, tend to be more robust, as a general rule.
- moksly 7y ago
  We have 50 such systems, most don’t see the “thousands of users” a day though.
bluecmd 7y ago
I doubt that would be 1 and not developed using 2 followed by a "don't touch" policy, tbh
- imtringued 7y ago
  We have one project under a don't touch policy. Sure it is probably the most stable project because no features are getting added but it is also becoming a maintenance nightmare the longer we accumulate interest on our technical debt. There is a version 2 planned that will add a lot more features and I'm sure we will spend the first weeks on upgrading the libraries it is using because they are a major version or two behind.

westoncb 7y ago

I think what you're describing here may be more about which domains are being modeled by software than it is about approaches to writing the software. You can't have a program that's like a mathematical proof if the concept for the program isn't itself like a mathematical structure. I agree in essence with your statement that the first approach should be more widely used. What that comes down to though is that the set of program concepts that do resemble mathematical structures is larger than most programmers realize—but this a separate question from whether that set is large at all relative to the totality of software being written. I'm pretty sure that, "It's not the right approach for everything," is putting it far too mildly. And the dichotomy designating software which isn't of this kind as "crap" is taking a shallow view of things: it's because the problem is more complex/irregular/unpredictable that corrective feedback must be relied on over a compact explicit model. The software isn't crap, the problem is just hard. I think we'd agree that whether the label "crap" is appropriate or not depends on whether it falls into that set of programs where 1) it is amenable to compact mathematical representation but 2) feedback methods were used instead. This is the set of Genuine Crap. Problem is, there is (ime) a much larger set of only apparent crap, which takes on that appearance as a direct consequence of the corrective feedback process. All of this leads to a state of affairs where through conflation of Apparent Crap with Genuine Crap, a perception has been created that normal software engineers are mostly doing it wrong in some way and should feel bad. I would encourage people to stave off that view until it's more definitively established that the sets of Apparent Crap and Genuine Crap actually have a large intersection. And maybe more importantly: it should be considered whether the crappiness truly derives from engineering decisions or business requirements. (I don't mean to rant too much at your comment specifically pjscott, it's just that this is a frequently expressed sentiment that I get tired of hearing and which seems to be possibly mistaken.)

lincpa 7y ago
There is a right approach for everything, That is the "Warehouse/Workshop Model" commonly used in industry. Enable software to be produced like integrated circuits. https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow

ryandrake 7y ago

The sad thing is most companies I've seen don't do either 1 or 2. They do: 3. Your write normal software, i.e. unreliable crap, and you fix only the bugs that block ship. Once you ship, you either: 3a. Don't touch it. Move on to a different project, leaving the software as perpetual crap, or 3b. Cram in more features, only fixing new bugs that prevent shipping those new features. Repeat 3b until nobody can maintain it anymore, then blame some nebulous "technical debt" for why we need a rewrite.