8 ms·

Edsger Dijkstra famously derided software engineering as "how to program when you cannot". But if being able to program means being able to write large program

Edsger Dijkstra famously derided software engineering as "how to program when you cannot". But if being able to program means being able to write large programs and get them perfectly correct, then no one has ever been able to program, the great Prof. Dr. Dijkstra not excepted. We all have puny brains next to a million lines of code. That being the case, the study of "how to program when you cannot" is critical for those of us who actually need to write these large programs. I like to say that software engineering is the art of managing complexity. Designing abstractions to be as general as possible, with interfaces as simple and clean as possible, and minimizing coupling, makes it much easier to think about how the major components of the system interact. And that's the key: if it's too hard to think about the system at a high level of abstraction, because the interfaces have too many special cases in them, more mistakes will be made. I don't think I can agree with the OP that formal verification is the only thing that can fairly be called software engineering. I do agree that the more verification we can figure out how to do, the better. But I would point out that well-designed abstractions are still important even when you're doing formal verification; indeed, possibly more so, because without them, the proofs get to be even more complicated and difficult than they need to be. It's really the same phenomenon: systems that are hard to think about are also hard to prove things about.

ellius 9y ago
I think Rich Hickey's "Simple Made Easy" is a perfect example. He enumerates an intellectual toolbox and why it's useful. He admits "you may not end up here [using these tools] but there's no reason not to start here." There are clearly cognitive and technical benefits to arranging software in particular architectures. That seems like engineering to me. You can't make an ideal engine either; it depends on whether you're powering a car or an airplane. Solving domain specific problems through technical architecting is about a good a definition of engineering as you can get. Software certainly fits that definition.
haskellandchill 9y ago
We can use mathematics to formalize what a "well-designed abstraction" is, so there is not much of a conflict between the two ways of thinking.
- ScottBurson 9y ago
  Please elaborate!
  - haskellandchill 9y ago
    I can't, my understanding of category theory and universal algebra is only enough to sniff at it. Joseph Goguen's work is a good direction to look to if you are equipped with those tools. He cites Parnas's work which is the easier to grasp analysis of modular abstraction. Here is a representative paper: A hidden agenda Joseph Goguena, Grant Malcolmb http://ac.els-cdn.com/S0304397599002753/1-s2.0-S0304397599002753-main.pdf?_tid=4cc6e324-4e3f-11e7-bf5b-00000aacb361&acdnat=1497142123_773ab59c590d4c067c585b032db64b5e http://ac.els-cdn.com/S0304397599002753/1-s2.0-S030439759900...
Koshkin 9y ago
Actually, it is thinking in general that is the art of managing complexity. Software engineering is no different from any other type of engineering, in that it is also the art of making reasonable compromises.
- ScottBurson 9y ago
  Making reasonable compromises is certainly important. But I think software engineering is different, because the state spaces of programs -- particularly large ones, but even most small ones -- are astronomically larger than those of any other kind of engineered artifact. Most procedures contain one or more conditional branches, which test some aspect of the state of the system and then take one or another action based on that. If you're not careful, you wind up with a system where you have to think about all of those conditionals at the same time, which becomes exponentially difficult as the system grows. Other kinds of engineering certainly involve complexity, but they don't involve this much complexity.
barrkel 9y ago
Designing abstractions to be as general as possible, with interfaces as simple and clean as possible, and minimizing coupling, makes it much easier to think about how the major components of the system interact. I think this is a false statement. In fact I think you're confusing modularity with abstraction. Abstraction hides complexity, and abstractions are leaky; how the whole behaves can be strongly influenced by behaviours that leak from the abstractions. It took a couple of goes, but eventually we realized that DCOM and CORBA were poor ways to design distributed applications. Abstractions should be as general as necessary, where necessity is subject to judgement. Make code too abstract, and people may not understand it because it's not concrete enough, full of indirections and compositions and factory-factories; make it too concrete, and people may not understand it because it's too complex. Creating an abstraction to share some code may pay off if there's sufficient clients for the code and they all have a similar usage profile, but it may be a false economy if the clients are few or the usage varied. Modularity isn't an unalloyed good either. Too many modules and you may write code that acts optimally in a local way, but suboptimally globally because it has no access or awareness of other modules' behaviour. Modules designed around an abstraction may be substituted, but designing for module replacement is more often than not over-engineering. It's strongly encouraged by modern OO testing approaches, but IMO it puts the cart before the horse. I'd like more people to realize that abstractions represent unknowns, and that any given implementation needs to be tested in an engineering materials sense, to find out what its force limits are, how it reacts under scale and pressure, so people using the abstraction are not just thinking about the abstraction, they are considering the real thing they're dealing with as well. It saves a lot of time going down dead ends.
- jnwatson 9y ago
  That we're not all programming in machine code is evidence that some abstractions are useful. The most vital abstraction is that most software is blithely unaware of the physical construct embodying it. Abstraction isn't the enemy. I think much of what we call good "software engineering" is identifying, designing, and implementing the "right" abstraction.
- ScottBurson 9y ago
  I'm not sure I understand what you're saying, but it's also not clear to me that we really disagree. Abstractions are often leaky, yes; sometimes unavoidably so, and then those leakages have to be taken into account in the design of the system; but when leakage is avoidable, do you not agree that it's important to avoid it? I do agree that there's such a thing as over-abstraction, and over-engineering in general. I recall a discussion here on HN, oh, probably 2 or 3 years ago if not more, where somebody said they had written an app and then hired a programmer to maintain it, and the programmer started by "enterprisifying" it, adding lots of dependency injection and factory-factories and suchlike, rendering the code almost unrecognizable to the initial developer. Without having seen the code in both "before" and "after" states, I can't form a firm opinion about whether this was really an improvement, but I could easily believe that it was not. Certainly if I were the initial developer, I would want to know specifically what the benefit of each of these changes was expected to be.
801699 9y ago
"But if being able to program means being able to write large programs..." What does it mean when someone is only able (or only wants) to write small programs? For example, small programs that can work together to form a "system" for managing something. Is that not considered "software engineering"? If not, what is it called? (Embedded software engineering? But not all small software is "embedded".) What has a better chance of being error-free, a small program or a large one? Should "ability to program" be measured by how large a program one can write? For example, if someone writes a small UNIX utility, but does not write a large program, should we assume he does not know how to program?
- RangerScience 9y ago
  No program is small, as it must include the operating system. Now - you could go down to machine code, or work on embedded systems, etc etc - but the point remains. It's not just the software you are writing that's part your program, it's also all the other, existing software (and physical designs) it makes use of, or interacts with. Edit: Might be called "software systems engineering", or maybe "dev ops"?
  - Jtsummers 9y ago
    That's not really accurate. I can write a small program that does not include the OS, but rather assumes it. Just as a vehicle designer doesn't have to include the gas or electric distribution infrastructure in their design, they can assume it. If I'm conforming to someone else's interface (say the C standard library), that's equivalent to an electrical engineer assuming various standard interfaces for the equipment they're designing. It's a design consideration, but not a design task. So the project itself may still remain quite small. When I'm designing the interfaces that others are using, or designing the interfaces between 100 [0] smaller programs so they can interact effectively with each other and their environment, then it's a larger engineering task. [0] Arbitrary number.
    - paulddraper 9y ago
      And likewise, the OS relies on the ISA, not having to know about chip design.

ellius 9y ago

I think Rich Hickey's "Simple Made Easy" is a perfect example. He enumerates an intellectual toolbox and why it's useful. He admits "you may not end up here [using these tools] but there's no reason not to start here." There are clearly cognitive and technical benefits to arranging software in particular architectures. That seems like engineering to me. You can't make an ideal engine either; it depends on whether you're powering a car or an airplane. Solving domain specific problems through technical architecting is about a good a definition of engineering as you can get. Software certainly fits that definition.

haskellandchill 9y ago

We can use mathematics to formalize what a "well-designed abstraction" is, so there is not much of a conflict between the two ways of thinking.

ScottBurson 9y ago
Please elaborate!
- haskellandchill 9y ago
  I can't, my understanding of category theory and universal algebra is only enough to sniff at it. Joseph Goguen's work is a good direction to look to if you are equipped with those tools. He cites Parnas's work which is the easier to grasp analysis of modular abstraction. Here is a representative paper: A hidden agenda Joseph Goguena, Grant Malcolmb http://ac.els-cdn.com/S0304397599002753/1-s2.0-S0304397599002753-main.pdf?_tid=4cc6e324-4e3f-11e7-bf5b-00000aacb361&acdnat=1497142123_773ab59c590d4c067c585b032db64b5e http://ac.els-cdn.com/S0304397599002753/1-s2.0-S030439759900...

Koshkin 9y ago

Actually, it is thinking in general that is the art of managing complexity. Software engineering is no different from any other type of engineering, in that it is also the art of making reasonable compromises.

ScottBurson 9y ago
Making reasonable compromises is certainly important. But I think software engineering is different, because the state spaces of programs -- particularly large ones, but even most small ones -- are astronomically larger than those of any other kind of engineered artifact. Most procedures contain one or more conditional branches, which test some aspect of the state of the system and then take one or another action based on that. If you're not careful, you wind up with a system where you have to think about all of those conditionals at the same time, which becomes exponentially difficult as the system grows. Other kinds of engineering certainly involve complexity, but they don't involve this much complexity.

barrkel 9y ago

Designing abstractions to be as general as possible, with interfaces as simple and clean as possible, and minimizing coupling, makes it much easier to think about how the major components of the system interact. I think this is a false statement. In fact I think you're confusing modularity with abstraction. Abstraction hides complexity, and abstractions are leaky; how the whole behaves can be strongly influenced by behaviours that leak from the abstractions. It took a couple of goes, but eventually we realized that DCOM and CORBA were poor ways to design distributed applications. Abstractions should be as general as necessary, where necessity is subject to judgement. Make code too abstract, and people may not understand it because it's not concrete enough, full of indirections and compositions and factory-factories; make it too concrete, and people may not understand it because it's too complex. Creating an abstraction to share some code may pay off if there's sufficient clients for the code and they all have a similar usage profile, but it may be a false economy if the clients are few or the usage varied. Modularity isn't an unalloyed good either. Too many modules and you may write code that acts optimally in a local way, but suboptimally globally because it has no access or awareness of other modules' behaviour. Modules designed around an abstraction may be substituted, but designing for module replacement is more often than not over-engineering. It's strongly encouraged by modern OO testing approaches, but IMO it puts the cart before the horse. I'd like more people to realize that abstractions represent unknowns, and that any given implementation needs to be tested in an engineering materials sense, to find out what its force limits are, how it reacts under scale and pressure, so people using the abstraction are not just thinking about the abstraction, they are considering the real thing they're dealing with as well. It saves a lot of time going down dead ends.

jnwatson 9y ago
That we're not all programming in machine code is evidence that some abstractions are useful. The most vital abstraction is that most software is blithely unaware of the physical construct embodying it. Abstraction isn't the enemy. I think much of what we call good "software engineering" is identifying, designing, and implementing the "right" abstraction.
ScottBurson 9y ago
I'm not sure I understand what you're saying, but it's also not clear to me that we really disagree. Abstractions are often leaky, yes; sometimes unavoidably so, and then those leakages have to be taken into account in the design of the system; but when leakage is avoidable, do you not agree that it's important to avoid it? I do agree that there's such a thing as over-abstraction, and over-engineering in general. I recall a discussion here on HN, oh, probably 2 or 3 years ago if not more, where somebody said they had written an app and then hired a programmer to maintain it, and the programmer started by "enterprisifying" it, adding lots of dependency injection and factory-factories and suchlike, rendering the code almost unrecognizable to the initial developer. Without having seen the code in both "before" and "after" states, I can't form a firm opinion about whether this was really an improvement, but I could easily believe that it was not. Certainly if I were the initial developer, I would want to know specifically what the benefit of each of these changes was expected to be.

801699 9y ago

"But if being able to program means being able to write large programs..." What does it mean when someone is only able (or only wants) to write small programs? For example, small programs that can work together to form a "system" for managing something. Is that not considered "software engineering"? If not, what is it called? (Embedded software engineering? But not all small software is "embedded".) What has a better chance of being error-free, a small program or a large one? Should "ability to program" be measured by how large a program one can write? For example, if someone writes a small UNIX utility, but does not write a large program, should we assume he does not know how to program?

RangerScience 9y ago
No program is small, as it must include the operating system. Now - you could go down to machine code, or work on embedded systems, etc etc - but the point remains. It's not just the software you are writing that's part your program, it's also all the other, existing software (and physical designs) it makes use of, or interacts with. Edit: Might be called "software systems engineering", or maybe "dev ops"?
- Jtsummers 9y ago
  That's not really accurate. I can write a small program that does not include the OS, but rather assumes it. Just as a vehicle designer doesn't have to include the gas or electric distribution infrastructure in their design, they can assume it. If I'm conforming to someone else's interface (say the C standard library), that's equivalent to an electrical engineer assuming various standard interfaces for the equipment they're designing. It's a design consideration, but not a design task. So the project itself may still remain quite small. When I'm designing the interfaces that others are using, or designing the interfaces between 100 [0] smaller programs so they can interact effectively with each other and their environment, then it's a larger engineering task. [0] Arbitrary number.
  - paulddraper 9y ago
    And likewise, the OS relies on the ISA, not having to know about chip design.