13 ms·

Is Software Engineering Possible?

157 points by danarmak 9y ago

haskellandchill 9y ago
I hope things go this way. I'm betting my future on it.
- archgoon 9y ago
  The best way to predict the future is to invent it ;) Good luck!
walshemj 9y ago
Yes
- eeZah7Ux 9y ago
  It depends on what the product is. Industrial production of software is as difficult as industrial productions of theorems, patents, scientific discoveries and cocktail recipes. If the "product" is solving a real-world problem e.g. driving a robot car, than software becomes a tool, like an industrial machine. In that case software engineering exists, and it's doing very well, and it's about building reliable systems with the smallest amount of software and complexity a possible. Distributed systems that survive hardware failures and scale up/down automatically and machine learning are prime examples.
  - convolvatron 9y ago
    where are they building reliable systems with the smallest amount of software and complexity and possible? every place I seem to look people are turning trivial problems into huge and complicated problems by throwing more and more software onto the pile.
    - eeZah7Ux 9y ago
      Well-known tech companies that sell services instead of software. When a small team is entirely responsible for designing, developing and running a service there's a huge incentive to avoid unnecessary complexity and maximize reliability...
    - daics 9y ago
      Ethereum, MakerDAO
johnbender 9y ago
FSCQ is a really great example of a large system with proofs of correctness using extraction from Coq. Another well known project is CompCert the certified C compiler [1]. Which has seen a fair amount of external testing and use in verification of GCC and Clang as a reference for checking invalid compiled semantics [2] (to say nothing of compiling programs). 1 http://compcert.inria.fr http://compcert.inria.fr 2 https://blog.regehr.org/archives/1052 https://blog.regehr.org/archives/1052
deleted 9y ago
[deleted]
dang 9y ago
"Whither Software Engineering?", discussed in the OP, sounds like an interesting historical piece about how long it took for mathematics to be integrated into engineering work. The author is John Allen, who wrote Anatomy of Lisp. Can anyone find that paper online? https://news.ycombinator.com/item?id=962733 https://news.ycombinator.com/item?id=962733 points to a blog post about it from 2009, but none of the links seem to work.
- deleted 9y ago
  [deleted]
- vmarsy 9y ago
  You can find papers more easily one "Google scholar", I found it there: https://news.ycombinator.com/item?id=14528514 https://news.ycombinator.com/item?id=14528514
  - dang 9y ago
    Ah of course. Thanks!
- roskilli 9y ago
  It's here: https://pdfs.semanticscholar.org/3a6b/0506b8b922d7e6cc933a39dc165b5f34797e.pdf https://pdfs.semanticscholar.org/3a6b/0506b8b922d7e6cc933a39...
Joeri 9y ago
There is a movement going on from weak typing back to strong typing which is the same idea approached from the opposite (practical) end. Have the computer do more work to verify program correctness by rejecting invalid expressions. Maybe the two will meet at some point?
- Joe-Z 9y ago
  I never got the excitement that some of my college classmates had for JavaScript because they didn't have to 'deal with types'. I always much preferred a compiler telling me that this won't work instead of having to find some obscure type bug at runtime. Don't get me wrong, I still very much liked programming in JavaScript for other merits, especially the functional aspect or flexibility of extending your program simply in a browser command line window. EDIT: I kind of got distracted by my JavaScript-story here. What I originally wanted to post was: I always had an interest in philosophy and the logical thinking you can apply in software development (i.e. you can save yourself so much trouble by just thinking through your system thoroughly before writing even one line of code). However, the general sentinment that I also got from a lot of my classmates was that they didn't want have anything to do 'with any of this stuff' (this stuff being the humanities) but didn't have a problem with cracking hard math problems day in day out (which for me at least is pretty similar to using logical thinking / philosophy on a less abstract level). Sorry, if by my large edit I confused some people!
  - supernintendo 9y ago
    You should look into TypeScript if you haven't already. It's not perfect, but I think it strikes a nice balance between those two approaches. It's my "go-to" for new frontend projects because I enjoy the compiler's flexibility in regards to type checking. I can explore alternate approaches to solving the same problem (and by proxy, tests) and gradually introduce stronger compile-time checks as the solution comes together.
    - Joe-Z 9y ago
      Thanks for the input. In my place of work, where I mainly use C#/.NET, we are in the process of planning to implement a new frontend for one of our services. I was totally looking forward to using this to get some experience with TypeScript!
exabrial 9y ago
Weeks of coding can save you hours of planning.
- Joe-Z 9y ago
  Haha, you just summed up perfectly what I was trying to describe with my child-comment to another poster on here!
- cordite 9y ago
  I wear this shirt to work.
- LeoNatan25 9y ago
  But then it wouldn’t be called “hacking”, and so wouldn’t pass as software development these days. Instead, let’s have no planing whatsoever, no design and no requirements, have minimal time for actual development, “hack” code together and see where bugs take us. Aka “scrum”, “agile” and whatnot.
  - Joe-Z 9y ago
    I feel like the problems you describe here (constantly changing requirements, minimal time for development, "hacking" code together) have less to do with agile methods and more with people-stuff. I think agile methods can help a great deal in software development, especially breaking apart your projects into more discreet steps and supervising progress or giving the possibility for correcting course via sprints. Also using eXtreme programming tools like pair programming or code reviews to avoid having people slip into a frenzy of their individual domain / help keep obvious bugs out. Having minimal time and changing requirements is just part of the game. And people "hacking" code together is just... well bad craftsmanship really
    - javabean22 9y ago
      >>using eXtreme programming tools like pair programming or code reviews to avoid having people slip into a frenzy of their individual domain / help keep obvious bugs out. Wtf does this even mean? What are you talking about. If the bugs are obvious why do you need extreme programming to find them? What frenzy? Extreme programming is garbage.
personjerry 9y ago
No, it's necessary
tunesmith 9y ago
So what would such a curriculum look like? I've tried diving into this a few times and it often feels like hurling yourself against a wall in hopes of picking up bricks. You actually do make progress after a time but it's difficult to determine the curriculum flow. For me I think it started with reading a throwaway aside on "type verification" and wondering what the heck that was. After a bunch of wikipedia searching into different types of type calculus (not fun), I saw a reference on dependent types, which led me to Coq tutorials, and then, in an effort to be more practical, learning more Scala and a some Haskell - reasoning being that even if they didn't have full support for dependent types, at least their type systems were on the right track. Failed efforts so far have been learning TLA+ and Idris (I haven't yet found a way to apply any learning), and trying to get through some youtube lectures on homotopic type theory. Maybe next time.
- pjmlp 9y ago
  I don't know, maybe something like this, degree in informatics engineering (sorry Google translate): https://translate.google.com/translate?sl=pt&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http%3A%2F%2Fwww.di.fct.unl.pt%2Fensino%2Foferta-formativa%2Festrutura-curricular&edit-text=&act=url https://translate.google.com/translate?sl=pt&tl=en&js=y&prev... Check the "Consolidation Units" and "Specialization Units" required for earning enough credits. Also the degree is validated by the Portuguese Engineers Society. Just one example, most Portuguese universities offer similar degrees. I am a strong proponent of Engineering in our work and how software should be under the same quality regulations as other products in general, not only when there are human life's at stake.
  - reitanqild 9y ago
    Looks a lot like mine (also western European but with more sysadmin topics and less graphics and AI).
- haskellandchill 9y ago
  Luckily someone has paved the path for you :) https://functionalcs.github.io/curriculum/ https://functionalcs.github.io/curriculum/
esfandia 9y ago
Calling the discipline Software "Engineering" was the worst thing that could be done, at least given the current state of the art. Because of that, people thought you should write software the way you build a bridge or a car. The waterfall method came out of that, and it took decades to realize and recover. Software development is more reliant on discrete math (and more generally the ability to reason abstractly) than math that helps model the physical world. Hopefully, Professional Engineering bodies will one day realize this and will adapt their program accreditation criteria accordingly.
- nikki93 9y ago
  If you write software that needs discrete math, like probabilistic stuff or graphs, it will need discrete math. If you write software that is meant to simulate, like maybe even simulate real bridges, or video games, or generative art, non-photorealistic rendering, ... it will involve some of the math that helps model the physical world. It makes more sense to just learn a lot of things and apply the union of that to software, even ideas surrounding managing complexity in architecture (Notes on the Synthesis of Form?) and so on and expose that through oneself to "software" as a new universe where creations can exist; than to quibble about what's more important and what's not in a prescriptive way about what others want to and don't want to do. Of course, I'm doing the latter here. :) Such is the cycle.
- rmah 9y ago
  "Because of that, people thought you should write software the way you build a bridge or a car." The reason people think that is because they have no idea how the engineering to build a bridge or a car is done. Note, the term is Software ENGINEERING not Software Assembly or Software Construction. The engineering work necessary to design a bridge or car is actually not TOO dissimilar to designing anything else. You have requirements (often vague, conflicting and changing) and constraints and you have to trade things off against each other to come up with a workable design solution. The construction is just the compilation. Or perhaps the final coding/debugging. Or a combination. In construction, tiny engineering issues are often resolved by the construction crew at the site. Sometimes with input from engineers, sometimes not. The problem is that building software merges the engineering (design/architecture) with the construction (coding/debugging/compiling/installation) that requires a rather broad range of skills. Most people working on small teams have to jump back and forth between the big picture (more engineeringy) to the small details (more constructiony) and back. Finally, software development is no longer a new thing. People have been doing it for a half century. There are many people in the field today who's parents wrote software. And a few who's grandparents did it. It's high time we stop trying to draw analogies to building bridges or designing cars. Software Engineering is its own discipline with its own unique challenges and dynamics. Better to just take it for what it is.
westoncb 9y ago
I think it's probably not possible. Here's a way of thinking about why that may be: what we're doing when writing software has as much in common with doing science as it does with engineering. By 'science' I mean coming up with a formal description of how some physical system behaves. If we're willing to generalize this activity to include non-physical systems then, I'll argue, software development would be included as the same kind of thing. Fundamentally writing software is about constructing a formal description of a less formal 'specification' of some abstract system. The architecture used in this formal description corresponds to what's ordinarily called theory in science; when we execute our formal description, it corresponds to performing an experiment, which may lead to the discovery of anomalies (i.e. bugs); sometimes this anomalies are significant enough that we are led to revise our theory (i.e. refactor); if our re-conceptualization is significant enough and we decide to use an entirely new architecture, you're running into something like a paradigm shift with all the resistance and other social factors this entails. Coming up with architectures/theories to formally describe systems in the foregoing manner is not something we know how to make reliable. If that goal is clearly out of sight for science, why would we expect it to be achievable in the sibling discipline of software development? The exception I see to this (which admittedly is a large exception) is that probably most programs which need to be written are very similar to programs that have been written before. If we had some unified process for evaluating architectures for these programs that we write over and over again, then we'll probably eventually stabilize on 'true' architecture for that class of program. We see this happening to some extent in the creation of 'frameworks,' but maybe we could come up with better measures for comparing the aptitude of competing frameworks? It could also be that the physical systems described in science tend to be amenable to simpler description than typical software specs, which may be loaded with non-unifiable features. In this case, frameworks/theory only gets you so far because most of the work is just describing special cases... Anyway, maybe the 'science' we need to found software engineering on top of would be a formalization of this general process of coming up with formal descriptions of either abstract or physical systems. I bet there are regularities there which could be exploited for systematization—and if nothing else, there are likely insights to be had just through seeing software development as a thing having significant relations to the practice of science. Edit: phrasing, typos, etc.
- 9y ago
dkarapetyan 9y ago
I've looked into Coq and like a lot of others here tried to learn the theory and application but it is so far removed from my daily work that the benefits are basically non-existent. I've gotten way more mileage out of trying to learn systems theory and cybernetics than anything related to dependent types. People think this is a tool problem but it's not. Well, the tools kinda suck but fundamentally it comes down to culture and mind share. My background is in pure math but my programming knowledge is steeped in the hacker tradition. It is very hard to overcome all that momentum. There is no killer application for verified programming. Whatever you can do in Coq in 10 weeks you can do in C in 2 weeks with a few extra buffer overflows. We'd be further ahead if there were more tools for validating C code than trying to rewrite the world with dependent types.
- Joe-Z 9y ago
  You're right it does come down to culture and mind share. That's why the author already admits this would be a long process. But I think he's right and to truly call what we're doing software _engineering_ there needs to be a more formal basis for how to do things and especially to discern between this is right and this is wrong. No offense to you, but often times people that entered programming originally from related fields and taught it themselves give me the hardest time working with them. Like, no it's not enough that the software now provides this feature but you copied 1.200 lines of code for it and just squeezed in your if-branches and for-loops somewhere. Your job as software developer is to think how you can wed these new requirements with our existing codebase for others to build upon. Sorry, but as a bit of a pedantic and clean code enthusiast this is kind of a passion topic for me :)
lr4444lr 9y ago
When you build a bridge, a power station, or a wind turbine, you're creating something of known economic value for several decades, even centuries. The whole process is expected to have little to no business deliverables for years. That lends physical engineering nicely to an established set of standards. In software, we're often innovating entirely new products, and people want to see working proofs of concept, and see organic growth with the user base without getting locked in to preexisting concepts of what the product should do the business figures out what the market is. The nexus in products like vehicular and medical equipment software might be a good place to start implementing something like a discipline, but those systems are so much more singular and simplistic in their product function (not the implementation, I realize) than what the average software dev is working on that I don't even know how much downstream influence it would have.
- wbl 9y ago
  Your hot new gamified Pets.Com for mobile doesn't need to be engineered. A web server handling payments absolutely should be.
deleted 9y ago
[deleted]
mbell 9y ago
I think a significant issue in software is the comparatively unconstrained input space. I would imagine the average natural gas power plant would have more than a few issues if you fed it pure hydrogen instead of whatever methane based mixture it was designed for. Similarly I imagine there are other inputs that have tight specification, lubricants, etc. But, these inputs are relatively easy to control via both human and machine processes and there aren't that many combinations you would consider valid. There are complex inputs in the physical world, for example the impact of complex wind patterns on bridges, but I still feel this is simpler than many input spaces in software. In software, at some level you're almost always dealing with human or machine generated input which may be very complex in nature such that that even after all reasonable efforts at validation the size of the possible input space is massive if not effectively infinite. We're getting a bit better at dealing with this, e.g. fuzzing, but there is a long way to go for fuzzing to be viable on more complicated inputs.
protonfish 9y ago
The problem with this line of thinking is that a piece of software is ultimately built to solve a problem in our real, messy, uncertain world. There is no way to "prove" that an application served its intended purpose. As Fred Brooks wrote: "Even perfect program verification can only establish that a program meets its specification. […] Much of the essence of building a program is in fact the debugging of the specification."
- donovanm 9y ago
  Exactly by the time you could formally prove the application is correct, the market will have moved on to something different. Right now the combination of quick to develop and mostly correct seems to be more desirable to companies then slow to develop and formally correct. The success of languages like JavaScript and python show this.
nradov 9y ago
We have existence proofs that Software Engineering is possible. For example, look at what NASA did with the Space Shuttle flight control software. But for most business domains, real engineering simply isn't cost effective. https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff
powera 9y ago
There’s a terrible trend in the San Francisco tech scene where “engineer” is viewed as a synonym for “software engineer”, which is itself a synonym for “programming”. It isn’t. That said, I tend to disagree with people who believe the solution is "use Coq".
maxander 9y ago
There exists a body of theory about how to do software engineering as described here, but there doesn't exist a sizable community of software engineers willing to do this kind of work on a reasonably large scale. Judging by HN (which itself probably represents the upper-end of 'engaged' software developers), most people fidget uncomfortably when forced to explicitly declare derived types, and gnash their teeth when faced with Haskell monads; who among you would make a career out of writing Coq proofs about system correctness? Optimistically, the community might come around to realizing that this sort of development is at least desirable, and we might wind up with the basic components of popular OSes, or of firmware for dangerous things like cars and medical devices, being produced via provably-correct processes. But that will almost certainly wholly exhaust humanity's reserves of strongly math-educated software engineers (not to mention, industry's willingness to pay for it.) Software engineering may be impossible because most humans don't like math.
- LoSboccacc 9y ago
  > there doesn't exist a sizable community of software engineers willing to do this kind of work on a reasonably large scale also on the other side there's a very limited pool of employers willing to pay for this kind of engineering - aereospace and military mostly, and the second is cheapening out too.
- haskellandchill 9y ago
  I'm here for it, and I know a couple hundred working programmers personally who are with me.
ScottBurson 9y ago
Edsger Dijkstra famously derided software engineering as "how to program when you cannot". But if being able to program means being able to write large programs and get them perfectly correct, then no one has ever been able to program, the great Prof. Dr. Dijkstra not excepted. We all have puny brains next to a million lines of code. That being the case, the study of "how to program when you cannot" is critical for those of us who actually need to write these large programs. I like to say that software engineering is the art of managing complexity. Designing abstractions to be as general as possible, with interfaces as simple and clean as possible, and minimizing coupling, makes it much easier to think about how the major components of the system interact. And that's the key: if it's too hard to think about the system at a high level of abstraction, because the interfaces have too many special cases in them, more mistakes will be made. I don't think I can agree with the OP that formal verification is the only thing that can fairly be called software engineering. I do agree that the more verification we can figure out how to do, the better. But I would point out that well-designed abstractions are still important even when you're doing formal verification; indeed, possibly more so, because without them, the proofs get to be even more complicated and difficult than they need to be. It's really the same phenomenon: systems that are hard to think about are also hard to prove things about.
- ellius 9y ago
  I think Rich Hickey's "Simple Made Easy" is a perfect example. He enumerates an intellectual toolbox and why it's useful. He admits "you may not end up here [using these tools] but there's no reason not to start here." There are clearly cognitive and technical benefits to arranging software in particular architectures. That seems like engineering to me. You can't make an ideal engine either; it depends on whether you're powering a car or an airplane. Solving domain specific problems through technical architecting is about a good a definition of engineering as you can get. Software certainly fits that definition.
- haskellandchill 9y ago
  We can use mathematics to formalize what a "well-designed abstraction" is, so there is not much of a conflict between the two ways of thinking.
faragon 9y ago
In my opinion, "Software Engineering" has two main problems: - Incompetent management. - Incompetent programmers. Incompetent management, because of "sky is the limit" effect, pretending being some Steve Jobs or similar, instead of doing the real job: risk management, resource allocation, put competent people in charge, and being involved so the project(s) are kept on track. Incompetent programmers, because not being qualified enough, not having enough experience, or whatever reason making the programmer being nihilist, or a mix of the previous. If you have competent management, and competent programmers, you'll deliver. The problem comes when you have incompetent and/or corrupt management, taking decisions from emotional impulses, or thinking in their bonus first, hiding the problems, and making the "bomb" explode years later. Then, you'll receive the excuses: legacy code, architectural problems, blaming contractors, processes, methodology, etc.
mtraven 9y ago
Nice title, although I don't think the proposed solution (formal proof methods) is the answer. My own view is that building software is more of a design process than an engineering process, for the simple reason that software rarely has the kind of fixed requirements that make engineering solutions possible. Most software is constantly evolving and the factors that make it possible to evolve well include things like conceptual clarity and modularity, about which formal methods have nothing to say. There was a movement around 20 years ago to establish a field of practice called"software design" but I don't think it went anywhere. The problems remain the same though. http://hci.stanford.edu/publications/bds/ http://hci.stanford.edu/publications/bds/
leovonl 9y ago
Computer Science is a science field, therefore computer scientists usually see themselves as something more than an engineer - you don't want to apply the same knowledge over and over to make something exactly as the known knowledge tells you to do, you want to try new things and experiment with new ideas. That's the general mentality of the degree by itself. Real "software engineering" would have to be done with Agda, Coq, or similarly tools - theorem provers that guarantee your solution is sound. It is done that way for software that absolutely cannot fail (eg, aerospatial). Unfortunately, engineers are usually too far away from this kind of knowledge - and sometimes even doubt this is possible or doable, even if you direct them to the research (!). By the way, software engineering disciplines are usually depressive OO garbage, frameworks and all that non-sense that comes with excessive Java usage, and frankly never teach anything about reliability or even proving programs correct - which should have been their primary goal.
- pjmlp 9y ago
  Having a degree in informatics engineering I strongly disagree with that, given the lectures we had as part of our curriculum.
ebcode 9y ago
I'd like to contribute a definition of the words "engineer" and "engineering" from an old dictionary I had lying around. From Webster's (1948): engineer: n. 1. [Rare], a person who makes engines. 2. a person skilled or occupied in some branch of engineering: as, a mechanical engineer, an electrical engineer. 3. the operator of an engine; especially the driver of a railroad locomotive. 4. in military science, a member of that branch of the army which is concerned with the construction and demolition of bridges, roads, and fortifications, the laying and sapping of mines, etc. Abbreviated E., e., eng., engin., engr. v.t. 1. to plan, construct, or manage as an engineer; hence, 2. to contrive; manage skillfully; superintend; guide (a measure, action, etc. through). engineering: n. 1. the planning, designing, construction, or management of machinery, roads, bridges, buildings, fortifications, waterways, etc.; science, profession, or work of an engineer: abbreviated E., e., eng., egin. 2. a maneuvering or managing.
millisecond 9y ago
Feels like some stability in computing generations (CPU/GPU) or interaction paradigms (mobile) will be necessary to formalize software engineering. Until bridges/etc had a relative standard building each one was a craft. Not to say there hasn't been generational tech in traditional engineering, but generally not on the order of every few years.
- Jtsummers 9y ago
  It'll help, but it's not essential. There are practices that we can adopt today that will formalize software engineering. We should look to systems engineering and modeling (as used by other engineering disciplines: prototypes, mockups, math models, etc.). We already have many of the necessary formalisms, either from the fields of Math and CS, or from other engineering disciplines (notably systems engineering). The problem for software engineering is that it's really easy to model with code, and those have an unfortunate tendency to become the final deliverable. We have short iteration cycles that make the benefit of systems engineering and large scale engineering approaches less obvious. In CE it's quite obvious that a model of a bridge (physical and mathematical) will have benefits towards the final bridge construction. It's also obvious that it's just a model. In software, the model is too often functional enough to pass as the real thing, but fails in various ways: was constructed haphazardly allowing security vulnerabilities or stability issues; wasn't designed to run quickly or to scale effectively across multiple servers; not developed with maintenance in mind.
nunez 9y ago
Teams that are 100% invested on having clean, tested and reliable code can have software engineering, absolutely. The problem is that outside of the tech industry, most of the people with the pursestrings don't see the value in it most of the time. Why? When their main widget makes $100M/month (for example) and its current maintenance, as fucked as it might be, is ($1-10M/month, or 1-10% of revenue), spending, say, $1-10M to drop that OpEx to half of current costs, prima facie, doesn't make a lot of sense. What changes that equation is when some event that could have been prevented by good software engineering principles occurs. Good examples: the latest British Airways IT meltdown, LinkedIn's massive data breach, 100's of millions of CC's stolen from TJ Maxx and others. You see, when something happens that threatens that $100M/month cash machine, many millions of dollars of bonuses, raises, hirings, parties, expansion plans, and other general symbols of growth and progress get threatened as well. THAT is what people with the pursestrings don't want to see messed with. The overall problem with this is that it's reactionary, and every reactive event has a lifespan. So it takes finding someone with connections that can either capitalize on an unfortunate event trailblazing a path for software engineering to happen, or tell a good enough story of previous woes and misfortunes that can make that happen all the same. TL;DR: If you want "software engineering" to happen at your company, you need to tell a compelling story of how NOT making it happen will lead to great losses, and then you need to be extremely patient.

157 points by danarmak 9y ago

haskellandchill 9y ago

I hope things go this way. I'm betting my future on it.

archgoon 9y ago
The best way to predict the future is to invent it ;) Good luck!

walshemj 9y ago

Yes

eeZah7Ux 9y ago
It depends on what the product is. Industrial production of software is as difficult as industrial productions of theorems, patents, scientific discoveries and cocktail recipes. If the "product" is solving a real-world problem e.g. driving a robot car, than software becomes a tool, like an industrial machine. In that case software engineering exists, and it's doing very well, and it's about building reliable systems with the smallest amount of software and complexity a possible. Distributed systems that survive hardware failures and scale up/down automatically and machine learning are prime examples.
- convolvatron 9y ago
  where are they building reliable systems with the smallest amount of software and complexity and possible? every place I seem to look people are turning trivial problems into huge and complicated problems by throwing more and more software onto the pile.
  - eeZah7Ux 9y ago
    Well-known tech companies that sell services instead of software. When a small team is entirely responsible for designing, developing and running a service there's a huge incentive to avoid unnecessary complexity and maximize reliability...
  - daics 9y ago
    Ethereum, MakerDAO

johnbender 9y ago

FSCQ is a really great example of a large system with proofs of correctness using extraction from Coq. Another well known project is CompCert the certified C compiler [1]. Which has seen a fair amount of external testing and use in verification of GCC and Clang as a reference for checking invalid compiled semantics [2] (to say nothing of compiling programs). 1 http://compcert.inria.fr http://compcert.inria.fr 2 https://blog.regehr.org/archives/1052 https://blog.regehr.org/archives/1052

deleted 9y ago

[deleted]

dang 9y ago

"Whither Software Engineering?", discussed in the OP, sounds like an interesting historical piece about how long it took for mathematics to be integrated into engineering work. The author is John Allen, who wrote Anatomy of Lisp. Can anyone find that paper online? https://news.ycombinator.com/item?id=962733 https://news.ycombinator.com/item?id=962733 points to a blog post about it from 2009, but none of the links seem to work.

deleted 9y ago
[deleted]
vmarsy 9y ago
You can find papers more easily one "Google scholar", I found it there: https://news.ycombinator.com/item?id=14528514 https://news.ycombinator.com/item?id=14528514
- dang 9y ago
  Ah of course. Thanks!
roskilli 9y ago
It's here: https://pdfs.semanticscholar.org/3a6b/0506b8b922d7e6cc933a39dc165b5f34797e.pdf https://pdfs.semanticscholar.org/3a6b/0506b8b922d7e6cc933a39...

Joeri 9y ago

There is a movement going on from weak typing back to strong typing which is the same idea approached from the opposite (practical) end. Have the computer do more work to verify program correctness by rejecting invalid expressions. Maybe the two will meet at some point?

Joe-Z 9y ago
I never got the excitement that some of my college classmates had for JavaScript because they didn't have to 'deal with types'. I always much preferred a compiler telling me that this won't work instead of having to find some obscure type bug at runtime. Don't get me wrong, I still very much liked programming in JavaScript for other merits, especially the functional aspect or flexibility of extending your program simply in a browser command line window. EDIT: I kind of got distracted by my JavaScript-story here. What I originally wanted to post was: I always had an interest in philosophy and the logical thinking you can apply in software development (i.e. you can save yourself so much trouble by just thinking through your system thoroughly before writing even one line of code). However, the general sentinment that I also got from a lot of my classmates was that they didn't want have anything to do 'with any of this stuff' (this stuff being the humanities) but didn't have a problem with cracking hard math problems day in day out (which for me at least is pretty similar to using logical thinking / philosophy on a less abstract level). Sorry, if by my large edit I confused some people!
- supernintendo 9y ago
  You should look into TypeScript if you haven't already. It's not perfect, but I think it strikes a nice balance between those two approaches. It's my "go-to" for new frontend projects because I enjoy the compiler's flexibility in regards to type checking. I can explore alternate approaches to solving the same problem (and by proxy, tests) and gradually introduce stronger compile-time checks as the solution comes together.
  - Joe-Z 9y ago
    Thanks for the input. In my place of work, where I mainly use C#/.NET, we are in the process of planning to implement a new frontend for one of our services. I was totally looking forward to using this to get some experience with TypeScript!

exabrial 9y ago

Weeks of coding can save you hours of planning.

Joe-Z 9y ago
Haha, you just summed up perfectly what I was trying to describe with my child-comment to another poster on here!
cordite 9y ago
I wear this shirt to work.
LeoNatan25 9y ago
But then it wouldn’t be called “hacking”, and so wouldn’t pass as software development these days. Instead, let’s have no planing whatsoever, no design and no requirements, have minimal time for actual development, “hack” code together and see where bugs take us. Aka “scrum”, “agile” and whatnot.
- Joe-Z 9y ago
  I feel like the problems you describe here (constantly changing requirements, minimal time for development, "hacking" code together) have less to do with agile methods and more with people-stuff. I think agile methods can help a great deal in software development, especially breaking apart your projects into more discreet steps and supervising progress or giving the possibility for correcting course via sprints. Also using eXtreme programming tools like pair programming or code reviews to avoid having people slip into a frenzy of their individual domain / help keep obvious bugs out. Having minimal time and changing requirements is just part of the game. And people "hacking" code together is just... well bad craftsmanship really
  - javabean22 9y ago
    >>using eXtreme programming tools like pair programming or code reviews to avoid having people slip into a frenzy of their individual domain / help keep obvious bugs out. Wtf does this even mean? What are you talking about. If the bugs are obvious why do you need extreme programming to find them? What frenzy? Extreme programming is garbage.

personjerry 9y ago

No, it's necessary

tunesmith 9y ago

So what would such a curriculum look like? I've tried diving into this a few times and it often feels like hurling yourself against a wall in hopes of picking up bricks. You actually do make progress after a time but it's difficult to determine the curriculum flow. For me I think it started with reading a throwaway aside on "type verification" and wondering what the heck that was. After a bunch of wikipedia searching into different types of type calculus (not fun), I saw a reference on dependent types, which led me to Coq tutorials, and then, in an effort to be more practical, learning more Scala and a some Haskell - reasoning being that even if they didn't have full support for dependent types, at least their type systems were on the right track. Failed efforts so far have been learning TLA+ and Idris (I haven't yet found a way to apply any learning), and trying to get through some youtube lectures on homotopic type theory. Maybe next time.

pjmlp 9y ago
I don't know, maybe something like this, degree in informatics engineering (sorry Google translate): https://translate.google.com/translate?sl=pt&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http%3A%2F%2Fwww.di.fct.unl.pt%2Fensino%2Foferta-formativa%2Festrutura-curricular&edit-text=&act=url https://translate.google.com/translate?sl=pt&tl=en&js=y&prev... Check the "Consolidation Units" and "Specialization Units" required for earning enough credits. Also the degree is validated by the Portuguese Engineers Society. Just one example, most Portuguese universities offer similar degrees. I am a strong proponent of Engineering in our work and how software should be under the same quality regulations as other products in general, not only when there are human life's at stake.
- reitanqild 9y ago
  Looks a lot like mine (also western European but with more sysadmin topics and less graphics and AI).
haskellandchill 9y ago
Luckily someone has paved the path for you :) https://functionalcs.github.io/curriculum/ https://functionalcs.github.io/curriculum/

esfandia 9y ago

Calling the discipline Software "Engineering" was the worst thing that could be done, at least given the current state of the art. Because of that, people thought you should write software the way you build a bridge or a car. The waterfall method came out of that, and it took decades to realize and recover. Software development is more reliant on discrete math (and more generally the ability to reason abstractly) than math that helps model the physical world. Hopefully, Professional Engineering bodies will one day realize this and will adapt their program accreditation criteria accordingly.

nikki93 9y ago
If you write software that needs discrete math, like probabilistic stuff or graphs, it will need discrete math. If you write software that is meant to simulate, like maybe even simulate real bridges, or video games, or generative art, non-photorealistic rendering, ... it will involve some of the math that helps model the physical world. It makes more sense to just learn a lot of things and apply the union of that to software, even ideas surrounding managing complexity in architecture (Notes on the Synthesis of Form?) and so on and expose that through oneself to "software" as a new universe where creations can exist; than to quibble about what's more important and what's not in a prescriptive way about what others want to and don't want to do. Of course, I'm doing the latter here. :) Such is the cycle.
rmah 9y ago
"Because of that, people thought you should write software the way you build a bridge or a car." The reason people think that is because they have no idea how the engineering to build a bridge or a car is done. Note, the term is Software ENGINEERING not Software Assembly or Software Construction. The engineering work necessary to design a bridge or car is actually not TOO dissimilar to designing anything else. You have requirements (often vague, conflicting and changing) and constraints and you have to trade things off against each other to come up with a workable design solution. The construction is just the compilation. Or perhaps the final coding/debugging. Or a combination. In construction, tiny engineering issues are often resolved by the construction crew at the site. Sometimes with input from engineers, sometimes not. The problem is that building software merges the engineering (design/architecture) with the construction (coding/debugging/compiling/installation) that requires a rather broad range of skills. Most people working on small teams have to jump back and forth between the big picture (more engineeringy) to the small details (more constructiony) and back. Finally, software development is no longer a new thing. People have been doing it for a half century. There are many people in the field today who's parents wrote software. And a few who's grandparents did it. It's high time we stop trying to draw analogies to building bridges or designing cars. Software Engineering is its own discipline with its own unique challenges and dynamics. Better to just take it for what it is.

westoncb 9y ago

I think it's probably not possible. Here's a way of thinking about why that may be: what we're doing when writing software has as much in common with doing science as it does with engineering. By 'science' I mean coming up with a formal description of how some physical system behaves. If we're willing to generalize this activity to include non-physical systems then, I'll argue, software development would be included as the same kind of thing. Fundamentally writing software is about constructing a formal description of a less formal 'specification' of some abstract system. The architecture used in this formal description corresponds to what's ordinarily called theory in science; when we execute our formal description, it corresponds to performing an experiment, which may lead to the discovery of anomalies (i.e. bugs); sometimes this anomalies are significant enough that we are led to revise our theory (i.e. refactor); if our re-conceptualization is significant enough and we decide to use an entirely new architecture, you're running into something like a paradigm shift with all the resistance and other social factors this entails. Coming up with architectures/theories to formally describe systems in the foregoing manner is not something we know how to make reliable. If that goal is clearly out of sight for science, why would we expect it to be achievable in the sibling discipline of software development? The exception I see to this (which admittedly is a large exception) is that probably most programs which need to be written are very similar to programs that have been written before. If we had some unified process for evaluating architectures for these programs that we write over and over again, then we'll probably eventually stabilize on 'true' architecture for that class of program. We see this happening to some extent in the creation of 'frameworks,' but maybe we could come up with better measures for comparing the aptitude of competing frameworks? It could also be that the physical systems described in science tend to be amenable to simpler description than typical software specs, which may be loaded with non-unifiable features. In this case, frameworks/theory only gets you so far because most of the work is just describing special cases... Anyway, maybe the 'science' we need to found software engineering on top of would be a formalization of this general process of coming up with formal descriptions of either abstract or physical systems. I bet there are regularities there which could be exploited for systematization—and if nothing else, there are likely insights to be had just through seeing software development as a thing having significant relations to the practice of science. Edit: phrasing, typos, etc.

9y ago

dkarapetyan 9y ago

I've looked into Coq and like a lot of others here tried to learn the theory and application but it is so far removed from my daily work that the benefits are basically non-existent. I've gotten way more mileage out of trying to learn systems theory and cybernetics than anything related to dependent types. People think this is a tool problem but it's not. Well, the tools kinda suck but fundamentally it comes down to culture and mind share. My background is in pure math but my programming knowledge is steeped in the hacker tradition. It is very hard to overcome all that momentum. There is no killer application for verified programming. Whatever you can do in Coq in 10 weeks you can do in C in 2 weeks with a few extra buffer overflows. We'd be further ahead if there were more tools for validating C code than trying to rewrite the world with dependent types.

Joe-Z 9y ago
You're right it does come down to culture and mind share. That's why the author already admits this would be a long process. But I think he's right and to truly call what we're doing software _engineering_ there needs to be a more formal basis for how to do things and especially to discern between this is right and this is wrong. No offense to you, but often times people that entered programming originally from related fields and taught it themselves give me the hardest time working with them. Like, no it's not enough that the software now provides this feature but you copied 1.200 lines of code for it and just squeezed in your if-branches and for-loops somewhere. Your job as software developer is to think how you can wed these new requirements with our existing codebase for others to build upon. Sorry, but as a bit of a pedantic and clean code enthusiast this is kind of a passion topic for me :)

lr4444lr 9y ago

When you build a bridge, a power station, or a wind turbine, you're creating something of known economic value for several decades, even centuries. The whole process is expected to have little to no business deliverables for years. That lends physical engineering nicely to an established set of standards. In software, we're often innovating entirely new products, and people want to see working proofs of concept, and see organic growth with the user base without getting locked in to preexisting concepts of what the product should do the business figures out what the market is. The nexus in products like vehicular and medical equipment software might be a good place to start implementing something like a discipline, but those systems are so much more singular and simplistic in their product function (not the implementation, I realize) than what the average software dev is working on that I don't even know how much downstream influence it would have.

wbl 9y ago
Your hot new gamified Pets.Com for mobile doesn't need to be engineered. A web server handling payments absolutely should be.

deleted 9y ago

[deleted]

mbell 9y ago

I think a significant issue in software is the comparatively unconstrained input space. I would imagine the average natural gas power plant would have more than a few issues if you fed it pure hydrogen instead of whatever methane based mixture it was designed for. Similarly I imagine there are other inputs that have tight specification, lubricants, etc. But, these inputs are relatively easy to control via both human and machine processes and there aren't that many combinations you would consider valid. There are complex inputs in the physical world, for example the impact of complex wind patterns on bridges, but I still feel this is simpler than many input spaces in software. In software, at some level you're almost always dealing with human or machine generated input which may be very complex in nature such that that even after all reasonable efforts at validation the size of the possible input space is massive if not effectively infinite. We're getting a bit better at dealing with this, e.g. fuzzing, but there is a long way to go for fuzzing to be viable on more complicated inputs.

protonfish 9y ago

The problem with this line of thinking is that a piece of software is ultimately built to solve a problem in our real, messy, uncertain world. There is no way to "prove" that an application served its intended purpose. As Fred Brooks wrote: "Even perfect program verification can only establish that a program meets its specification. […] Much of the essence of building a program is in fact the debugging of the specification."

donovanm 9y ago
Exactly by the time you could formally prove the application is correct, the market will have moved on to something different. Right now the combination of quick to develop and mostly correct seems to be more desirable to companies then slow to develop and formally correct. The success of languages like JavaScript and python show this.

nradov 9y ago

We have existence proofs that Software Engineering is possible. For example, look at what NASA did with the Space Shuttle flight control software. But for most business domains, real engineering simply isn't cost effective. https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff

powera 9y ago

There’s a terrible trend in the San Francisco tech scene where “engineer” is viewed as a synonym for “software engineer”, which is itself a synonym for “programming”. It isn’t. That said, I tend to disagree with people who believe the solution is "use Coq".

maxander 9y ago

There exists a body of theory about how to do software engineering as described here, but there doesn't exist a sizable community of software engineers willing to do this kind of work on a reasonably large scale. Judging by HN (which itself probably represents the upper-end of 'engaged' software developers), most people fidget uncomfortably when forced to explicitly declare derived types, and gnash their teeth when faced with Haskell monads; who among you would make a career out of writing Coq proofs about system correctness? Optimistically, the community might come around to realizing that this sort of development is at least desirable, and we might wind up with the basic components of popular OSes, or of firmware for dangerous things like cars and medical devices, being produced via provably-correct processes. But that will almost certainly wholly exhaust humanity's reserves of strongly math-educated software engineers (not to mention, industry's willingness to pay for it.) Software engineering may be impossible because most humans don't like math.

LoSboccacc 9y ago
> there doesn't exist a sizable community of software engineers willing to do this kind of work on a reasonably large scale also on the other side there's a very limited pool of employers willing to pay for this kind of engineering - aereospace and military mostly, and the second is cheapening out too.
haskellandchill 9y ago
I'm here for it, and I know a couple hundred working programmers personally who are with me.

ScottBurson 9y ago

Edsger Dijkstra famously derided software engineering as "how to program when you cannot". But if being able to program means being able to write large programs and get them perfectly correct, then no one has ever been able to program, the great Prof. Dr. Dijkstra not excepted. We all have puny brains next to a million lines of code. That being the case, the study of "how to program when you cannot" is critical for those of us who actually need to write these large programs. I like to say that software engineering is the art of managing complexity. Designing abstractions to be as general as possible, with interfaces as simple and clean as possible, and minimizing coupling, makes it much easier to think about how the major components of the system interact. And that's the key: if it's too hard to think about the system at a high level of abstraction, because the interfaces have too many special cases in them, more mistakes will be made. I don't think I can agree with the OP that formal verification is the only thing that can fairly be called software engineering. I do agree that the more verification we can figure out how to do, the better. But I would point out that well-designed abstractions are still important even when you're doing formal verification; indeed, possibly more so, because without them, the proofs get to be even more complicated and difficult than they need to be. It's really the same phenomenon: systems that are hard to think about are also hard to prove things about.

ellius 9y ago
I think Rich Hickey's "Simple Made Easy" is a perfect example. He enumerates an intellectual toolbox and why it's useful. He admits "you may not end up here [using these tools] but there's no reason not to start here." There are clearly cognitive and technical benefits to arranging software in particular architectures. That seems like engineering to me. You can't make an ideal engine either; it depends on whether you're powering a car or an airplane. Solving domain specific problems through technical architecting is about a good a definition of engineering as you can get. Software certainly fits that definition.
haskellandchill 9y ago
We can use mathematics to formalize what a "well-designed abstraction" is, so there is not much of a conflict between the two ways of thinking.

faragon 9y ago

In my opinion, "Software Engineering" has two main problems: - Incompetent management. - Incompetent programmers. Incompetent management, because of "sky is the limit" effect, pretending being some Steve Jobs or similar, instead of doing the real job: risk management, resource allocation, put competent people in charge, and being involved so the project(s) are kept on track. Incompetent programmers, because not being qualified enough, not having enough experience, or whatever reason making the programmer being nihilist, or a mix of the previous. If you have competent management, and competent programmers, you'll deliver. The problem comes when you have incompetent and/or corrupt management, taking decisions from emotional impulses, or thinking in their bonus first, hiding the problems, and making the "bomb" explode years later. Then, you'll receive the excuses: legacy code, architectural problems, blaming contractors, processes, methodology, etc.

mtraven 9y ago

Nice title, although I don't think the proposed solution (formal proof methods) is the answer. My own view is that building software is more of a design process than an engineering process, for the simple reason that software rarely has the kind of fixed requirements that make engineering solutions possible. Most software is constantly evolving and the factors that make it possible to evolve well include things like conceptual clarity and modularity, about which formal methods have nothing to say. There was a movement around 20 years ago to establish a field of practice called"software design" but I don't think it went anywhere. The problems remain the same though. http://hci.stanford.edu/publications/bds/ http://hci.stanford.edu/publications/bds/

leovonl 9y ago

Computer Science is a science field, therefore computer scientists usually see themselves as something more than an engineer - you don't want to apply the same knowledge over and over to make something exactly as the known knowledge tells you to do, you want to try new things and experiment with new ideas. That's the general mentality of the degree by itself. Real "software engineering" would have to be done with Agda, Coq, or similarly tools - theorem provers that guarantee your solution is sound. It is done that way for software that absolutely cannot fail (eg, aerospatial). Unfortunately, engineers are usually too far away from this kind of knowledge - and sometimes even doubt this is possible or doable, even if you direct them to the research (!). By the way, software engineering disciplines are usually depressive OO garbage, frameworks and all that non-sense that comes with excessive Java usage, and frankly never teach anything about reliability or even proving programs correct - which should have been their primary goal.

pjmlp 9y ago
Having a degree in informatics engineering I strongly disagree with that, given the lectures we had as part of our curriculum.

ebcode 9y ago

I'd like to contribute a definition of the words "engineer" and "engineering" from an old dictionary I had lying around. From Webster's (1948): engineer: n. 1. [Rare], a person who makes engines. 2. a person skilled or occupied in some branch of engineering: as, a mechanical engineer, an electrical engineer. 3. the operator of an engine; especially the driver of a railroad locomotive. 4. in military science, a member of that branch of the army which is concerned with the construction and demolition of bridges, roads, and fortifications, the laying and sapping of mines, etc. Abbreviated E., e., eng., engin., engr. v.t. 1. to plan, construct, or manage as an engineer; hence, 2. to contrive; manage skillfully; superintend; guide (a measure, action, etc. through). engineering: n. 1. the planning, designing, construction, or management of machinery, roads, bridges, buildings, fortifications, waterways, etc.; science, profession, or work of an engineer: abbreviated E., e., eng., egin. 2. a maneuvering or managing.

millisecond 9y ago

Feels like some stability in computing generations (CPU/GPU) or interaction paradigms (mobile) will be necessary to formalize software engineering. Until bridges/etc had a relative standard building each one was a craft. Not to say there hasn't been generational tech in traditional engineering, but generally not on the order of every few years.

Jtsummers 9y ago
It'll help, but it's not essential. There are practices that we can adopt today that will formalize software engineering. We should look to systems engineering and modeling (as used by other engineering disciplines: prototypes, mockups, math models, etc.). We already have many of the necessary formalisms, either from the fields of Math and CS, or from other engineering disciplines (notably systems engineering). The problem for software engineering is that it's really easy to model with code, and those have an unfortunate tendency to become the final deliverable. We have short iteration cycles that make the benefit of systems engineering and large scale engineering approaches less obvious. In CE it's quite obvious that a model of a bridge (physical and mathematical) will have benefits towards the final bridge construction. It's also obvious that it's just a model. In software, the model is too often functional enough to pass as the real thing, but fails in various ways: was constructed haphazardly allowing security vulnerabilities or stability issues; wasn't designed to run quickly or to scale effectively across multiple servers; not developed with maintenance in mind.

nunez 9y ago

Teams that are 100% invested on having clean, tested and reliable code can have software engineering, absolutely. The problem is that outside of the tech industry, most of the people with the pursestrings don't see the value in it most of the time. Why? When their main widget makes $100M/month (for example) and its current maintenance, as fucked as it might be, is ($1-10M/month, or 1-10% of revenue), spending, say, $1-10M to drop that OpEx to half of current costs, prima facie, doesn't make a lot of sense. What changes that equation is when some event that could have been prevented by good software engineering principles occurs. Good examples: the latest British Airways IT meltdown, LinkedIn's massive data breach, 100's of millions of CC's stolen from TJ Maxx and others. You see, when something happens that threatens that $100M/month cash machine, many millions of dollars of bonuses, raises, hirings, parties, expansion plans, and other general symbols of growth and progress get threatened as well. THAT is what people with the pursestrings don't want to see messed with. The overall problem with this is that it's reactionary, and every reactive event has a lifespan. So it takes finding someone with connections that can either capitalize on an unfortunate event trailblazing a path for software engineering to happen, or tell a good enough story of previous woes and misfortunes that can make that happen all the same. TL;DR: If you want "software engineering" to happen at your company, you need to tell a compelling story of how NOT making it happen will lead to great losses, and then you need to be extremely patient.