3 ms·

To date, software engineering has failed to deliver on one critical goal, which is why we are in this mess. The world of software development needs to be divid

To date, software engineering has failed to deliver on one critical goal, which is why we are in this mess. The world of software development needs to be divided in two -- the component creators, and the component assemblers. To stick with the given analogy, component creators are the people inventing new kinds of plumbing: easier ways of connecting pipes, taps that don't ever drip. Component assemblers are the people fitting out bathrooms. Creating new components is high-end engineering. It needs to happen far away from the day-to-day challenges of making a client happy. To a large extent, this division is already present, but it doesn't go far enough. There has been amazing progress - nowadays we work on top of an incredible stack of technology that we don't have to re-invent, but so far we've not achieved the "last mile". We'll know when we're there because component assembly (i.e. making something for a client) will start to look more like a trade. Today, just getting a regular been-done-a-million-times-before database-driven website (or whatever), requires FAR too much low-level code. This, I think is what is meant here by "hand made". We should be snapping things together, and often we are, but suddenly you get to an awkward bit and you're back to forging a new kind of pipe joint that never existed before.

kragen 14y ago
Programming has delivered spectacularly on that critical goal, although software engineering did indeed fail, which is why the "software crisis" ended decades ago. The division you propose already exists. The "component creators" are called "programmers". The "component assemblers" are called "users". That's why you can use, say, Google Docs Forms to slap together a signup form in ten minutes that would have taken you a couple of hours in PHP with MySQL in 2000.
redwood 14y ago
I think you're spot on. Just as the personal computer and GUI brought "computing" to the masses for their use and massive scale return on investment, we need to see the equivalent innovation in programming components. In other words just as it's now easier than ever to edit movies, write a paper, polish a photo: we need to see this ease of interface brought to programming. I know hard core CS types will say that making programming easy for the masses will mean it won't be fast, it won't allow for the optimal algorithms. However computing speed and bandwidth is rising exponentially and countless applications for software do not need to be as optimal as possible. In the future people who are not programmers but who are one step removed (e.g. okay with spreadsheets, SQL, some basic scripting when needed, visual programming like Labview, and who understand software architecture) will need to be able to do more complex things with computers, things that today only programmers can do. We need to see the separation between programmer and the technical creative masses disappear a bit. Or at least as tablatom says, there has to be room for two types: those who build the tools for easy programming and those who do easy programming.
philwelch 14y ago
This is one of those classic ideas that's been around forever. It was more or less implied when the idea of "code reuse" became popular alongside object-oriented programming, I think. But I don't think it's ever really going to happen, because one of two things will happen. One is that you'll run into a set of requirements that no one has run into before, not because any of the requirements is unusual but because the space of possible requirements is very big if not infinite in size, and once you combine requirements that space increases exponentially. The other is that you'll run into a requirement that's simpler to implement directly than it is to write the glue code for all the components you could use to solve it. Both of these more or less happen already; the real question is what would surmount these problems.
- tablatom 14y ago
  > You'll run into a set of requirements that no one has run into before. Inevitably. Which means you'll have to resort to "real programming" (as opposed to component assembly) for /that part/ of your project. I don't see this as a fundamental reason why component-assembly can never become viable. As we get better at creating flexible components, these situations will get less common, but they will never go away. > you'll run into a requirement that's simpler to implement directly than it is to write the glue code for all the components you could use to solve it 100% this. For me this pretty much sums up why component-assembly isn't viable today. For all but the simplest components this turns out to be the case. (e.g. date-picker, file-uploader, or maybe something a bit bigger with /very/ fixed requirements, like a disqus comment trail). But jumping from this to "I don't think it's every going to happen" is overly pessimistic. The glue code is too hard to write? We need a better way to write glue code. That could be a fundamentally different type of language, or a fundamentally different conception of what we mean by "component". (Aside: In my foolishness I am working on such things).
  - philwelch 14y ago
    > Inevitably. Which means you'll have to resort to "real programming" (as opposed to component assembly) for /that part/ of your project. It's not just that you'll run into one unique requirement; you might run into a unique combination of requirements, each of which already has proven solutions, but with no good way to glue it all together. That's the reason C programmers still sometimes write their own string handling or memory allocation code despite that stuff being literally in the standard library. Writing general purpose software components is hard. If you're creating a product, you know what kind of component you need, and you don't really care about anyone else. If you're making a general purpose component, you have almost literally no possible way of even comprehending, much less fulfilling, all the requirements of every product that could potentially use your component. You're always going to notice a difference between something that's been cobbled together out of spare parts and something that's been designed to fit an integral product vision. There's a reason we've been hearing about reusing program components for literally decades. I'm sure some chunk of the problem will be broken off and solved, some kind of standard solution to the CRUD app or something, but there are still going to be products out there that need real engineering, not just component assembly.
kaiwetzel 14y ago
I'm just reading the book Flow-Based Programming (2nd Edition) by Paul Morrison[1]. I would say the main idea is to shift focus from control-flow to how data ("information packets") flow through a network of (black box) components and only use traditional control-flow style programming for the most simple, atomic components. The book presents some convincing examples from business programming but I think the idea should work very well for other areas than text processing, certainly for image manipulation or sound processing. The split between component creators and component assemblers (application programmers) is highlighted in the book, I hate it how often I tend to slip from one role to the other in regular programming, maybe such an explicit split would help a lot (well, or certainly a lot of experience will ...) [1] http://www.jpaulmorrison.com/fbp/ http://www.jpaulmorrison.com/fbp/

kragen 14y ago

Programming has delivered spectacularly on that critical goal, although software engineering did indeed fail, which is why the "software crisis" ended decades ago. The division you propose already exists. The "component creators" are called "programmers". The "component assemblers" are called "users". That's why you can use, say, Google Docs Forms to slap together a signup form in ten minutes that would have taken you a couple of hours in PHP with MySQL in 2000.

redwood 14y ago

I think you're spot on. Just as the personal computer and GUI brought "computing" to the masses for their use and massive scale return on investment, we need to see the equivalent innovation in programming components. In other words just as it's now easier than ever to edit movies, write a paper, polish a photo: we need to see this ease of interface brought to programming. I know hard core CS types will say that making programming easy for the masses will mean it won't be fast, it won't allow for the optimal algorithms. However computing speed and bandwidth is rising exponentially and countless applications for software do not need to be as optimal as possible. In the future people who are not programmers but who are one step removed (e.g. okay with spreadsheets, SQL, some basic scripting when needed, visual programming like Labview, and who understand software architecture) will need to be able to do more complex things with computers, things that today only programmers can do. We need to see the separation between programmer and the technical creative masses disappear a bit. Or at least as tablatom says, there has to be room for two types: those who build the tools for easy programming and those who do easy programming.

philwelch 14y ago

This is one of those classic ideas that's been around forever. It was more or less implied when the idea of "code reuse" became popular alongside object-oriented programming, I think. But I don't think it's ever really going to happen, because one of two things will happen. One is that you'll run into a set of requirements that no one has run into before, not because any of the requirements is unusual but because the space of possible requirements is very big if not infinite in size, and once you combine requirements that space increases exponentially. The other is that you'll run into a requirement that's simpler to implement directly than it is to write the glue code for all the components you could use to solve it. Both of these more or less happen already; the real question is what would surmount these problems.

tablatom 14y ago
> You'll run into a set of requirements that no one has run into before. Inevitably. Which means you'll have to resort to "real programming" (as opposed to component assembly) for /that part/ of your project. I don't see this as a fundamental reason why component-assembly can never become viable. As we get better at creating flexible components, these situations will get less common, but they will never go away. > you'll run into a requirement that's simpler to implement directly than it is to write the glue code for all the components you could use to solve it 100% this. For me this pretty much sums up why component-assembly isn't viable today. For all but the simplest components this turns out to be the case. (e.g. date-picker, file-uploader, or maybe something a bit bigger with /very/ fixed requirements, like a disqus comment trail). But jumping from this to "I don't think it's every going to happen" is overly pessimistic. The glue code is too hard to write? We need a better way to write glue code. That could be a fundamentally different type of language, or a fundamentally different conception of what we mean by "component". (Aside: In my foolishness I am working on such things).
- philwelch 14y ago
  > Inevitably. Which means you'll have to resort to "real programming" (as opposed to component assembly) for /that part/ of your project. It's not just that you'll run into one unique requirement; you might run into a unique combination of requirements, each of which already has proven solutions, but with no good way to glue it all together. That's the reason C programmers still sometimes write their own string handling or memory allocation code despite that stuff being literally in the standard library. Writing general purpose software components is hard. If you're creating a product, you know what kind of component you need, and you don't really care about anyone else. If you're making a general purpose component, you have almost literally no possible way of even comprehending, much less fulfilling, all the requirements of every product that could potentially use your component. You're always going to notice a difference between something that's been cobbled together out of spare parts and something that's been designed to fit an integral product vision. There's a reason we've been hearing about reusing program components for literally decades. I'm sure some chunk of the problem will be broken off and solved, some kind of standard solution to the CRUD app or something, but there are still going to be products out there that need real engineering, not just component assembly.

kaiwetzel 14y ago

I'm just reading the book Flow-Based Programming (2nd Edition) by Paul Morrison[1]. I would say the main idea is to shift focus from control-flow to how data ("information packets") flow through a network of (black box) components and only use traditional control-flow style programming for the most simple, atomic components. The book presents some convincing examples from business programming but I think the idea should work very well for other areas than text processing, certainly for image manipulation or sound processing. The split between component creators and component assemblers (application programmers) is highlighted in the book, I hate it how often I tend to slip from one role to the other in regular programming, maybe such an explicit split would help a lot (well, or certainly a lot of experience will ...) [1] http://www.jpaulmorrison.com/fbp/ http://www.jpaulmorrison.com/fbp/