5 ms·

So I have been thinking about software projects lately, and I have come to the conclusion that a lot of these tools/solutions exist to "build houses" when most

So I have been thinking about software projects lately, and I have come to the conclusion that a lot of these tools/solutions exist to "build houses" when most of us are just throwing together lean-to sheds and dog houses. Software projects today are naturally more complex and have more complex tooling the same way building a house today requires more knowledge and skill than it did 50 years ago. Then there are some folks/organizations building cathedrals, and the associated tooling (react, angular, maven, etc) and all the rest of us look up in awe and thing "well I guess if I want to be that good I need to use those tools on this dog house." But your dog house doesn't have the need to host parties, provide security, or even real weather protection other than a roof to keep the rain and sun out. Yet we all try to build our dog houses in ways that might be better if they are one day converted to a proper living quarters but likely will never have a need for running Water or windows.

colechristensen 6y ago
French cooking's mise en place or first order retrievability by Adam Savage or Lean Engineering or whatever. There's a common thread among them, having what you need, when you need it, where you need it, and understanding how to use it. This is what is lacking. The original Unix philosophy build by greybeards and university linguistics and language professors had this at it's core. The "do one thing well" combined with how the shell worked being driven by people who really really understood language and came up with one that made sense to them for interacting with computers... was, is still somewhat, wonderful. What's missing today is exactly what you mention. People designing tool sheds with materials often shoddily designed for cathedrals. Complexity. This is the enemy, the second enemy is bad attempts to reduce complexity which often end up adding more complexity than they take away, just harder to find. The favourite example of this is - perhaps apocryphal, but entirely believable - is replacing dozens of nodes using fancy big data tools with one node running sed/awk,etc. One thing is clear, nowhere I've been has had the tools readily available, the documentation clear and forthcoming, and the scale in the right range for projects. I found myself recently solving a problem with Hashicorp Vault using GCP to verify identity of machines wanting secrets. It was a stretch goal which had been on my plate for six months, every once in a while I would try to go back and figure out how to make it work, and months and months and months after trying, I put it together and it worked perfectly. The documentation to lead to this understanding had to be read out of order on several different pages with some lucky guesses to arrive at the solution, which in the end was just a few steps easily explained. Afterwards the documentation was fine and made perfect sense. Before I grokked the issue the documentation just seemed like a bunch on nonsense which led me to believe what I wanted to do wasn't possible in a constrained security environment. That is the kind of problem I solve all the time as someone with a decade of DevOps,SysAdmin,whatever experience behind me. Not using knowledge and tools to amplify what I do, but spending 60% of my time confused as hell about something which should be obvious and is only obvious afterwards, 20% trying to convince people of things they're often reluctant to believe, and 20% actually using built up knowledge and tools to do many many things very quickly. It's frustrating.
- breeny592 6y ago
  > Complexity. This is the enemy, the second enemy is bad attempts to reduce complexity which often end up adding more complexity than they take away, just harder to find. This is true at every level of the systems design process - often by trying to make a system "simpler" i.e. less complex for the end user, the complexity is shifted further down the stack into application code, or even to the underlying infrastructure. It's easy for those of us with technical backgrounds to see the beauty and simplicity in well designed interfaces, but as the realm of computing and computer interaction shifts away from technical to non-technical people, we start to absorb some of that complexity into our systems design to make up for knowledge shortcomings of end users. Your example of sed being better than the "fancy data tools" I feel is a good one - whilst sed is incredibly powerful for this use case, if the consumer of what needs to be run there only knows how to use excel, it's often required to create these abstraction layers to allow the end user to do their own primary function/role.
  - zrm 6y ago
    There is a difference between inherent complexity and manufactured complexity. Inherent complexity can only be moved around, but even then you still want to move it to a place that can handle it properly. If the best you can do is a complex decision tree then it should at least be well-audited and well-documented instead of having a dozen separate buggy ones that all give different results for unknown reasons. But much of today's complexity isn't inherent. It's manufactured. It's just not required to be there at all and there is much to be gained by taking it out.
- tlarkworthy 6y ago
  BTW Google Secrets Manager is rediculously easy to use. It's not quite at the level of Vault but it does enough for me and is 100x easier to deploy
- zrm 6y ago
  > Not using knowledge and tools to amplify what I do, but spending 60% of my time confused as hell about something which should be obvious and is only obvious afterwards, 20% trying to convince people of things they're often reluctant to believe, and 20% actually using built up knowledge and tools to do many many things very quickly. It's frustrating. It also creates some nasty perverse incentives. Because if you spend the hours to find the best way to do something that should be simple, it often turns out to actually be simple. API function #487 does just exactly what you need, if only it hadn't taken seven hours to find it. So then you spend a day writing five lines of code that should have taken half an hour, but in the end you have a good solution. But there is an alternative to that. You can reinvent the wheel. Instead of spending your time understanding the unnecessarily complicated thing with poor documentation, write some new code to do just your thing. And who looks more productive to the boss? The one who spent all day to write five lines of code, or the one who added a complex new feature that clearly took a lot of time and effort (but will now take ten times more effort to maintain)? This is especially nasty because it's not always an obvious answer. Sometimes the existing wheel is made of glass and is shaped like a triangle and reinventing it is totally worth it, and then you spend all day discovering that and still have to spend tomorrow doing what a lazier person did yesterday. Whereas without the unnecessary complexity the right answer would have been obvious much sooner.
m463 6y ago
Software engineering evolves to get the most out of your hardware. If you sort of turn that idea around in your head, software will always end up trying to barely work on the shittiest hardware. Think about it. Chips work on 30 nm. What's our next step, to make 30 nm more solid and reliable? No, it's to go to 20 nm or 10 nm and get the same software to work. The same goes with systems stuff. Do we work on having much more reliable cpus? No, we work on using lots more less-reliable cpus (or gpus) with more unreliable storage over less reliable interconnects and networks and get the software to (imperfectly) abstract away the problems.
- hinkley 6y ago
  I've heard it said that the reason bits of the Roman aqueducts are still standing is that they were overbuilt by at least a factor of 2. The craftsman often wants to overbuild something. It's part of what gets them out of bed in the morning, or lets them sleep at night. What if you - or someone else - got your margin of error wrong, like the Citibank building? A little extra to deal with unexpected situations isn't so bad.
  - jupp0r 6y ago
    If you compare the aqueducts to software, you might draw the completely wrong conclusions. Software that was extremely well built in the 70s is mostly gone (or if it's still there, it's not exactly admired by todays software craftsmen (looking at you, airline ticket booking systems)). What makes software endure over time is its ability to react to changes of wildly different varieties (market shifts, changing hardware, new technology, ...). The Linux kernel might still be popular in 30 years, but it will likely look very different from today. If it looks the same in the way the aqueducts do, it won't be relevant anymore and will most likely be forgotten eventually.
  - m463 6y ago
    > gets them out of bed in the morning I always thought you should do it as best as you could too. It took a bunch of time in the real world before I could actually grasp the concept of over-engineering in a meaningful way. The first example of over-engineering I could "get" was when someone (jwz?) ranted about over-engineered high-end audio. The deeper you look, having $1000 00-gauge cables just won't make anything sound better. It's marketing. The second example was japanese motorcycles. I started realizing that there was an envelope. If you made something bigger and stronger, the weight would go up and the acceleration would get worse. If you made it all motor, acceleration would go up, but handling would get worse. Some of the fastest bikes were actually lighter less powerful bikes that could still accelerate, but also go through turns. as an example, before the s1000rr era, BMW made big strong durable motorcycles... that weighed a lot and handled in a very mediocre fashion. (as to safety - yes, bringing safety or security into the equation ... you change the equation)
hinkley 6y ago
You have one group building dog houses like the dog will always be a puppy, and another group looking at those people, laughing, and wandering off to build a kennel, just in case you might decide at some point to start raising show dogs for 4 different breeds at once. What is most likely instead is that you'll get a second dog in a couple years and need to make a better version of what you already have. And if it's nice enough, make a couple more for friends or as a gift.
RangerScience 6y ago
"The stone mason lives in a wooden house."

colechristensen 6y ago

French cooking's mise en place or first order retrievability by Adam Savage or Lean Engineering or whatever. There's a common thread among them, having what you need, when you need it, where you need it, and understanding how to use it. This is what is lacking. The original Unix philosophy build by greybeards and university linguistics and language professors had this at it's core. The "do one thing well" combined with how the shell worked being driven by people who really really understood language and came up with one that made sense to them for interacting with computers... was, is still somewhat, wonderful. What's missing today is exactly what you mention. People designing tool sheds with materials often shoddily designed for cathedrals. Complexity. This is the enemy, the second enemy is bad attempts to reduce complexity which often end up adding more complexity than they take away, just harder to find. The favourite example of this is - perhaps apocryphal, but entirely believable - is replacing dozens of nodes using fancy big data tools with one node running sed/awk,etc. One thing is clear, nowhere I've been has had the tools readily available, the documentation clear and forthcoming, and the scale in the right range for projects. I found myself recently solving a problem with Hashicorp Vault using GCP to verify identity of machines wanting secrets. It was a stretch goal which had been on my plate for six months, every once in a while I would try to go back and figure out how to make it work, and months and months and months after trying, I put it together and it worked perfectly. The documentation to lead to this understanding had to be read out of order on several different pages with some lucky guesses to arrive at the solution, which in the end was just a few steps easily explained. Afterwards the documentation was fine and made perfect sense. Before I grokked the issue the documentation just seemed like a bunch on nonsense which led me to believe what I wanted to do wasn't possible in a constrained security environment. That is the kind of problem I solve all the time as someone with a decade of DevOps,SysAdmin,whatever experience behind me. Not using knowledge and tools to amplify what I do, but spending 60% of my time confused as hell about something which should be obvious and is only obvious afterwards, 20% trying to convince people of things they're often reluctant to believe, and 20% actually using built up knowledge and tools to do many many things very quickly. It's frustrating.

breeny592 6y ago
> Complexity. This is the enemy, the second enemy is bad attempts to reduce complexity which often end up adding more complexity than they take away, just harder to find. This is true at every level of the systems design process - often by trying to make a system "simpler" i.e. less complex for the end user, the complexity is shifted further down the stack into application code, or even to the underlying infrastructure. It's easy for those of us with technical backgrounds to see the beauty and simplicity in well designed interfaces, but as the realm of computing and computer interaction shifts away from technical to non-technical people, we start to absorb some of that complexity into our systems design to make up for knowledge shortcomings of end users. Your example of sed being better than the "fancy data tools" I feel is a good one - whilst sed is incredibly powerful for this use case, if the consumer of what needs to be run there only knows how to use excel, it's often required to create these abstraction layers to allow the end user to do their own primary function/role.
- zrm 6y ago
  There is a difference between inherent complexity and manufactured complexity. Inherent complexity can only be moved around, but even then you still want to move it to a place that can handle it properly. If the best you can do is a complex decision tree then it should at least be well-audited and well-documented instead of having a dozen separate buggy ones that all give different results for unknown reasons. But much of today's complexity isn't inherent. It's manufactured. It's just not required to be there at all and there is much to be gained by taking it out.
tlarkworthy 6y ago
BTW Google Secrets Manager is rediculously easy to use. It's not quite at the level of Vault but it does enough for me and is 100x easier to deploy
zrm 6y ago
> Not using knowledge and tools to amplify what I do, but spending 60% of my time confused as hell about something which should be obvious and is only obvious afterwards, 20% trying to convince people of things they're often reluctant to believe, and 20% actually using built up knowledge and tools to do many many things very quickly. It's frustrating. It also creates some nasty perverse incentives. Because if you spend the hours to find the best way to do something that should be simple, it often turns out to actually be simple. API function #487 does just exactly what you need, if only it hadn't taken seven hours to find it. So then you spend a day writing five lines of code that should have taken half an hour, but in the end you have a good solution. But there is an alternative to that. You can reinvent the wheel. Instead of spending your time understanding the unnecessarily complicated thing with poor documentation, write some new code to do just your thing. And who looks more productive to the boss? The one who spent all day to write five lines of code, or the one who added a complex new feature that clearly took a lot of time and effort (but will now take ten times more effort to maintain)? This is especially nasty because it's not always an obvious answer. Sometimes the existing wheel is made of glass and is shaped like a triangle and reinventing it is totally worth it, and then you spend all day discovering that and still have to spend tomorrow doing what a lazier person did yesterday. Whereas without the unnecessary complexity the right answer would have been obvious much sooner.

m463 6y ago

Software engineering evolves to get the most out of your hardware. If you sort of turn that idea around in your head, software will always end up trying to barely work on the shittiest hardware. Think about it. Chips work on 30 nm. What's our next step, to make 30 nm more solid and reliable? No, it's to go to 20 nm or 10 nm and get the same software to work. The same goes with systems stuff. Do we work on having much more reliable cpus? No, we work on using lots more less-reliable cpus (or gpus) with more unreliable storage over less reliable interconnects and networks and get the software to (imperfectly) abstract away the problems.

hinkley 6y ago
I've heard it said that the reason bits of the Roman aqueducts are still standing is that they were overbuilt by at least a factor of 2. The craftsman often wants to overbuild something. It's part of what gets them out of bed in the morning, or lets them sleep at night. What if you - or someone else - got your margin of error wrong, like the Citibank building? A little extra to deal with unexpected situations isn't so bad.
- jupp0r 6y ago
  If you compare the aqueducts to software, you might draw the completely wrong conclusions. Software that was extremely well built in the 70s is mostly gone (or if it's still there, it's not exactly admired by todays software craftsmen (looking at you, airline ticket booking systems)). What makes software endure over time is its ability to react to changes of wildly different varieties (market shifts, changing hardware, new technology, ...). The Linux kernel might still be popular in 30 years, but it will likely look very different from today. If it looks the same in the way the aqueducts do, it won't be relevant anymore and will most likely be forgotten eventually.
- m463 6y ago
  > gets them out of bed in the morning I always thought you should do it as best as you could too. It took a bunch of time in the real world before I could actually grasp the concept of over-engineering in a meaningful way. The first example of over-engineering I could "get" was when someone (jwz?) ranted about over-engineered high-end audio. The deeper you look, having $1000 00-gauge cables just won't make anything sound better. It's marketing. The second example was japanese motorcycles. I started realizing that there was an envelope. If you made something bigger and stronger, the weight would go up and the acceleration would get worse. If you made it all motor, acceleration would go up, but handling would get worse. Some of the fastest bikes were actually lighter less powerful bikes that could still accelerate, but also go through turns. as an example, before the s1000rr era, BMW made big strong durable motorcycles... that weighed a lot and handled in a very mediocre fashion. (as to safety - yes, bringing safety or security into the equation ... you change the equation)

hinkley 6y ago

You have one group building dog houses like the dog will always be a puppy, and another group looking at those people, laughing, and wandering off to build a kennel, just in case you might decide at some point to start raising show dogs for 4 different breeds at once. What is most likely instead is that you'll get a second dog in a couple years and need to make a better version of what you already have. And if it's nice enough, make a couple more for friends or as a gift.

RangerScience 6y ago

"The stone mason lives in a wooden house."