4 ms·

I think there’s a tendency in software for people to start out without understanding all the complexities they’re going to encounter. I think this is just human

I think there’s a tendency in software for people to start out without understanding all the complexities they’re going to encounter. I think this is just human nature. When you start out doing RPC you think, I don’t want to bother with schemas, I don’t want to bother with hierarchical error codes, I don’t foresee the need to set the user’s password but not retrieve it. So you don’t want to bother with a technology which makes your life more difficult, to solve problems you don’t have and cannot foresee. So you choose something simple. But you run into all these problems anyway, because they exist, no matter if you were capable of foreseeing them or not. But by then it’s too late. You’ve written 50 KLOC and you just have to keep going. I believe this is why many technologies become popular which are actually too simple to handle the types of problems they try to solve. I blogged about this concept here: https://www.databasesandlife.com/the-cycle-of-programming-languages/ https://www.databasesandlife.com/the-cycle-of-programming-la...

bthornbury 9y ago
You're right of course, like the first time you hit a race condition (with a week of debugging) and build a distributed lock system. You publish it and people find it useful! Only to realize later postgres offers fine locking capabilities far beyond what you've created (now that you get it). Then you realize that all anybody is doing is creating subsets of Erlang (half serious). So why aren't we all using that? In the grand scope, is it really a bad thing? We end up with X ways to solve problem Y and you never know X + 1 could have some advantages. Exploration should be encouraged IMO!
- mannykannot 9y ago
  Unfortunately, we get a whole lot of ad-hoc, informally-specified, bug-ridden, slow reinventions of the wheel for every better mousetrap. Usually (though not always) the best simple solutions come from those who understand where the complexity lies.
- andrewflnr 9y ago
  Exploration will be far more effective, and get farther, if it learns from previous expeditions, starts from established frontier outposts, etc. Is there a superset of Common Lisp and Erlang? Such a language would be unmatched for already containing everyone's clever ideas!
  - matt_kantor 9y ago
    > Is there a superset of Common Lisp and Erlang? http://lfe.io http://lfe.io :)
iovrthoughtthis 9y ago
I think that approach of creating abstractions and concepts just in time is good tbh. It allows for thing to get incrementally complex. I wish there were a way to manage concepts through that process so that you could have all of the simple, easy to change stuff for new concepts while using the more comprehensive and often complex stuff for more "hardened" concepts.
nitrogen 9y ago
Then you have phase two of engineer development, where having been burned by something surprising engineers overbuild everything. That's when you get technologies that are too complex to handle the problems they try to solve. That's when you get seven layers of dependency injection for something that could be a single-line algebraic statement with five variables.
hkon 9y ago
Yes, and it is this lack of understanding which ironically enough pushes technology forward. One RPC implementation after another.
barrkel 9y ago
Yup. It's the mechanism behind programming as a pop culture. Kids without a lot of experience are sick of the old way because it's too hairy and complicated, they come out with a fresh new approach that isn't nearly as broadly applicable, then it gets improved until it's fit for general purpose, at which point it's hairy and complicated and the cycle starts again. I don't think that everything is standing still, though. Usually each successive generation has an edge on the previous one; either the previous generation was constrained by memory or CPU or bandwidth and had self-limiting architecture because of it, or the next generation needs to solve a problem involving an order of magnitude more data or compute and it needs a different approach. But, of course, not everyone (or, realistically, not many people at all) is constrained by the thing that causes the revolution; people usually just get on the bandwagon because you must, if you don't you won't be as employable, won't be as hip, you'll find it harder to employ engineers to work on your project, etc.
- bluesnowmonkey 9y ago
  This is a self limiting mindset. "All good solutions to existing problems have been discovered. There may be new problems which need new solutions, but no one will ever improve on what we have already done. Anyone who things they can is a child playing in the dirt."
  - draw_down 9y ago
    Nope.
  - barrkel 9y ago
    Did you read the bit about why I think we're making progress?
- noonespecial 9y ago
  The best technologies can be understood and used in a simple case by a beginner but still "unfurl" to handle the general case. The worst force you to embrace the entire complexity before you can even hello world. Progress is being made.
borplk 9y ago
This is consistent with my experience. One element of it is also that in many cases the path to "becoming big" starts with bootstrapping and experimental projects that you may not know if it is going to survive and exist in a few years or if you are going to throw it away in 3 months. So in those cases you may still choose to be scrappy, knowing that it will come back to bite you, but at that stage that is thought of the success scenario and a good problem to have (if it happens at all). This is in contrast to large serious projects for large existing companies where you can much more confidently know that X and Y are going to be required because from day 1 you know the project is not just some casual thing. This is also why I think the software creation process and tools need to seriously think about having adjustable safety/pain knobs to allow for cheap scrappy prototyping but also allow to tighten the screws for production. You can kind of see a glimpse of this between various programming languages, particularly in their type system. But the general concept is broader than just that.
- mwcampbell 9y ago
  It occurs to me that young people may be better at bootstrapping in a scrappy way, because they don't yet have the knowledge and experience to immediately consider all the things that would be required in a mature implementation. Speaking for myself, now that I'm approaching middle age (37), if I contemplate developing a new product from scratch, I risk paralysis by analysis, overthinking every aspect of it. I certainly didn't do that when I was 21. Scrappy bootstrapping is a double-edged sword. On the one hand, it brings us great new products. On the other hand, ignoring some real-world concerns can be a major problem for users. As just one example, consider the impact for people with disabilities (e.g. blind, mobility impaired) who need to use an app that was developed with no regard for accessibility. And I've blissfully ignored other real-world concerns myself. For instance, the first desktop app that I worked on (in my 20s) had no support for HTTP proxies (as often found in corporate networks back then).
andybak 9y ago
I think this is best labelled a meta-Greenspun: > "Any sufficiently complicated X contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Y." (where X is the new "simple" way and Y is whatever the greybeards are using.
digi_owl 9y ago
Applies just as much to "middleware" as languages, sadly. And thus we have a constant churn of dependencies an broken APIs/ABIs...
thijsvandien 9y ago
That's exactly what YAGNI/KISS encourages. Don't make things more complicated than they need to be right now. That includes used technologies. I'd much rather use a simplistic tool while I still get away with it, risking having to switch to a more complicated tool later, than starting off with something way too complicated for what I need. Chances are the difficulty isn't in switching from REST to SOAP, or the other way around, but in dealing with the all the assumptions that permeate those 50 KLOC. In the end it just comes down to having a clean code base that can be steered in another direction. Note that I only mean not dealing with what cannot be easily be foreseen. Turning a blind eye to what you should know will soon be a problem, is a different story. As an example, if you ignore concurrency from the start, that will be hard to set straight later.
- amarkov 9y ago
  But you can't just decide to avoid making assumptions, because the shape of your API strongly dictates the shape of your handler methods. I've never seen an API migration that didn't involve rewriting nearly the entire API layer. Even worse, you have to maintain both the old and new versions of the API code, because turning off the old API endpoint is going to take at least a year.

bthornbury 9y ago

You're right of course, like the first time you hit a race condition (with a week of debugging) and build a distributed lock system. You publish it and people find it useful! Only to realize later postgres offers fine locking capabilities far beyond what you've created (now that you get it). Then you realize that all anybody is doing is creating subsets of Erlang (half serious). So why aren't we all using that? In the grand scope, is it really a bad thing? We end up with X ways to solve problem Y and you never know X + 1 could have some advantages. Exploration should be encouraged IMO!

mannykannot 9y ago
Unfortunately, we get a whole lot of ad-hoc, informally-specified, bug-ridden, slow reinventions of the wheel for every better mousetrap. Usually (though not always) the best simple solutions come from those who understand where the complexity lies.
andrewflnr 9y ago
Exploration will be far more effective, and get farther, if it learns from previous expeditions, starts from established frontier outposts, etc. Is there a superset of Common Lisp and Erlang? Such a language would be unmatched for already containing everyone's clever ideas!
- matt_kantor 9y ago
  > Is there a superset of Common Lisp and Erlang? http://lfe.io http://lfe.io :)

iovrthoughtthis 9y ago

I think that approach of creating abstractions and concepts just in time is good tbh. It allows for thing to get incrementally complex. I wish there were a way to manage concepts through that process so that you could have all of the simple, easy to change stuff for new concepts while using the more comprehensive and often complex stuff for more "hardened" concepts.

nitrogen 9y ago

Then you have phase two of engineer development, where having been burned by something surprising engineers overbuild everything. That's when you get technologies that are too complex to handle the problems they try to solve. That's when you get seven layers of dependency injection for something that could be a single-line algebraic statement with five variables.

hkon 9y ago

Yes, and it is this lack of understanding which ironically enough pushes technology forward. One RPC implementation after another.

barrkel 9y ago

Yup. It's the mechanism behind programming as a pop culture. Kids without a lot of experience are sick of the old way because it's too hairy and complicated, they come out with a fresh new approach that isn't nearly as broadly applicable, then it gets improved until it's fit for general purpose, at which point it's hairy and complicated and the cycle starts again. I don't think that everything is standing still, though. Usually each successive generation has an edge on the previous one; either the previous generation was constrained by memory or CPU or bandwidth and had self-limiting architecture because of it, or the next generation needs to solve a problem involving an order of magnitude more data or compute and it needs a different approach. But, of course, not everyone (or, realistically, not many people at all) is constrained by the thing that causes the revolution; people usually just get on the bandwagon because you must, if you don't you won't be as employable, won't be as hip, you'll find it harder to employ engineers to work on your project, etc.

bluesnowmonkey 9y ago
This is a self limiting mindset. "All good solutions to existing problems have been discovered. There may be new problems which need new solutions, but no one will ever improve on what we have already done. Anyone who things they can is a child playing in the dirt."
- draw_down 9y ago
  Nope.
- barrkel 9y ago
  Did you read the bit about why I think we're making progress?
noonespecial 9y ago
The best technologies can be understood and used in a simple case by a beginner but still "unfurl" to handle the general case. The worst force you to embrace the entire complexity before you can even hello world. Progress is being made.

borplk 9y ago

This is consistent with my experience. One element of it is also that in many cases the path to "becoming big" starts with bootstrapping and experimental projects that you may not know if it is going to survive and exist in a few years or if you are going to throw it away in 3 months. So in those cases you may still choose to be scrappy, knowing that it will come back to bite you, but at that stage that is thought of the success scenario and a good problem to have (if it happens at all). This is in contrast to large serious projects for large existing companies where you can much more confidently know that X and Y are going to be required because from day 1 you know the project is not just some casual thing. This is also why I think the software creation process and tools need to seriously think about having adjustable safety/pain knobs to allow for cheap scrappy prototyping but also allow to tighten the screws for production. You can kind of see a glimpse of this between various programming languages, particularly in their type system. But the general concept is broader than just that.

mwcampbell 9y ago
It occurs to me that young people may be better at bootstrapping in a scrappy way, because they don't yet have the knowledge and experience to immediately consider all the things that would be required in a mature implementation. Speaking for myself, now that I'm approaching middle age (37), if I contemplate developing a new product from scratch, I risk paralysis by analysis, overthinking every aspect of it. I certainly didn't do that when I was 21. Scrappy bootstrapping is a double-edged sword. On the one hand, it brings us great new products. On the other hand, ignoring some real-world concerns can be a major problem for users. As just one example, consider the impact for people with disabilities (e.g. blind, mobility impaired) who need to use an app that was developed with no regard for accessibility. And I've blissfully ignored other real-world concerns myself. For instance, the first desktop app that I worked on (in my 20s) had no support for HTTP proxies (as often found in corporate networks back then).

andybak 9y ago

I think this is best labelled a meta-Greenspun: > "Any sufficiently complicated X contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Y." (where X is the new "simple" way and Y is whatever the greybeards are using.

digi_owl 9y ago

Applies just as much to "middleware" as languages, sadly. And thus we have a constant churn of dependencies an broken APIs/ABIs...

thijsvandien 9y ago

That's exactly what YAGNI/KISS encourages. Don't make things more complicated than they need to be right now. That includes used technologies. I'd much rather use a simplistic tool while I still get away with it, risking having to switch to a more complicated tool later, than starting off with something way too complicated for what I need. Chances are the difficulty isn't in switching from REST to SOAP, or the other way around, but in dealing with the all the assumptions that permeate those 50 KLOC. In the end it just comes down to having a clean code base that can be steered in another direction. Note that I only mean not dealing with what cannot be easily be foreseen. Turning a blind eye to what you should know will soon be a problem, is a different story. As an example, if you ignore concurrency from the start, that will be hard to set straight later.

amarkov 9y ago
But you can't just decide to avoid making assumptions, because the shape of your API strongly dictates the shape of your handler methods. I've never seen an API migration that didn't involve rewriting nearly the entire API layer. Even worse, you have to maintain both the old and new versions of the API code, because turning off the old API endpoint is going to take at least a year.