6 ms·

Everything is broken (2014)

84 points by gpvos 8y ago

NikolaNovak 8y ago
Interesting article. The first 75% I would send to my non-technical friends and family as well as the technical ones, as an awareness builder. The last 25% are a surprising call for action (surprising as from the tone of article, I expected it to be a fun rant only). It's tricky because a lot of people tend to discount knowledge imparted once they perceive it was with the objective of persuading you to The Cause. It makes the informational portion more suspect of cherry picking or bias. In other words, while I personally agree both with the technical rant, and the call to action, I feel majority of my friends would react negatively to two of them together, just on general principles/reflexes. Not sure what the solution would be; perhaps two linked articles - first the rant, and then a link to "Proposed solution" article at the bottom? Most people I know would react better if they felt they were in control of asking for solution guidance, rather than part of a seeming bait & switch...
commandlinefan 8y ago
One thing not addressed here is: I can make better software. I can fix these problems. I can find them first, and then fix them. I _can't_ do that when I'm under ridiculous time pressure from an "agile coach" who barely knows what an Excel macro is but who, for some reason, is responsible for deciding what gets prioritized and who gets a bonus and who gets downsized. I can't identify and correct complex technical problems when I'm attending a one-hour daily "scrum" explaining that yes, I'm still working on "task" 5398129 and yes, it was a 4 "point" story (because that's the maximum we're allowed), but as it turns out, almost everything interesting and useful is also complex -- in direct opposition to the mindset that drives the oxymoronic atrocity that people now refer to as "agile".
- sametmax 8y ago
  That's because security doesn't bring money on the table for most products. People are ok with their private life being violated, they don't mind discreet censorship or mass monitoring. Since we make money with those users, and because the market moves fast, the only security most companies accept to afford is protection against script kiddies and bad PR. But there is a reason the average Joe doesn't care about all those things: it works for him, most of the time. The amount of persons really having their life seriously disturbed because of a security flaw is small. Even during the last decades of data leaks, the actual individuals suffering the consequences are a small number. So economically and socially, it's been a winning strategy up to now. Just like pollution, mass consumption, etc.
- tarunkotia 8y ago
  I wish you had used "task" 19143203 instead ;)
- abritinthebay 8y ago
  Nothing you described there is “agile” however. In fact they run counter to every fundamental thing agile stands for. Sounds like your company slammed an agile label on a broken process and thought it would fix it. Agile has nothing to do with those problems, if it wasn’t called that it would a crappy implementation of some other idea. The problem is you have a terrible engineering culture & leadership at your company.
- Bartweiss 8y ago
  Another way I've seen this put: we can already write software that's reliable the way bridges are reliable. The only problem is that people don't want to pay for it, wait for it, or accommodate it. Cleanroom development supposedly lead to production code with 1 bug per 10k lines back in the 80s, and NASA famously managed Shuttle code with 0 bugs in 500,000 checked lines. Other projects like Ariane 5 were not so lucky, but the general error rate is still incredibly low. To my knowledge, software in general has killed fewer spacecraft than other narrow issues like voltage inconsistencies or launch vibrations. And after the Therac-25 incident, medical device software has followed similar a development track. Even automotive software, subjected to far fewer restrictions and more varied environments, has a better critical-failure track record than well-understood components like brakes and tires. Producing software like that is orders of magnitude more expensive than producing 'normal' code. It takes far longer, in ways that more money can't neatly displace. It rarely talks to other systems, because doing so requires exhaustive testing at the interface (for every version), or testing both systems as one (with exponential testing requirements). And it's severely limited in range; freeform inputs and best-guess behavior when adjacent systems fail are out of the question. There are institutional changes that could bring us better software under existing constraints, definitely. But it's also true that the general approach to design is sort of hopeless; the sheer amount of software getting built is inconsistent with what rigorous design looks like in pretty much any field.
  - jrochkind1 8y ago
    > Producing software like that is orders of magnitude more expensive than producing 'normal' code. Yes. Which I think points out that it's not exactly just a question of _will_ or _desire_, that we don't "want" to pay for it, wait for it, accomodate it. It would mean anyone producing software would need _an order of magnitude more money_ to do it. If we're talking about all software... it seems likely that our society literally could not afford it. I suppose we could produce an order of magnitude _less but good software_ instead. But we've built a society which is largely based on software, and been able to afford it as a society only because it's largely based on crappy software.
    - Bartweiss 8y ago
- vkuruthers 8y ago
  100% this. I also find it extremely annoying to have to give daily status updates on highly complex, time consuming problem. It shows a complete lack of understanding of the work being done. The sooner we move om from this Agile garbage the better.
  - silveroriole 8y ago
    Agree. Over many years I realise I actually found waterfall great as a dev. There the worst that could happen was a clueless PM poking in every week and having to be given a vague estimate of how done we were with various tasks. Sure, it probably sucked for that PM, but I was having a good time. Now? Sometimes having to meaninglessly sum up where I am with a piddly ‘agile’ task MULTIPLE times a DAY. It takes more time to update jira/managers/coworkers than to just do the damn thing. Really winds me up.
    - vkuruthers 8y ago
      Yes, I strongly prefer waterfall. Back in the good old days we had a biz. quarter to deliver a set of features. This allowed time for a spec., thinking about how it would all work, doing a design doc, working with QA on how they'd effectively test the features etc. Weekly status updates were adequate and an appropriate "polling frequency". All of that is now thrown out the window in the mad hacking frenzy to make progress on the stupid "points" and try and show progress at the dreaded daily standup meeting (even though it's not supposed to be a progress meeting). Then to make it worse, various team's progress are compared by looking at "points completed" without any kind of normalization of difficulty or risk for the different teams. Then the final insult is the whole debacle is deemed a "success" because we've "switched to Agile", thereby checking off a box that some CTO with too much time on his hands read about in some MIT technology review. I would actively refuse any job offering "Agile development" unless they had a very practical & pragmatic way of implementing it. The only one getting any benefits from this crap are the Agile consultants and the PMs.
precurse 8y ago
I agree with most of the article. However, what I do disagree with is how they lump "security experts" into a single category: Computer experts like to pretend they use a whole different, more awesome class of software that they understand, that is made of shiny mathematical perfection and whose interfaces happen to have been shat out of the business end of a choleric donkey. I'm sorry, but there are vastly different grades of security experts. Security experts make Kali Linux. I'm pretty sure everyone runs their user as root despite it being created by security experts. Now, look at the OpenBSD developers in comparison. Sure, bugs are found as they inevitably are, but they make it very difficult to take advantage of bugs that might be disastrous on other operating systems. They use privilege separation throughout their operating system (and packages if possible), announced recently their way of making ROP-chain exploits basically useless, and relink their kernel any time it's booted so that no two instances are alike (even if it's the same version on another computer). Using defense in depth is key. Unfortunately it's easy to talk yourself up in this field and not walking the walk. There's a reason OpenSSH is such a highly deployed application and yet isn't constantly having RCE bugs. Sure there are bugs (as all software inevitably has), but there are definitely different degrees of security experts that the article fails to mention and lumps them all in one bucket.
LaserToy 8y ago
Postgres was misusing fsync for 20 years. Yes, everything is broken. I’m actually thinking it might be worth snapshotting my bank accounts, to make sure super reliable bank software doesn’t lose my money. At the end of the day it is written by the same folks as me :)
- jb3689 8y ago
  The number of programming fuckups (or bugs - call them what you want) causing me real life stress seems to be increasing exponentially as I'm getting older
  - carapace 8y ago
    That's because the amount of software affecting your real world ("meatspace") life is increasing exponentially without a corresponding decline in bugs-per-line. This is a real problem, we have to adopt exponentially safer code generation methods or reduce the absolute amount of code in use, or both. (Or we can kiss our asses goodbye.)
    - pojzon 8y ago
      I would say the number of bugs increases, because ASA you push stuff out there is never enough time to refactor and fix. Because well.. too many ppl think it "does not bring money"
      - pixl97 8y ago
        Not completely sure about that. Grew up in the early industry and it was rife with complete crap software. Really the only thing somewhat protecting the software at that time was the lack of internet access and threats. Attempting to use that same software now would lead it to be exploited in minutes. I remember messing with Windows 95 with IE 3 installed. It was really easy to create. NetMeeting shortcut that would crash with a buffer overflow error.
  - coldtea 8y ago
zaarn 8y ago
Reminds me of Programming Sucks: https://www.stilldrinking.org/programming-sucks https://www.stilldrinking.org/programming-sucks
blacksqr 8y ago
A classic perennial favorite. Everyone shakes their head knowingly about the problem, but whenever someone here suggests using safer and saner tools, the response is howls of protest. "Buggy code is the developer's fault. Just try harder!" "It's BNF/not BNF!" "It uses/doesn't use types!" "It doesn't let you achieve the last epsilon of speed!" "It doesn't allow/does wrong my preferred form of prototypes/mixins/inheritance!" "Everybody is using something else!" "This is fine."
fallingfrog 8y ago
"Facebook and Google seem very powerful, but they live about a week from total ruin all the time. ... The US government would fall to a general revolt in a matter of days. It wouldn’t take a total defection or a general revolt to change everything, because corporations and governments would rather bend to demands than die. These entities do everything they can get away with — but we’ve forgotten that we’re the ones that are letting them get away with things." Amen, is all I have to say to that

84 points by gpvos 8y ago

Interesting article. The first 75% I would send to my non-technical friends and family as well as the technical ones, as an awareness builder. The last 25% are a surprising call for action (surprising as from the tone of article, I expected it to be a fun rant only). It's tricky because a lot of people tend to discount knowledge imparted once they perceive it was with the objective of persuading you to The Cause. It makes the informational portion more suspect of cherry picking or bias. In other words, while I personally agree both with the technical rant, and the call to action, I feel majority of my friends would react negatively to two of them together, just on general principles/reflexes. Not sure what the solution would be; perhaps two linked articles - first the rant, and then a link to "Proposed solution" article at the bottom? Most people I know would react better if they felt they were in control of asking for solution guidance, rather than part of a seeming bait & switch...

commandlinefan 8y ago

One thing not addressed here is: I can make better software. I can fix these problems. I can find them first, and then fix them. I _can't_ do that when I'm under ridiculous time pressure from an "agile coach" who barely knows what an Excel macro is but who, for some reason, is responsible for deciding what gets prioritized and who gets a bonus and who gets downsized. I can't identify and correct complex technical problems when I'm attending a one-hour daily "scrum" explaining that yes, I'm still working on "task" 5398129 and yes, it was a 4 "point" story (because that's the maximum we're allowed), but as it turns out, almost everything interesting and useful is also complex -- in direct opposition to the mindset that drives the oxymoronic atrocity that people now refer to as "agile".

sametmax 8y ago
That's because security doesn't bring money on the table for most products. People are ok with their private life being violated, they don't mind discreet censorship or mass monitoring. Since we make money with those users, and because the market moves fast, the only security most companies accept to afford is protection against script kiddies and bad PR. But there is a reason the average Joe doesn't care about all those things: it works for him, most of the time. The amount of persons really having their life seriously disturbed because of a security flaw is small. Even during the last decades of data leaks, the actual individuals suffering the consequences are a small number. So economically and socially, it's been a winning strategy up to now. Just like pollution, mass consumption, etc.
tarunkotia 8y ago
I wish you had used "task" 19143203 instead ;)
abritinthebay 8y ago
Nothing you described there is “agile” however. In fact they run counter to every fundamental thing agile stands for. Sounds like your company slammed an agile label on a broken process and thought it would fix it. Agile has nothing to do with those problems, if it wasn’t called that it would a crappy implementation of some other idea. The problem is you have a terrible engineering culture & leadership at your company.
Bartweiss 8y ago
Another way I've seen this put: we can already write software that's reliable the way bridges are reliable. The only problem is that people don't want to pay for it, wait for it, or accommodate it. Cleanroom development supposedly lead to production code with 1 bug per 10k lines back in the 80s, and NASA famously managed Shuttle code with 0 bugs in 500,000 checked lines. Other projects like Ariane 5 were not so lucky, but the general error rate is still incredibly low. To my knowledge, software in general has killed fewer spacecraft than other narrow issues like voltage inconsistencies or launch vibrations. And after the Therac-25 incident, medical device software has followed similar a development track. Even automotive software, subjected to far fewer restrictions and more varied environments, has a better critical-failure track record than well-understood components like brakes and tires. Producing software like that is orders of magnitude more expensive than producing 'normal' code. It takes far longer, in ways that more money can't neatly displace. It rarely talks to other systems, because doing so requires exhaustive testing at the interface (for every version), or testing both systems as one (with exponential testing requirements). And it's severely limited in range; freeform inputs and best-guess behavior when adjacent systems fail are out of the question. There are institutional changes that could bring us better software under existing constraints, definitely. But it's also true that the general approach to design is sort of hopeless; the sheer amount of software getting built is inconsistent with what rigorous design looks like in pretty much any field.
- jrochkind1 8y ago
  > Producing software like that is orders of magnitude more expensive than producing 'normal' code. Yes. Which I think points out that it's not exactly just a question of _will_ or _desire_, that we don't "want" to pay for it, wait for it, accomodate it. It would mean anyone producing software would need _an order of magnitude more money_ to do it. If we're talking about all software... it seems likely that our society literally could not afford it. I suppose we could produce an order of magnitude _less but good software_ instead. But we've built a society which is largely based on software, and been able to afford it as a society only because it's largely based on crappy software.
  - Bartweiss 8y ago
vkuruthers 8y ago
100% this. I also find it extremely annoying to have to give daily status updates on highly complex, time consuming problem. It shows a complete lack of understanding of the work being done. The sooner we move om from this Agile garbage the better.
- silveroriole 8y ago
  Agree. Over many years I realise I actually found waterfall great as a dev. There the worst that could happen was a clueless PM poking in every week and having to be given a vague estimate of how done we were with various tasks. Sure, it probably sucked for that PM, but I was having a good time. Now? Sometimes having to meaninglessly sum up where I am with a piddly ‘agile’ task MULTIPLE times a DAY. It takes more time to update jira/managers/coworkers than to just do the damn thing. Really winds me up.
  - vkuruthers 8y ago
    Yes, I strongly prefer waterfall. Back in the good old days we had a biz. quarter to deliver a set of features. This allowed time for a spec., thinking about how it would all work, doing a design doc, working with QA on how they'd effectively test the features etc. Weekly status updates were adequate and an appropriate "polling frequency". All of that is now thrown out the window in the mad hacking frenzy to make progress on the stupid "points" and try and show progress at the dreaded daily standup meeting (even though it's not supposed to be a progress meeting). Then to make it worse, various team's progress are compared by looking at "points completed" without any kind of normalization of difficulty or risk for the different teams. Then the final insult is the whole debacle is deemed a "success" because we've "switched to Agile", thereby checking off a box that some CTO with too much time on his hands read about in some MIT technology review. I would actively refuse any job offering "Agile development" unless they had a very practical & pragmatic way of implementing it. The only one getting any benefits from this crap are the Agile consultants and the PMs.

precurse 8y ago

I agree with most of the article. However, what I do disagree with is how they lump "security experts" into a single category: Computer experts like to pretend they use a whole different, more awesome class of software that they understand, that is made of shiny mathematical perfection and whose interfaces happen to have been shat out of the business end of a choleric donkey. I'm sorry, but there are vastly different grades of security experts. Security experts make Kali Linux. I'm pretty sure everyone runs their user as root despite it being created by security experts. Now, look at the OpenBSD developers in comparison. Sure, bugs are found as they inevitably are, but they make it very difficult to take advantage of bugs that might be disastrous on other operating systems. They use privilege separation throughout their operating system (and packages if possible), announced recently their way of making ROP-chain exploits basically useless, and relink their kernel any time it's booted so that no two instances are alike (even if it's the same version on another computer). Using defense in depth is key. Unfortunately it's easy to talk yourself up in this field and not walking the walk. There's a reason OpenSSH is such a highly deployed application and yet isn't constantly having RCE bugs. Sure there are bugs (as all software inevitably has), but there are definitely different degrees of security experts that the article fails to mention and lumps them all in one bucket.

LaserToy 8y ago

Postgres was misusing fsync for 20 years. Yes, everything is broken. I’m actually thinking it might be worth snapshotting my bank accounts, to make sure super reliable bank software doesn’t lose my money. At the end of the day it is written by the same folks as me :)

jb3689 8y ago
The number of programming fuckups (or bugs - call them what you want) causing me real life stress seems to be increasing exponentially as I'm getting older
- carapace 8y ago
  That's because the amount of software affecting your real world ("meatspace") life is increasing exponentially without a corresponding decline in bugs-per-line. This is a real problem, we have to adopt exponentially safer code generation methods or reduce the absolute amount of code in use, or both. (Or we can kiss our asses goodbye.)
  - pojzon 8y ago
    I would say the number of bugs increases, because ASA you push stuff out there is never enough time to refactor and fix. Because well.. too many ppl think it "does not bring money"
    - pixl97 8y ago
      Not completely sure about that. Grew up in the early industry and it was rife with complete crap software. Really the only thing somewhat protecting the software at that time was the lack of internet access and threats. Attempting to use that same software now would lead it to be exploited in minutes. I remember messing with Windows 95 with IE 3 installed. It was really easy to create. NetMeeting shortcut that would crash with a buffer overflow error.
- coldtea 8y ago

zaarn 8y ago

Reminds me of Programming Sucks: https://www.stilldrinking.org/programming-sucks https://www.stilldrinking.org/programming-sucks

blacksqr 8y ago

A classic perennial favorite. Everyone shakes their head knowingly about the problem, but whenever someone here suggests using safer and saner tools, the response is howls of protest. "Buggy code is the developer's fault. Just try harder!" "It's BNF/not BNF!" "It uses/doesn't use types!" "It doesn't let you achieve the last epsilon of speed!" "It doesn't allow/does wrong my preferred form of prototypes/mixins/inheritance!" "Everybody is using something else!" "This is fine."

fallingfrog 8y ago

"Facebook and Google seem very powerful, but they live about a week from total ruin all the time. ... The US government would fall to a general revolt in a matter of days. It wouldn’t take a total defection or a general revolt to change everything, because corporations and governments would rather bend to demands than die. These entities do everything they can get away with — but we’ve forgotten that we’re the ones that are letting them get away with things." Amen, is all I have to say to that