8 ms·
In their defense, the software industry is a complete joke in terms of quality control. I hope that this and the spate of ransomware will wake the industry up t
by coldpie 7y ago
In their defense, the software industry is a complete joke in terms of quality control. I hope that this and the spate of ransomware will wake the industry up to realize we need new processes, languages and tooling to make software provably correct. It's clear we can't use our existing languages and tooling to make high quality software. Realistically though, that's not going to happen. The smart move is to eliminate as much software from your life as possible. It's only going to get worse as decades of laziness catches up to us.
- m_fayer 7y agoWatching Android descend from a place of dorky stability to sleek sealed glitch-city, kinda makes me think it's the institutions, as opposed to the tooling, that's behind the plunge in the quality of mainstream software.
- peteradio 7y agoFeed me features I shit backlogs.
- salawat 7y agoI couldn't help but laugh, because it is so damn true. It's taken me a while to come to the realization that all project management is is a backlog manufacturing and prioritization layer that operates on top of actual software engineers. Most just want to implement things, and don't care what it is they are implementing as long as they are getting paid.
- shadowgovt 7y agoWas Android the stable one? I recall relative to iOS that it was the one likely to be eating its entire battery in the early iterations.
- jsight 7y agoI can't agree. I've been using Android since the Nexus One days, and it is generally much more stable for me than some of those early devices.
- starfallg 7y agoNah. Just introduce 'modern software development methodologies'. Put in a CI/CD pipeline and Blue/Green with Canary. Roll back the release on hull loss. /s
- nerpderp82 7y agoThe snark is not warranted. Those techniques are absolutely fine when you are building mail order catalogs on the internet.
- whynaut 7y agoThe "on hull loss" bit addresses the fact that the discussion is around much more critical software. I didn't read it as snark.
- Waterluvian 7y agoI think the underlying issue is that when it comes to safety, privacy, and security, even if people aren't dying in a plane crash, you still need to do it with a basic level of competence. You cannot roll back a privacy leak. Now if you're talking about features that don't touch these things, go nuts. And there lies a second issue: the identification and distinction between what stuff is "go nuts" and what stuff is "do it right the first time". You can't approach every project with the latter or you'll waste so much time in the design phase when really you should just throw your new CSS theme out there and see how it goes. But you just can't approach the "account creation" feature the same way.
- ikiris 7y agoUntil privacy leaks have more than 0 consequences, why would anyone care to prevent them?
- catalogia 7y agoBasic morality? Do you consider yourself a good person? Do you do the right thing, even when nobody is forcing you to? Do you steal when nobody is looking? And yes, corporations aren't people, corporations don't have morality, I've heard all those excuses. Those corporations are comprised of people and those people should not be held to a lower standard.
- SilasX 7y agoThe "software industry in general", sure, but my understanding is that any flight-critical software goes under a much more rigorous testing regime and uses much better practices the average software project. It's probably why such issues were caught so early here.
- DigitalBison 7y agoThis topic reminds me of this relatively old (but still super interesting) article about the team that worked on the space shuttle's onboard computer systems and the rigorous processes they followed to ensure correctness and safety: https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff.
- Seenso 7y ago> the team that worked on the space shuttle's onboard computer systems and the rigorous processes they followed to ensure correctness and safety Those processes are expensive. It'd imagine that they're a huge political problem to maintain, giving cost-cutting pressures and the temptations of COTS [1]. [1] https://www.faa.gov/aircraft/air_cert/design_approvals/air_software/media/03-77_COTS_RTOS.pdf https://www.faa.gov/aircraft/air_cert/design_approvals/air_s...
- DigitalBison 7y agoOh totally, I can't dispute that. It's just good reading in general, but is more of a response to the grandparent comment stating "the software industry is a complete joke in terms of quality control". I think that's probably not a completely off-base assessment of quality control on average in the tech industry, but the point is it's not like no one knows how to do better; rigorous QA processes exist, but it's a huge tradeoff between cost/agility/etc and the expected cost/damage of not getting it right.
- bradknowles 7y agoThere are SEI Level 5 operations in the world. They do exist. Or at least, we've been told they do. But they are also super expensive. And no one has any money to spend on anything like testing or quality any more.
- pinkfoot 7y agoIn our defence, our customers are a compete joke in terms of the price they are prepared to pay for quality sfotware. See MCAS programmers at $9/h.
- dotancohen 7y agoI just checked and it is true. The subcontractors who wrote the MCAS software were paid $9/h: > Increasingly, the iconic American planemaker and its > subcontractors have relied on temporary workers making > as little as $9 an hour to develop and test software, > often from countries lacking a deep background in > aerospace -- notably India. https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-...
- salawat 7y agoNote those engineers (HCL) only worked on the Multi-Function display code. I don't have the article on hand, but they did ascertain at one point that the MCAS code was written in house. I'll try to find the article again. I think it was Seattle Times. Found it. Was Bloomberg. Here is the old thread. https://news.ycombinator.com/item?id=20309052 https://news.ycombinator.com/item?id=20309052
- slumdev 7y agoIt doesn't mean much that the software was written in-house. Boeing has literally relocated their plant operations cross-country in the past in order to break the workers' unions. Just think about how much pressure they can exert on a non-union profession like software development.
- thawaway1837 7y agoIt’s a lot more comfortable to blame Indian engineers (and of course the HN crowd loves that explanation, no matter how invalid it is) than to place the blame where it belongs. Squarely on Boeing management.
- 7y ago
- anshou- 7y agoThe people, processes, languages, and tools aren't the root problem. The root problem is always time and money.
- rfmw19 7y ago> The root problem is always time and money. I agree with you. However, if you're not some high level manager who controls this, what's the next best thing you can do? I think most problems are less technical and are more about people and processes. You can still argue you don't have enough influence there, and that's completely possible and realistic. But that should be where we direct attention. Yes, some technical advances in better tools and languages that provide stricter proofs and so on are needed and will help. But ultimately it's still the people that need to learn, use, and enforce the processes.
- salawat 7y ago>It's clear we can't use our existing languages and tooling to make high quality software. Gotta fundamentally disagree with you right there. We absolutely can make high-quality software with our current tooling. The issue is that doing that is expensive and time consuming, and the Market optimizes on good enough to be sold and not dropped. This expense and difficulty isn't an inherent fault of the tools, but rather the monstrous other side of the coin in proving what your system isn't. There are many implementations that are composable to generate an end result, the trick is to expend the energy to ensure you've made the specific one that also doesn't run into undefined behavior, domain specific or otherwise. You will never escape from the tyranny of having to clearly communicate to a perfectly obedient machine exactly what it is you want it to do; part of which is being able to identify when you don't have all your requirements right.
- thawaway1837 7y agoAirbus has been doing it for a while. The big difference between Airbus and Boeing has been how much more reliance Airbus places on technologies like MCAS vs straight up pilot skill. Airbus has designed their planes to counteract bugs by adding tremendous amounts of redundancies. The Max is Boeing’s effort at creating an Airbus like plane without the redundantsafety features.
- rayhendricks 7y agoThat is why I refuse to fly in anything that has Boeing on the side of it. Based on the 737 max fiasco and previous incidents they it looks like they are willing to make compromises in safety for profit. The regulatory agencies including the FAA here in the USA are captured and thus irrelevant, and the attempt to fix them died https://en.m.wikipedia.org/wiki/Regulatory_capture#Federal_Aviation_Administration https://en.m.wikipedia.org/wiki/Regulatory_capture#Federal_A... Unless we see fines based on a percentage of revenue or the CEO/Board of Boeing being thrown in prison, then we won’t see change.
- Pmop 7y agoResearch on software reliability is plentiful and throughful, and there are existing tools to use such knowledge in a way to ensure safety of critical systems, from development to deploy. The thing is that they most likely ran their calculations and concluded that it's more profitable to cheap out on software development and QA, and pay for insurance or fines, should anything go wrong later. Engineers who devoted their time and effort to learn aforementioned techniques are very unlikely to work for 9$/h, it's more profitable and less stressful to go wash cars or whatever.
- thawaway1837 7y agoThis defense basically says that software engineering is far more of a joke than hardware engineering. But it’s not a real defense for the Max because the problem with the Max is that Boeing has shifted a lot of the work from hardware to software. This defense only punts the ball a few yards because the question now is why Boeing chose to shift a lot of the reliability from hardware, which this defense admits is better engineered, to software, which it admits is almost intrinsically worse. If anything, this defense leads to the conclusion that the Max is an intrinsically unsafer plane, because it has shifted far more of the burden to software which is an intrinsically worse engineering discipline. If it was just a matter of the software QA being rushed then the additional time gained by the grounding of the planes could solve the problem. But if it is software engineering itself which is the problem then no additional time will help unless it leads to shifting some of the burden of safely flying the plane from software to hardware, which doesn’t even appear to be an option Boeing has considered.
- harikb 7y agoThis seems like a weird way to look at the problem. Hardware also has software/logic. The only reason it appears more stable is that people are shit scared of finding a bug that can't be fixed. If we delay the software release as long as a hardware development takes, with corresponding QA, I don't see why switching it all to hardware is going to make any more sense. Immutability is all it has, unless you are somehow implying that one can be hacked (by bad actors) and the other can't be.
- nomel 7y agoI've worked for a couple of fabless semiconductor startups. When fixing a hardware mistake costs millions for a new mask, there are bugs in all the silicon, even the big guys [1]. [1] https://www.theregister.co.uk/2018/01/26/cloudflare_crashes_intel_broadwell_bug/ https://www.theregister.co.uk/2018/01/26/cloudflare_crashes_...
- bradstewart 7y agoWhile I do agree that hardware engineering generally employs more verification and validation, look at the errata sheets for modern processors. They have a lot of bugs as well--most of which are patched in microcode (software) or handled by operating system kernels (also software).
- robocat 7y ago> In their defense, the software industry is a complete joke in terms of quality control. My dishwasher stopped working today. Last week I had to replace a wheel bearing in my van ($900). I just got a refund for a pond aerator that stopped working after a few weeks. And the head came off my plastic toy. Hardware fails all the time: does that mean I can generically say all manufacturing has shitty QC?
- blt 7y agoI'd say the aerator and toy are analogous to software bugs, but the wheel bearing is a wear item. Even the best-engineered hardware has parts that wear out. That kind of planned maintenance has no equivalent in software.
- reilly3000 7y agoNo new processes needed. ISO 90003 covers all kinds of software engineering practices that ensure quality and correctness. I wish more firms used what is already out there.
- purpleidea 7y agoProvably correct software (and memory and type safe languages!) are both important aspects. But they can still cause you to die in a software bug. The third and most important aspect is that the code must be viewable by any third party or individual. Open source here would be ideal, but at a bare minimum, we should be allowed to see the code that our lives depend on.
- gmanley 7y agoNot all software has the same level of criticalness. The difference between a bug in an airplane vs one in a game or entertainment software is gigantic. Sometimes it really is more important to get the software out quicker vs it being bug free. Plus, formally provable software exists and is used in some mission critical applications including some Airbus software. It's just that it takes a lot longer to develop. These problems are more a business and process issue, as in Boeing prioritizing speed over quality.
- kjs3 7y agowe need new processes, languages and tooling to make software provably correct This is so not true. Processes? Like T-CMM/TSM, FAA-iCMM, ISO 90003, TCSEC, TSP, etc., etc.? Languages? Like Ada, MISRA, Modula-2/IEC1131, Cyclone, etc., etc.? Tools? Like FRAMA-C, MALPAS, SPADE, SPARK, TLA+, etc., etc.? The simple fact of the matter is that we've had all the processes, languages & tools to write high-reliability, secure code for decades. Go check out the CMU SEI for how long that one institution has been trying to get people to do the right thing. The joke is that the software industry by and large wants to pretend we don't, and claim that there's some magical new tech that needs to get invented and when it arrives everyone will jump on it and suddenly software bugs will be a thing of the past. And every new tool, tech or whatever that shows up re-proves some unpleasant basic facts: writing secure, provable, reliable code is time consuming, usually difficult, which translates to relatively expensive. And the software industry doesn't want to hear that, or invest in it, and excuses it's miserable security record with "it's not us, the tech doesn't exist to do the job right". As a sidenote, this is why I have a visceral distaste for the Rust people (the language seems fine). If they'd spent those (presumedly) thousands of man years building on and improving Ada instead of being all NIH averse we'd be much further down the road. They could have participated in the process and gotten their wish list included in the Ada-2012 standard and built on 35+ years of experience. But, hey, it's more fun to start from scratch every decade or so.