6 ms·

I feel like the slow decline of software quality has been in lockstep with the gradual transition from (expensive and non-measurable) manual software/hardware t

by airza 6y ago

I feel like the slow decline of software quality has been in lockstep with the gradual transition from (expensive and non-measurable) manual software/hardware testing and QA to automated frameworks and rollout-based quality assurance. I constantly encounter broken functionality, buggy or unpleasant UIs, just as the author has. It feels like many of these problems could be avoided if you just had one person whose job it was to sit there and look for broken stuff. (I'm sure I'm biased as someone whose first job out of college was to sit there and look for broken stuff.)

deleted 6y ago
[deleted]
ntsplnkv2 6y ago
I agree with this sentiment in general. Of course, we have a ton more features than we used to have, but I think given the newness of software in years past, we were OK with bugs and issues because of the novelty of it all. Now we are in 2020 and iPhone updates STILL cause battery issues. The iPhone has been out for 14 years... Our expectations have changed. Tools like Excel should just work - and yet, when I try to save a file, sometimes it freezes and crashes. How is that acceptable now?
jjoonathan 6y ago
Yeah, but then you'd have to raise the price to pay for the testing and the jokers a block down the street who YOLO'd their competing product into the marketplace without testing would grab all your sales. You'd be out of business and your competitors would be laughing their way to the bank while the customers still suffered constantly from a broken product.
- Jtsummers 6y ago
  If it were only for novel products this would be a reasonable argument. See Netscape's rush to release their browser as an example of what you're talk about. But the worst part is that this is an issue with established products that have secured their market, and will even receive money every year from their customers. They have both the money and the time to pace themselves and test things properly, but they don't.
  - pas 6y ago
    Can you name a few of those who really truly have time but don't take their time? Also it seems there's always an opportunity cost. So time is always "of the essence".
    - jjoonathan 6y ago
      Right, you could be spending those dollars running ads, which the market is more likely to respond to :/
    - Jtsummers 6y ago
      Microsoft, Apple, Google. An MS bug in an enterprise piece of software (so they get paid every year for it) is Skype for Business. Perhaps my org hasn't updated their installation, but if I drag a contact from the chat window (say you message me and aren't in my contacts yet) and then drag it to the contact list (in status view) it will reliably crash the program. If I drag it to groups view it'll place the contact in a group. My guess is that dragging it to status there is no "default" group and so there's some kind of null pointer exception occurring (it tries to add it but with group given as either junk or null). I used to see more bugs in Outlook, but it seems somewhat better with the last update so I haven't noticed them. Though it was fun when I had negative 2 billion messages for a week or so (I certainly didn't have enough to cause overflow so I have no idea how it wrapped around like that). Google's is an issue of usability of their webapps (IME), not strictly buggy but not sufficiently tested. Behind the proxy at work Maps is incredibly unreliable. It takes several reloads for it to actually start working "correctly", but don't change where you're looking too much (you can zoom in, but do not pan around). That's not the only unreliable one in this situation, but it is the most easily demonstrated. Apple's Messages and Mail constantly tell me (they've been better the last few months, but it still happens) that I have unread messages, I'd search and search and never find them. Then I'd pull it up on a different device and finally see the unread message (which was both visible and marked as read on the original device). Some of these may be shallow or seem petty, but it's an unpleasant experience that after so many years and with so much money should've been resolved for each of them. I'm willing to tolerate an indie game crashing on me. I'm not willing to tolerate an enterprise software solution (MS) crashing for a natural user behavior. EDIT: I think resolved, but Apple's iOS calculator bugs were annoying for such a simple program. Not strictly a bug, but Windows' calculator, these days, is an unusable mess in many ways. It shouldn't require so many system resources to add some numbers together (the same could be said of many small utilities that were rewritten for, I think, Windows 10 or Windows 8). Gmail showing me email meant for <first><last>@gmail.com instead of <first>.<last>@gmail.com. The fact that they ignored the . at all in the user names. The Youtube app on iOS when I first installed it and quickly uninstalled it years ago was an incredible annoyance. It wouldn't reliably show me the video I'd actually clicked on that caused the app to open. Adobe is an almost perfect example of holding their market captive. I've run into a number of Adobe Acrobat issues over the years, though fewer recently (but I use it less often now). But especially Acrobat Reader on Mac OS X was awful, I actually once had to reinstall the OS to get it to stop fucking up PDF display even after uninstalling the software (I never found out what it had done to the system, and gave up). I needed it because I couldn't find anything else (at the time) that supported digital signatures in PDFs on the Mac (in the sense that it actually worked, I think Preview let me do it but what it made wasn't usable by the people receiving the file). EDIT2: Another Google one, with Chrome on macOS in full screen. Hiding the location bar means you straight up can't get to it. You have to reenable it, versus a sane behavior like autohiding and moving the cursor to the top restoring it.
colanderman 6y ago
You also need management willing to prioritize engineering time to fixing that broken stuff, and engineers who actually know how to make non-broken stuff. My experience is that having all three of these prerequisites is pretty rare.
- Jtsummers 6y ago
  I'm going to restate something that I said a few days ago: There are a number of hats developers are expected to wear today: 1. Developer of new features 2. Sustainer of prior code and features 3. Tester of all of this 4. Constant student (outside work because who'd pay their employees to learn?) The priority for the business is (1), so 2-4 get neglected. This compounds over time to mean that old code isn't properly refactored or rewritten when it should be, and none of the code is tested as thoroughly as it should be, and none but the smartest or most dedicated are really going to be perpetual students (or they'll choose to study things that interest them but don't help at work, like me). When the old code and poor tests create sufficient problems, you get a business failure or a total rewrite. Which strips out half (or more) of the features and the whole process gets restarted.
  - TeMPOraL 6y ago
    Don't forget: 5. The mentor of younger developers. It's another thing some companies expect you to do, but don't allocate time for it, so at the point you finally know enough to not do a bad job, you're suddenly being pulled out of 1-4 and expected to do 5.
    - Jtsummers 6y ago
      Oof, we had a guy retire over that. In his appraisal he was dinged for not doing enough. They counted his training of new hires as 1 "point" (or whatever, the system was weird). They neglected to consider that he was training 5+ new hires in that year. He'd had enough, put in one last year to wrap some stuff up and then he was out and free.
  - thelazydogsback 6y ago
pixl97 6y ago
Um, how long have you been in the industry? Software has always sucked and had these issues. It has nothing to to with automated QA. The reason you see more issues is 'way back in the day' your software had a very limited number of things that it did, and in general it did not involve accessing a network or chugging down massive volumes of data from untrusted sources. I work for a company that has a lot of individuals that test for QA issues, they have list miles long of things to check and write reports on. The problem is more of "It's much easier to write mountains of code than it is to ensure that it works in all cases"
- bredren 6y ago
  Software has never been better; there are just more users to please. And despite the issues lamented in this think piece, it is _Apple_ that the author should blame for setting technology expectations impossibly high. No organization had been remotely as successful in understanding and releasing tech products that were truly great. Everything since is just a comparison to expectations Apple set. Even when Apple fails, it is in comparison with an Apple that does not.
- tonyedgecombe 6y ago
  The one thing that has changed is updates can be delivered easily. This takes some of the pressure off in terms of QA because rolling out a fix to a centralised service delivered through the browser is quick and painless. The cost of pressing millions of CD's kept developers in check in the past.
  - cedilla 6y ago
    That's really not the only thing that changed. Games are also massively more complicated today. It's one thing for three people getting a 2.5 MB single player DOS game reasonably bug free. It's an entirely different thing to get the 5 GB game made by a team of 100 or 1,000 people.
    - rightbyte 6y ago
      Most of the size difference is assets anyway, right? Games today use off the shelf engines. It should be alot easier to make rubust games when most of the techincally hard parts are allready done.
wwright 6y ago
I work somewhere that primarily relies on manual testing rather than automated testing. It definitely does not make the software more reliable here, at the least. :-)
- Jtsummers 6y ago
  There're three kinds of manual testing: 1. Manual testing that should be manual (exploratory). 2. Manual tests that are new and haven't been automated yet (but will be). 3. Manual tests that should be automated. (3) is the one many people see and suffer through (I know I have). They need to be automated to free up time for (1), which is where many issues are actually discovered. But if (3) dominates your time, you can never get to (1) and you'll constantly ship broken things (or more broken than they should be).
  - wwright 6y ago
    This is very insightful, thanks!
brundolf 6y ago
I would tell a slightly different version of this story, focusing in on "rollout-based quality assurance". I would say that effortless, automatic updates are to blame. When you can always just push an update, the impact of a given bug goes way down. It's no longer mission-critical to exterminate flaws before shipping; a totally broken feature becomes a mere annoyance. So project prioritization shifts from polishing an artifact to outweighing the (presumed inevitable) constant stream of little annoyances with fixes and features. I think the shift towards automated testing is just a symptom; an attempt to bridge the gap in this brave new world. For a clear-cut example of this phenomenon, look to the video game industry. Until around 2007, games received no updates. Ever. Once a game shipped, it was shipped. There wasn't even a mechanism for installing an update from physical media. Right around that time, "glitches" went from very rare unicorns that people would spend lots of time actually seeking out, to nearly everyday occurrences. As long as it doesn't corrupt someone's save file, they mostly laugh it off and upload a clip to YouTube to show their friends. This is just how things are now. (Edit: I should have scoped this to "console games")
- derefr 6y ago
  > Until around 2007, games received no updates. Ever. Once a game shipped, it was shipped. There wasn't even a mechanism for installing an update from physical media. Sure, but they still (sometimes) released (a few) extra revisions of a game. They were just targeted at people who bought physical copies after the revision date, rather than at existing customers. Or said updates came on the 1.0 version of the game as shipped in markets that got the game later than others. (Just imagine — per-market release versioning. Every market effectively got its own fork of the codebase!) Or said updates came in the form of a re-release port. There are patches made to the emulated downloadable app-store re-releases of some games, that never made it into any physical edition of the game. Also, before home consoles, arcade game machines did receive bug-fix updates regularly. Arcade machines were essentially provided “as a service” from their manufacturers, with support contracts et al. Sort of like vending machines are today. If you reported a bug, they’d fix it and send you a new EEPROM chip to swap out in your cabinet. If there was a critical bug that affected all units, they’d send techs out to everybody’s machines to swap out the ROM for the newest revision. (For this reason, it’s actually kind of hard to do art-conservation / archiving of arcade games. The cabinets almost never have fully “original-release” components inside.)
rswail 6y ago
As someone who has been programming since 1984, software quality has greatly improved not declined. The level of complexity in modern day software is orders of magnitude greater than that of even a decade ago. What has changed is our reliance on that software. We are now so deeply embedded into our software existence we see these flaws up close.
- Nagyman 6y ago
  Great comment. People also used to use only a few pieces of software in any given day (or week, even!) e.g. email, web browser, and word processor. Now, our computer ("phone") is with us everywhere we go and we'll use dozens of complex applications per day, connected by dozens of APIs, networks, protocols, and hardware features. It's a miracle any of it works sometimes! Thank you to everyone for making this stuff seem like magic; my twelve-year-old self would be amazed at how well it works. I do feel like there are more UI bugs as we optimize for certain metrics over others. Timing updates has become far more complicated, so we get weird UI refreshes as new data comes in, stale caches, missed notifications, etc. Turning it off and on again often works, surprisingly (probably because devs start with clean environments often, so that's the functional baseline). Lastly, it is probably far more lucrative for a technology based business to use their most valuable minds for the Next Thing, rather than iterating on the current thing. Incremental revenue improvements just don't cut it in a capital-driven world; everyone is trying to escape the local maxima to find billion/trillion dollar businesses.
bsanr2 6y ago
This, but also remember that we're living through a transition from SAAP to SAAS, where the "service" is actually data extraction for advertisers. For that, software must be minimally useful to a user, but the true aim of UX is not user-satisfaction but data extraction or subscription lock-in.

by airza 6y ago

deleted 6y ago

[deleted]

ntsplnkv2 6y ago

I agree with this sentiment in general. Of course, we have a ton more features than we used to have, but I think given the newness of software in years past, we were OK with bugs and issues because of the novelty of it all. Now we are in 2020 and iPhone updates STILL cause battery issues. The iPhone has been out for 14 years... Our expectations have changed. Tools like Excel should just work - and yet, when I try to save a file, sometimes it freezes and crashes. How is that acceptable now?

jjoonathan 6y ago

Yeah, but then you'd have to raise the price to pay for the testing and the jokers a block down the street who YOLO'd their competing product into the marketplace without testing would grab all your sales. You'd be out of business and your competitors would be laughing their way to the bank while the customers still suffered constantly from a broken product.

Jtsummers 6y ago
If it were only for novel products this would be a reasonable argument. See Netscape's rush to release their browser as an example of what you're talk about. But the worst part is that this is an issue with established products that have secured their market, and will even receive money every year from their customers. They have both the money and the time to pace themselves and test things properly, but they don't.
- pas 6y ago
  Can you name a few of those who really truly have time but don't take their time? Also it seems there's always an opportunity cost. So time is always "of the essence".
  - jjoonathan 6y ago
    Right, you could be spending those dollars running ads, which the market is more likely to respond to :/
  - Jtsummers 6y ago
    Microsoft, Apple, Google. An MS bug in an enterprise piece of software (so they get paid every year for it) is Skype for Business. Perhaps my org hasn't updated their installation, but if I drag a contact from the chat window (say you message me and aren't in my contacts yet) and then drag it to the contact list (in status view) it will reliably crash the program. If I drag it to groups view it'll place the contact in a group. My guess is that dragging it to status there is no "default" group and so there's some kind of null pointer exception occurring (it tries to add it but with group given as either junk or null). I used to see more bugs in Outlook, but it seems somewhat better with the last update so I haven't noticed them. Though it was fun when I had negative 2 billion messages for a week or so (I certainly didn't have enough to cause overflow so I have no idea how it wrapped around like that). Google's is an issue of usability of their webapps (IME), not strictly buggy but not sufficiently tested. Behind the proxy at work Maps is incredibly unreliable. It takes several reloads for it to actually start working "correctly", but don't change where you're looking too much (you can zoom in, but do not pan around). That's not the only unreliable one in this situation, but it is the most easily demonstrated. Apple's Messages and Mail constantly tell me (they've been better the last few months, but it still happens) that I have unread messages, I'd search and search and never find them. Then I'd pull it up on a different device and finally see the unread message (which was both visible and marked as read on the original device). Some of these may be shallow or seem petty, but it's an unpleasant experience that after so many years and with so much money should've been resolved for each of them. I'm willing to tolerate an indie game crashing on me. I'm not willing to tolerate an enterprise software solution (MS) crashing for a natural user behavior. EDIT: I think resolved, but Apple's iOS calculator bugs were annoying for such a simple program. Not strictly a bug, but Windows' calculator, these days, is an unusable mess in many ways. It shouldn't require so many system resources to add some numbers together (the same could be said of many small utilities that were rewritten for, I think, Windows 10 or Windows 8). Gmail showing me email meant for <first><last>@gmail.com instead of <first>.<last>@gmail.com. The fact that they ignored the . at all in the user names. The Youtube app on iOS when I first installed it and quickly uninstalled it years ago was an incredible annoyance. It wouldn't reliably show me the video I'd actually clicked on that caused the app to open. Adobe is an almost perfect example of holding their market captive. I've run into a number of Adobe Acrobat issues over the years, though fewer recently (but I use it less often now). But especially Acrobat Reader on Mac OS X was awful, I actually once had to reinstall the OS to get it to stop fucking up PDF display even after uninstalling the software (I never found out what it had done to the system, and gave up). I needed it because I couldn't find anything else (at the time) that supported digital signatures in PDFs on the Mac (in the sense that it actually worked, I think Preview let me do it but what it made wasn't usable by the people receiving the file). EDIT2: Another Google one, with Chrome on macOS in full screen. Hiding the location bar means you straight up can't get to it. You have to reenable it, versus a sane behavior like autohiding and moving the cursor to the top restoring it.

colanderman 6y ago

You also need management willing to prioritize engineering time to fixing that broken stuff, and engineers who actually know how to make non-broken stuff. My experience is that having all three of these prerequisites is pretty rare.

Jtsummers 6y ago
I'm going to restate something that I said a few days ago: There are a number of hats developers are expected to wear today: 1. Developer of new features 2. Sustainer of prior code and features 3. Tester of all of this 4. Constant student (outside work because who'd pay their employees to learn?) The priority for the business is (1), so 2-4 get neglected. This compounds over time to mean that old code isn't properly refactored or rewritten when it should be, and none of the code is tested as thoroughly as it should be, and none but the smartest or most dedicated are really going to be perpetual students (or they'll choose to study things that interest them but don't help at work, like me). When the old code and poor tests create sufficient problems, you get a business failure or a total rewrite. Which strips out half (or more) of the features and the whole process gets restarted.
- TeMPOraL 6y ago
  Don't forget: 5. The mentor of younger developers. It's another thing some companies expect you to do, but don't allocate time for it, so at the point you finally know enough to not do a bad job, you're suddenly being pulled out of 1-4 and expected to do 5.
  - Jtsummers 6y ago
    Oof, we had a guy retire over that. In his appraisal he was dinged for not doing enough. They counted his training of new hires as 1 "point" (or whatever, the system was weird). They neglected to consider that he was training 5+ new hires in that year. He'd had enough, put in one last year to wrap some stuff up and then he was out and free.
- thelazydogsback 6y ago

pixl97 6y ago

Um, how long have you been in the industry? Software has always sucked and had these issues. It has nothing to to with automated QA. The reason you see more issues is 'way back in the day' your software had a very limited number of things that it did, and in general it did not involve accessing a network or chugging down massive volumes of data from untrusted sources. I work for a company that has a lot of individuals that test for QA issues, they have list miles long of things to check and write reports on. The problem is more of "It's much easier to write mountains of code than it is to ensure that it works in all cases"

bredren 6y ago
Software has never been better; there are just more users to please. And despite the issues lamented in this think piece, it is _Apple_ that the author should blame for setting technology expectations impossibly high. No organization had been remotely as successful in understanding and releasing tech products that were truly great. Everything since is just a comparison to expectations Apple set. Even when Apple fails, it is in comparison with an Apple that does not.
tonyedgecombe 6y ago
The one thing that has changed is updates can be delivered easily. This takes some of the pressure off in terms of QA because rolling out a fix to a centralised service delivered through the browser is quick and painless. The cost of pressing millions of CD's kept developers in check in the past.
- cedilla 6y ago
  That's really not the only thing that changed. Games are also massively more complicated today. It's one thing for three people getting a 2.5 MB single player DOS game reasonably bug free. It's an entirely different thing to get the 5 GB game made by a team of 100 or 1,000 people.
  - rightbyte 6y ago
    Most of the size difference is assets anyway, right? Games today use off the shelf engines. It should be alot easier to make rubust games when most of the techincally hard parts are allready done.

wwright 6y ago

I work somewhere that primarily relies on manual testing rather than automated testing. It definitely does not make the software more reliable here, at the least. :-)

Jtsummers 6y ago
There're three kinds of manual testing: 1. Manual testing that should be manual (exploratory). 2. Manual tests that are new and haven't been automated yet (but will be). 3. Manual tests that should be automated. (3) is the one many people see and suffer through (I know I have). They need to be automated to free up time for (1), which is where many issues are actually discovered. But if (3) dominates your time, you can never get to (1) and you'll constantly ship broken things (or more broken than they should be).
- wwright 6y ago
  This is very insightful, thanks!

brundolf 6y ago

I would tell a slightly different version of this story, focusing in on "rollout-based quality assurance". I would say that effortless, automatic updates are to blame. When you can always just push an update, the impact of a given bug goes way down. It's no longer mission-critical to exterminate flaws before shipping; a totally broken feature becomes a mere annoyance. So project prioritization shifts from polishing an artifact to outweighing the (presumed inevitable) constant stream of little annoyances with fixes and features. I think the shift towards automated testing is just a symptom; an attempt to bridge the gap in this brave new world. For a clear-cut example of this phenomenon, look to the video game industry. Until around 2007, games received no updates. Ever. Once a game shipped, it was shipped. There wasn't even a mechanism for installing an update from physical media. Right around that time, "glitches" went from very rare unicorns that people would spend lots of time actually seeking out, to nearly everyday occurrences. As long as it doesn't corrupt someone's save file, they mostly laugh it off and upload a clip to YouTube to show their friends. This is just how things are now. (Edit: I should have scoped this to "console games")

derefr 6y ago
> Until around 2007, games received no updates. Ever. Once a game shipped, it was shipped. There wasn't even a mechanism for installing an update from physical media. Sure, but they still (sometimes) released (a few) extra revisions of a game. They were just targeted at people who bought physical copies after the revision date, rather than at existing customers. Or said updates came on the 1.0 version of the game as shipped in markets that got the game later than others. (Just imagine — per-market release versioning. Every market effectively got its own fork of the codebase!) Or said updates came in the form of a re-release port. There are patches made to the emulated downloadable app-store re-releases of some games, that never made it into any physical edition of the game. Also, before home consoles, arcade game machines did receive bug-fix updates regularly. Arcade machines were essentially provided “as a service” from their manufacturers, with support contracts et al. Sort of like vending machines are today. If you reported a bug, they’d fix it and send you a new EEPROM chip to swap out in your cabinet. If there was a critical bug that affected all units, they’d send techs out to everybody’s machines to swap out the ROM for the newest revision. (For this reason, it’s actually kind of hard to do art-conservation / archiving of arcade games. The cabinets almost never have fully “original-release” components inside.)

rswail 6y ago

As someone who has been programming since 1984, software quality has greatly improved not declined. The level of complexity in modern day software is orders of magnitude greater than that of even a decade ago. What has changed is our reliance on that software. We are now so deeply embedded into our software existence we see these flaws up close.

Nagyman 6y ago
Great comment. People also used to use only a few pieces of software in any given day (or week, even!) e.g. email, web browser, and word processor. Now, our computer ("phone") is with us everywhere we go and we'll use dozens of complex applications per day, connected by dozens of APIs, networks, protocols, and hardware features. It's a miracle any of it works sometimes! Thank you to everyone for making this stuff seem like magic; my twelve-year-old self would be amazed at how well it works. I do feel like there are more UI bugs as we optimize for certain metrics over others. Timing updates has become far more complicated, so we get weird UI refreshes as new data comes in, stale caches, missed notifications, etc. Turning it off and on again often works, surprisingly (probably because devs start with clean environments often, so that's the functional baseline). Lastly, it is probably far more lucrative for a technology based business to use their most valuable minds for the Next Thing, rather than iterating on the current thing. Incremental revenue improvements just don't cut it in a capital-driven world; everyone is trying to escape the local maxima to find billion/trillion dollar businesses.

bsanr2 6y ago

This, but also remember that we're living through a transition from SAAP to SAAS, where the "service" is actually data extraction for advertisers. For that, software must be minimally useful to a user, but the true aim of UX is not user-satisfaction but data extraction or subscription lock-in.