5 ms·

I think the software industry itself has accumulated enough bugs over the past few decades. E.g. F-22 navigation system core dumps when crossing the internatio

by hliyan 3y ago

I think the software industry itself has accumulated enough bugs over the past few decades. E.g. F-22 navigation system core dumps when crossing the international date line: https://medium.com/alfonsofuggetta-it/software-bug-halts-f-22-flight-ed482a4434ce https://medium.com/alfonsofuggetta-it/software-bug-halts-f-2... Loss of Mars probe due to metric-imperial conversion error I've a few of these myself (e.g. a misplaced decimal that made $12mil into $120mil), but sadly cannot devulge details.

Exuma 3y ago
My worst bug was a typo in a single line of html that removed 3DS protection from many millions of dollars of credit card payments
- ownagefool 3y ago
  Pretty epic. I was working for a webhosting company, and someone asked me to rush a change just before leaving. Instead of updating 1500 A records, I updated about 50k. Someone senior managed to turn off the cron though, so what I actually lost was the delta of changes between last backup and my SQL. I was in the room for this though: https://www.theregister.com/2008/08/28/flexiscale_outage/ https://www.theregister.com/2008/08/28/flexiscale_outage/
  - CTDOCodebases 3y ago
    I love the title to that article "Engineer accidentally deletes cloud". It's like a single individual managed to delete the monolithic cloud where everyone's files are stored.
    - swalsh 3y ago
      At this point there's basically 3 clouds, and then everyone else.
      - epolanski 3y ago
        AWS, Azure and Cloudflare?
        yossi_peti 3y ago
        And Google Cloud Platform
    - jacquesm 3y ago
      I if any such pathways remain at AWS, Google, Apple and MS that would still allow a thing like that to happen.
    - jiscariot 3y ago
      That is eerily similar to what happened to us in IBM "Cloud", in a previous gig. An engineer was doing "account cleanup" and somehow our account got on the list and all our resources were blown away. The most interesting conversation was convincing the support person, that those deletion audit events were in fact not us, but rather (according to the engineer's Linked-In page) an SRE at IBM.
      - ownagefool 3y ago
        This was ~14 years ago and both MS & AWS had loss of data incidents iirc.
      - CTDOCodebases 3y ago
        Although it probably wasn't funny at the time I can imagine how comical that conversation was. Thinking about it further the term "cloud" is a good metaphor for storing files on someone else's computer because clouds just disappear.
    - ownagefool 3y ago
      Bare in mind, this was a small startup in 2008 that claims to be the 2nd cloud in the world ( read on-demand iaas provider ). Flexiscale at the time was a single region backed by a netapp. Each VM essentially had a thin-provisioned lun ( logical volume ), basically you copy on write the underlying OS image. So when someone accidently deletes vol0, they take out a whopping 6TB of data, that takes a ~20TB to restore because you're rebuilding filesystems from safe mode ( thanks netapp support ). It's fairly monolithic in that sense. I guess I was 23 at the time, but I'd written the v2 API, orchestrator & scheduling later. It was fairly naive, but filled the criteria of a cloud, i.e. elastic, on-demand, metered usage, despite using a SAN.
- hliyan 3y ago
  I posted this, in case we want to collect these gems: https://news.ycombinator.com/item?id=37160295 https://news.ycombinator.com/item?id=37160295
- swalsh 3y ago
  My worst bug was changing how a zip code zone was fetched from the cache in a large ecommerce site with tens of thousands users using it all day long. Worked great in DEV :D but when the thundering herd hit it, the entire site came down.
  - jacquesm 3y ago
    Startup, shutdown and migration are all periods of significantly elevated risk. Especially for systems that have been in the air for a long time there are all kinds of ways in which things can go pear shaped. Drives that die on shutdown (or the subsequent boot up), raids that fail to rebuild, cascading failures, power supplies that fail, UPS's that fail, generators that don't start (or that run for 30 seconds and then quit because someone made off with the fuel) and so on.
- GreenVulpine 3y ago
  You could call that a feature making payments easier for customers! All 3DS does is protect the banks by inconveniencing consumers since banks are responsible for fraud.
  - jrockway 3y ago
    I believe they pass on the risk to merchants now. If you let fraud through, $30 per incident or whatever. So typically things like 3DS are turned on because that cost got too high, and the banks assure you that it will fix everything.
  - feldrim 3y ago
    It's mostly the payment processor. It may or may not be the bank itself.
gostsamo 3y ago
my funniest was a wrong param in a template generator which turned off escaping parameter values provided indirectly by the users. good that it was discovered during the yearly pen testing analysis because it lead to shell execution in the cloud environment.
kefabean 3y ago
The worst bug I encountered was when physically relocating a multi rack storage array for a mobile provider. The array had never been powered down(!) so we anticipated that a good number of the spindles would fail to come up on restart. So we added an extra mirror to protect each existing raid set. Problem is a bug in the firmware meant the mere existence of this extra mirror caused the entire arrays volume layout to become corrupted at reboot time. Fortunately a field engineer managed to reconstruct the layout, but not before a lot of hair had been whitened.
- jacquesm 3y ago
  Close call. I know of a similar case where a fire suppression systems check ended up with massive data loss in a very large storage array.
zenkat 3y ago
Does anyone have a similar compendium specifically for software engineering disasters? Not of nasty bugs like the F-22 -- those are fun stories, but they don't really illustrate the systemic failures that led to the bug being deployed in the first place. Much more interested in systemic cultural/practice/process factors that led to a disaster.
- two_handfuls 3y ago
  Yes, the RISK mailing list.
- mrguyorama 3y ago
  Find and take a CS ethics class.
RetroTechie 3y ago
In terms of engineering (QC, processes etc), modern day software industry is worse than almost any other industry out there. :-( And no, just plain complexity or fast-moving environment, is a factor but not the issue. It's that steps are skipped which are not skipped in other branches of engineering (eg. continous improvement of processes, learning from mistakes & implementing those. In software land: same mistakes made again & again & again, poorly designed languages remain in use, the list goes on). A long way still to go.
- jacquesm 3y ago
  And reading this thread it doesn't look as if there is much awareness of that.
- fredley 3y ago
  In any other field of engineering, the engineers are all trained and qualified. In software 'engineering', not so much.
  - LeonenTheDK 3y ago
    This is exactly why I support "engineer" being a protected term, like Doctor. It should tell you that a certain level of training and qualification has been met, to the point that the engineer is responsible and accountable for the work they do and sign off on. Especially for things that affect safety. Many software engineers these days are often flying by the seat of their pants, moving quickly and breaking things. Thankfully this seems to largely be in places that aren't going to affect life or limb, but I'm still rubbed the wrong way seeing people (including myself mind you) building run of the mill CRUD apps under the title of engineer. Is it a big deal? Not really. It's probably even technically correct to use the term this way. But I do think it dilutes it a bit. For context, I'm in Canada where engineer is technically a protected term, and there are governing bodies that qualify and designate professional engineers.
    - zvmaz 3y ago
      Are all "engineers" trained on human error, safety principles, and the like? The failures described in the article are precisely not software failures.

by hliyan 3y ago

Exuma 3y ago

My worst bug was a typo in a single line of html that removed 3DS protection from many millions of dollars of credit card payments

ownagefool 3y ago
Pretty epic. I was working for a webhosting company, and someone asked me to rush a change just before leaving. Instead of updating 1500 A records, I updated about 50k. Someone senior managed to turn off the cron though, so what I actually lost was the delta of changes between last backup and my SQL. I was in the room for this though: https://www.theregister.com/2008/08/28/flexiscale_outage/ https://www.theregister.com/2008/08/28/flexiscale_outage/
- CTDOCodebases 3y ago
  I love the title to that article "Engineer accidentally deletes cloud". It's like a single individual managed to delete the monolithic cloud where everyone's files are stored.
  - swalsh 3y ago
    At this point there's basically 3 clouds, and then everyone else.
    - epolanski 3y ago
      AWS, Azure and Cloudflare?
      - yossi_peti 3y ago
        And Google Cloud Platform
  - jacquesm 3y ago
    I if any such pathways remain at AWS, Google, Apple and MS that would still allow a thing like that to happen.
  - jiscariot 3y ago
    That is eerily similar to what happened to us in IBM "Cloud", in a previous gig. An engineer was doing "account cleanup" and somehow our account got on the list and all our resources were blown away. The most interesting conversation was convincing the support person, that those deletion audit events were in fact not us, but rather (according to the engineer's Linked-In page) an SRE at IBM.
    - ownagefool 3y ago
      This was ~14 years ago and both MS & AWS had loss of data incidents iirc.
    - CTDOCodebases 3y ago
      Although it probably wasn't funny at the time I can imagine how comical that conversation was. Thinking about it further the term "cloud" is a good metaphor for storing files on someone else's computer because clouds just disappear.
  - ownagefool 3y ago
    Bare in mind, this was a small startup in 2008 that claims to be the 2nd cloud in the world ( read on-demand iaas provider ). Flexiscale at the time was a single region backed by a netapp. Each VM essentially had a thin-provisioned lun ( logical volume ), basically you copy on write the underlying OS image. So when someone accidently deletes vol0, they take out a whopping 6TB of data, that takes a ~20TB to restore because you're rebuilding filesystems from safe mode ( thanks netapp support ). It's fairly monolithic in that sense. I guess I was 23 at the time, but I'd written the v2 API, orchestrator & scheduling later. It was fairly naive, but filled the criteria of a cloud, i.e. elastic, on-demand, metered usage, despite using a SAN.
hliyan 3y ago
I posted this, in case we want to collect these gems: https://news.ycombinator.com/item?id=37160295 https://news.ycombinator.com/item?id=37160295
swalsh 3y ago
My worst bug was changing how a zip code zone was fetched from the cache in a large ecommerce site with tens of thousands users using it all day long. Worked great in DEV :D but when the thundering herd hit it, the entire site came down.
- jacquesm 3y ago
  Startup, shutdown and migration are all periods of significantly elevated risk. Especially for systems that have been in the air for a long time there are all kinds of ways in which things can go pear shaped. Drives that die on shutdown (or the subsequent boot up), raids that fail to rebuild, cascading failures, power supplies that fail, UPS's that fail, generators that don't start (or that run for 30 seconds and then quit because someone made off with the fuel) and so on.
GreenVulpine 3y ago
You could call that a feature making payments easier for customers! All 3DS does is protect the banks by inconveniencing consumers since banks are responsible for fraud.
- jrockway 3y ago
  I believe they pass on the risk to merchants now. If you let fraud through, $30 per incident or whatever. So typically things like 3DS are turned on because that cost got too high, and the banks assure you that it will fix everything.
- feldrim 3y ago
  It's mostly the payment processor. It may or may not be the bank itself.

gostsamo 3y ago

my funniest was a wrong param in a template generator which turned off escaping parameter values provided indirectly by the users. good that it was discovered during the yearly pen testing analysis because it lead to shell execution in the cloud environment.

kefabean 3y ago

The worst bug I encountered was when physically relocating a multi rack storage array for a mobile provider. The array had never been powered down(!) so we anticipated that a good number of the spindles would fail to come up on restart. So we added an extra mirror to protect each existing raid set. Problem is a bug in the firmware meant the mere existence of this extra mirror caused the entire arrays volume layout to become corrupted at reboot time. Fortunately a field engineer managed to reconstruct the layout, but not before a lot of hair had been whitened.

jacquesm 3y ago
Close call. I know of a similar case where a fire suppression systems check ended up with massive data loss in a very large storage array.

zenkat 3y ago

Does anyone have a similar compendium specifically for software engineering disasters? Not of nasty bugs like the F-22 -- those are fun stories, but they don't really illustrate the systemic failures that led to the bug being deployed in the first place. Much more interested in systemic cultural/practice/process factors that led to a disaster.

two_handfuls 3y ago
Yes, the RISK mailing list.
mrguyorama 3y ago
Find and take a CS ethics class.

RetroTechie 3y ago

In terms of engineering (QC, processes etc), modern day software industry is worse than almost any other industry out there. :-( And no, just plain complexity or fast-moving environment, is a factor but not the issue. It's that steps are skipped which are not skipped in other branches of engineering (eg. continous improvement of processes, learning from mistakes & implementing those. In software land: same mistakes made again & again & again, poorly designed languages remain in use, the list goes on). A long way still to go.

jacquesm 3y ago
And reading this thread it doesn't look as if there is much awareness of that.
fredley 3y ago
In any other field of engineering, the engineers are all trained and qualified. In software 'engineering', not so much.
- LeonenTheDK 3y ago
  This is exactly why I support "engineer" being a protected term, like Doctor. It should tell you that a certain level of training and qualification has been met, to the point that the engineer is responsible and accountable for the work they do and sign off on. Especially for things that affect safety. Many software engineers these days are often flying by the seat of their pants, moving quickly and breaking things. Thankfully this seems to largely be in places that aren't going to affect life or limb, but I'm still rubbed the wrong way seeing people (including myself mind you) building run of the mill CRUD apps under the title of engineer. Is it a big deal? Not really. It's probably even technically correct to use the term this way. But I do think it dilutes it a bit. For context, I'm in Canada where engineer is technically a protected term, and there are governing bodies that qualify and designate professional engineers.
  - zvmaz 3y ago
    Are all "engineers" trained on human error, safety principles, and the like? The failures described in the article are precisely not software failures.