5 ms·

My worst bug was a typo in a single line of html that removed 3DS protection from many millions of dollars of credit card payments

by Exuma 3y ago

ownagefool 3y ago
Pretty epic. I was working for a webhosting company, and someone asked me to rush a change just before leaving. Instead of updating 1500 A records, I updated about 50k. Someone senior managed to turn off the cron though, so what I actually lost was the delta of changes between last backup and my SQL. I was in the room for this though: https://www.theregister.com/2008/08/28/flexiscale_outage/ https://www.theregister.com/2008/08/28/flexiscale_outage/
- CTDOCodebases 3y ago
  I love the title to that article "Engineer accidentally deletes cloud". It's like a single individual managed to delete the monolithic cloud where everyone's files are stored.
  - swalsh 3y ago
    At this point there's basically 3 clouds, and then everyone else.
    - epolanski 3y ago
      AWS, Azure and Cloudflare?
      - yossi_peti 3y ago
        And Google Cloud Platform
  - jacquesm 3y ago
    I if any such pathways remain at AWS, Google, Apple and MS that would still allow a thing like that to happen.
  - jiscariot 3y ago
    That is eerily similar to what happened to us in IBM "Cloud", in a previous gig. An engineer was doing "account cleanup" and somehow our account got on the list and all our resources were blown away. The most interesting conversation was convincing the support person, that those deletion audit events were in fact not us, but rather (according to the engineer's Linked-In page) an SRE at IBM.
    - ownagefool 3y ago
      This was ~14 years ago and both MS & AWS had loss of data incidents iirc.
    - CTDOCodebases 3y ago
      Although it probably wasn't funny at the time I can imagine how comical that conversation was. Thinking about it further the term "cloud" is a good metaphor for storing files on someone else's computer because clouds just disappear.
  - ownagefool 3y ago
    Bare in mind, this was a small startup in 2008 that claims to be the 2nd cloud in the world ( read on-demand iaas provider ). Flexiscale at the time was a single region backed by a netapp. Each VM essentially had a thin-provisioned lun ( logical volume ), basically you copy on write the underlying OS image. So when someone accidently deletes vol0, they take out a whopping 6TB of data, that takes a ~20TB to restore because you're rebuilding filesystems from safe mode ( thanks netapp support ). It's fairly monolithic in that sense. I guess I was 23 at the time, but I'd written the v2 API, orchestrator & scheduling later. It was fairly naive, but filled the criteria of a cloud, i.e. elastic, on-demand, metered usage, despite using a SAN.
hliyan 3y ago
I posted this, in case we want to collect these gems: https://news.ycombinator.com/item?id=37160295 https://news.ycombinator.com/item?id=37160295
swalsh 3y ago
My worst bug was changing how a zip code zone was fetched from the cache in a large ecommerce site with tens of thousands users using it all day long. Worked great in DEV :D but when the thundering herd hit it, the entire site came down.
- jacquesm 3y ago
  Startup, shutdown and migration are all periods of significantly elevated risk. Especially for systems that have been in the air for a long time there are all kinds of ways in which things can go pear shaped. Drives that die on shutdown (or the subsequent boot up), raids that fail to rebuild, cascading failures, power supplies that fail, UPS's that fail, generators that don't start (or that run for 30 seconds and then quit because someone made off with the fuel) and so on.
GreenVulpine 3y ago
You could call that a feature making payments easier for customers! All 3DS does is protect the banks by inconveniencing consumers since banks are responsible for fraud.
- jrockway 3y ago
  I believe they pass on the risk to merchants now. If you let fraud through, $30 per incident or whatever. So typically things like 3DS are turned on because that cost got too high, and the banks assure you that it will fix everything.
- feldrim 3y ago
  It's mostly the payment processor. It may or may not be the bank itself.

My worst bug was a typo in a single line of html that removed 3DS protection from many millions of dollars of credit card payments

by Exuma 3y ago

My worst bug was a typo in a single line of html that removed 3DS protection from many millions of dollars of credit card payments

ownagefool 3y ago

Pretty epic. I was working for a webhosting company, and someone asked me to rush a change just before leaving. Instead of updating 1500 A records, I updated about 50k. Someone senior managed to turn off the cron though, so what I actually lost was the delta of changes between last backup and my SQL. I was in the room for this though: https://www.theregister.com/2008/08/28/flexiscale_outage/ https://www.theregister.com/2008/08/28/flexiscale_outage/

CTDOCodebases 3y ago
I love the title to that article "Engineer accidentally deletes cloud". It's like a single individual managed to delete the monolithic cloud where everyone's files are stored.
- swalsh 3y ago
  At this point there's basically 3 clouds, and then everyone else.
  - epolanski 3y ago
    AWS, Azure and Cloudflare?
    - yossi_peti 3y ago
      And Google Cloud Platform
- jacquesm 3y ago
  I if any such pathways remain at AWS, Google, Apple and MS that would still allow a thing like that to happen.
- jiscariot 3y ago
  That is eerily similar to what happened to us in IBM "Cloud", in a previous gig. An engineer was doing "account cleanup" and somehow our account got on the list and all our resources were blown away. The most interesting conversation was convincing the support person, that those deletion audit events were in fact not us, but rather (according to the engineer's Linked-In page) an SRE at IBM.
  - ownagefool 3y ago
    This was ~14 years ago and both MS & AWS had loss of data incidents iirc.
  - CTDOCodebases 3y ago
    Although it probably wasn't funny at the time I can imagine how comical that conversation was. Thinking about it further the term "cloud" is a good metaphor for storing files on someone else's computer because clouds just disappear.
- ownagefool 3y ago
  Bare in mind, this was a small startup in 2008 that claims to be the 2nd cloud in the world ( read on-demand iaas provider ). Flexiscale at the time was a single region backed by a netapp. Each VM essentially had a thin-provisioned lun ( logical volume ), basically you copy on write the underlying OS image. So when someone accidently deletes vol0, they take out a whopping 6TB of data, that takes a ~20TB to restore because you're rebuilding filesystems from safe mode ( thanks netapp support ). It's fairly monolithic in that sense. I guess I was 23 at the time, but I'd written the v2 API, orchestrator & scheduling later. It was fairly naive, but filled the criteria of a cloud, i.e. elastic, on-demand, metered usage, despite using a SAN.

hliyan 3y ago

I posted this, in case we want to collect these gems: https://news.ycombinator.com/item?id=37160295 https://news.ycombinator.com/item?id=37160295

swalsh 3y ago

My worst bug was changing how a zip code zone was fetched from the cache in a large ecommerce site with tens of thousands users using it all day long. Worked great in DEV :D but when the thundering herd hit it, the entire site came down.

jacquesm 3y ago
Startup, shutdown and migration are all periods of significantly elevated risk. Especially for systems that have been in the air for a long time there are all kinds of ways in which things can go pear shaped. Drives that die on shutdown (or the subsequent boot up), raids that fail to rebuild, cascading failures, power supplies that fail, UPS's that fail, generators that don't start (or that run for 30 seconds and then quit because someone made off with the fuel) and so on.

GreenVulpine 3y ago

You could call that a feature making payments easier for customers! All 3DS does is protect the banks by inconveniencing consumers since banks are responsible for fraud.

jrockway 3y ago
I believe they pass on the risk to merchants now. If you let fraud through, $30 per incident or whatever. So typically things like 3DS are turned on because that cost got too high, and the banks assure you that it will fix everything.
feldrim 3y ago
It's mostly the payment processor. It may or may not be the bank itself.