6 ms·

Lost your luggage? That's nothing – we just lost your whole flight

58 points by lawgimenez 3y ago

jacquesm 3y ago
rm -rf vv\ *
datadrivenangel 3y ago
Accidentally deleting prod is a rite of passage.
- davidthewatson 3y ago
  Been there, done that. The paradox is that every device I use interrupts me non-stop with modal dialogs ad nauseam. But the production delete? You said,"drop", we said, "gone". Either production database deletes need to adopt alert fatigue or the ops are already so fatigued that it doesn't matter.
  - ilyt 3y ago
    I'm pretty annoyed by those prompts. Yes, fuck you, do what I fucking said, my backups work fine. Also frankly I think most software does it wrong, if it is possible it shouldn't ask before delete but just allow to undo the step easily. Then work can go fast and uninterrupted while still being resilient to mistakes. Obviously not possible everywhere but where it is it should be the default.
    - Xylakant 3y ago
      Cool that you have a working backup, but restores are what counts. And they’re not instantaneous in almost all cases. So asking before certain destructive actions is still a good practice.
    - darkwater 3y ago
      > Also frankly I think most software does it wrong, if it is possible it shouldn't ask before delete but just allow to undo the step easily. Obviously. When undoing is not easy or possible automatically, there is the "Are you really really sure?" dialog.
  - Aerroon 3y ago
    >The paradox is that every device I use interrupts me non-stop with modal dialogs ad nauseam. Pop-up warnings that are too common are a problem itself. People get so used to accepting them immediately that they stop serving their purpose and just become an annoyance. With one exception: they can prevent misclicks.
    - davidthewatson 3y ago
      Right! That's why I used the term alert fatigue: https://en.wikipedia.org/wiki/Alarm_fatigue https://en.wikipedia.org/wiki/Alarm_fatigue The misclick point is beyond truth, particularly given the storm of marketing that is mobile web.
  - taneliv 3y ago
    In a company where we chose to do production SQL shenanigans in the spirit of "move fast, break things", I developed a habit of starting those with "BEGIN;" and paying careful attention to the number of records modified before calling it a "COMMIT;". In case of unexpected numbers, immediate "ROLLBACK;" so as to not hog the resources and make the applications suffer. Even fatigued, "DELETE 1182900" tends to stick out, if you're used to, and expecting to, see "DELETE 1". ("BEGIN" appears to be PostgreSQLism, "START TRANSACTION" is more standard.)
- moralestapia 3y ago
  Definitely! There's probably some mystical mechanism making sure newbies go through it. I consider myself an "experienced" developer, yet on the first day on my current gig I dropped a '--recursive' flag in an AWS command and deleted a bunch of stuff I shouldn't have. ¯\_(ツ)_/¯
  - _boffin_ 3y ago
    I’m sure some processes and procedures were updated on that day and the next.
    - Zobat 3y ago
      At some time we learned about delete locks in Azure and promptly created those for all production resources. Have had/heard of enough actual/almost "killed production" stories to understand their value. A bit cumbersome when you actually need to delete something but so worth it for the mental health of the team.
      - bombcar 3y ago
        The key with delete locks and similar is it HAS to be something out-of-band; if the normal way to delete is via a website, the delete lock removal should be API or command line, etc A simple --max-really-delete added to a command line is subject to "replay attacks" on yourself when you use CTRL+R.
- pjmlp 3y ago
  Yeah, raises hand. Even though I take all care to differenciate environments, always cross check, we had to live drop a table while on a meeting, and due to ongoing conversation the wrong window was selected, and multitasking ended up causing a major distraction.
- Moru 3y ago
  I sort of did that at my first web-job. I thought it was a bit unhandy that the customers had to be mailed a link to the admin interface, why not just put a link to the login on the front page? A couple of days later I learned why. There's always a good reason somewhere, you just don't know why yet. The reason? Well, the admin interface was written in ASP. According to the guy that called me in panic said that ASP version was so very old it was extremely insecure. I didn't have access to the database so could not verify anything but the webserver logs had a lot of "Bobby tables" in them so I have a strong suspicion it wasn't all ASP's fault... Anyway, we ended up with the database disappearing in the night or containing random data until I got that phonecall asking why we were linking directly to the admin interface :-) They were very good at restoring the database though, just took an hour or so every morning until they found the reason.
- briffle 3y ago
  At a previous position a co-worker reached his rite of passage. Lots of connections, and not paying attention to which screen window/session he was in. After that, we then changed the bash prompts on all prod servers to be red. Never had another incident..
- huehehue 3y ago
  I dropped ~100 records from a join table thinking I was connected to the dev db. "No worries, we have backups", I thought. Turns out those were more like replication, copying over every change immediately (including deletes). In the end, fixed with a short script that cross-referenced tables and checked for missing relationships, but terrifying nonetheless.
- nso 3y ago
  Worked for a CRM SaaS. I once did a DELETE FROM users; WHERE... Open office. Boss rolled over on his chair and started saying "so..." and I dismissed him with a "deleted all users leave me alone". He rolled away, kinda like Homer in that GIF where he disappears into the hedge. Luckily it was back up in minutes, but a new phobia was unlocked.
  - thedanbob 3y ago
    > and I dismissed him with a "deleted all users leave me alone" I think I'm going to start unplugging my desk phone when one of those moments happen to me. "Hey, there's something wrong with..." "YES I KNOW I WAS FIXING IT BEFORE YOU INTERRUPTED ME"
notjustanymike 3y ago
♫ Tale as old as time, True as it can be ♫
jan_Sate 3y ago
That's far worse than the gitlab incident.
secretsatan 3y ago
El Regs' "Who Me?" stories should be taken with a grain of salt
- fnord77 3y ago
  "Dear Penthouse" of IT
  - rswail 3y ago
    Now that's a phrase I haven't heard for a while. I wonder what the 2023 equivalent is.
  - FireBeyond 3y ago
    "... I never thought I'd be writing this letter to you."
- 123pie123 3y ago
  the one I had published about a support call - saying their PC was not working, whilst fault finding I found out they was in the middle of a powercut - they also hurt themselves because it was dark/ no lights. was 100% true. There was loads of comments saying BS I never bothered writing about the person putting tipex on the screen to delete something!! (true) or the time another person swore blind that they've not touched the keyboard - dispite all the keys being in the wrong order (they took the keys off to clean because they spilt coffee on it and put them back in the wrong order) or the time I nearly wiped out 4 man years of code.....
  - ilyt 3y ago
    > or the time another person swore blind that they've not touched the keyboard - dispite all the keys being in the wrong order (they took the keys off to clean because they spilt coffee on it and put them back in the wrong order) That reminds me when one user pissed one of our then-senior admins so they reordered his keyboard keys alphabetically. Or the fun we had where in old system user needed to come to admins to type their first password (that then they are supposed to change after) on account creation and one of admins had blank keycaps keyboard. Reaction was anywhere between "didn't even notice" to "bewilderment and panic"
hmry 3y ago
Happy ending, at least.
jnsie 3y ago
> the manager told him that he understood it wasn't his fault. He blamed the Senior DBA for poor supervision Would have been good management...had they not decided to replace said senior with the new hire junior DBA.
- ysavir 3y ago
  Agreed. Imagine the junior did this after the senior left and didn't have them to fall back on to restore things.
generalizations 3y ago
Poor senior DBA - another month, and he would've gotten away with forgetting about those backups.
hiyjnix 3y ago
Deleting prod is a rite of passage-- the junior DBA because senior that day.
vsareto 3y ago
One DBA was responsible for all backups for an airline? Why not make it a team effort?
- rawgabbit 3y ago
  Many years ago, I worked for a fortune 500 company that did billions in revenue. I was literally one of two DBAs for the entire company. Yes, we did have a revolving door of contractors and a team of "DBAs" in India. Did I ever let them touch production? No. I tasked them with monitoring and dealing with all the non prod environments. You have to realize in a big corp there are an infinite number of ways for things to fall apart. Much of it beyond your control. My goal was to minimize the blast radius. One time I took vacation for the first time in years. When I came back I was blamed for an outage while I was on holiday. Their reasoning was that they didn’t push any new code into production so it must be the database’s fault. After a day of troubleshooting, I learned the application code was not using a framework like Microsoft Linq to read the SQL output but was reading it raw. I also learned that the Windows admin had been pushing Windows updates into the servers. The issue was that a Windows update (not a SQL Server update) changed how timestamps were recorded. It was beyond stupid. Instead of yyyymmdd hhmmss like the application expected. It was now yyyymmdd hhmmss timezoneoffset +- hh. They now had some weird hour offset and the actual values changed. To fix the issue the windows update was rolled back. And the Windows admins were told never to touch the database servers without my permission. And the application was rewritten.
ilyt 3y ago
I got four eyes system to be implemented on our main loadbalances thanks to that roughly a decade - got hired and as junior managed to make change in CM that fucked them up... so seniors decided it's time to make it a bit more resilient
Zobat 3y ago
I've marked a few tens of thousands of records as not delivered to a customer, instead of as intended just one record. We had no development or test environment so development were done in production. The system lacked the necessary functions to correct errors so there were frequent updates/corrections made by writing SQL-queries directly towards the production database. The incident was when I should update a single record so the system would resend it to the customer, but ran the query before writing the WHERE-clause. Luckily there were working backups (incremental backups that ran every 15 min, almost like someone knew they might need them!) and no one had made any changes since the last one. We did a quick restore and let out a sigh of relief. We did eventually write a tool to do those updates (and deletes) we were doing multiple times a day and were able to hand that tool over to one of the users, saving us multiple five minute interruptions each day and boosting our productivity. Worth noting that this was part of a business that we bought from another company and we did transfer all customers to our own product with dev environments and no need to manually updating databases.
- sowbug 3y ago
  A sysadmin at one of my startups once mixed up c and x in a tar backup. Our source repository suddenly teleported back in time. We polled ourselves to see who'd updated most recently, and that guy's snapshot became commit #1 in the new repo. (Unfortunately this was the age of cvs, so the local copies were just source, not the history.)
slowhand09 3y ago
Lucky to have never done this... yet. But I did disable a major (and I do mean MAJOR) system for 20 minutes by misconfiguring an IP tables entry in Prod.
michaelmior 3y ago
This was over a decade ago, but out of paranoia, I found a way to use ROXTerm to colour code my terminals. That way whenever I connected to a production instance, things lit up red so I couldn't be confused about where I was. I don't do this anymore, but it was a fun little hack. https://michael.mior.ca/blog/coloured-ssh-terminals/ https://michael.mior.ca/blog/coloured-ssh-terminals/
- s_dev 3y ago
  Thanks for this -- I colour all my terminals manually with option+command+i and then select a dark colour where each shell has a colour coded purpose usually to indicate which machine it is. This link will help a lot!
- kotaKat 3y ago
  Ooh, my former hospital's EMR did something similar to this! In MEDITECH 6.x, your LIVE system instance was blue. TEST environment was green, and any management interface or screen on a server console that was RED was "this will fuck up LIVE" including giant ** WARNING ** LIVE ** WARNING ** LIVE ** MEDITECH warnings in the title bar for the window..
lelima 3y ago
If it ended bad, with no backups, Is there any legal implacations for the Senior DBA?
netsharc 3y ago
> Right on cue the phones started ringing. In every terminal in every airport around the world, the airline was suddenly unable to create, check or cancel tickets. That doesn't sound like they deleted a flight, how often do you hit a flight that's currently interesting to airports around the world? Of course there are exceptions, e.g. Emirates would route you via Dubai, so a flight Dubai to Frankfurt could have connecting passengers from a few dozen different originating countries, but not "every airport around the world". OK, come to think of it, maybe the inconsistency in the DB halted everything, e.g. the system saw a few hundred passengers booked on flight with ID 463523 but there's no record of a flight with that ID...
drra 3y ago
I doubt the sorry is true. It could only happen with a custom in-house system and any interconnected (and distributed worldwide several decades ago) airlines would use an external distribution system for creating and managing tickets.
squeaky-clean 3y ago
It's not deleting prod, but this reminds me of a funny story when we kind of prevented a low-cost airline from being able to sell any tickets for 8~ hours. We ran the booking websites for lots of airlines (Which was really just a frontend that connected to Sabre or Amadeus). Airline data standards were bad. They're probably still bad. You were lucky if you could get an airline's upcoming schedule or pricing rules in some format other than "manually typed into an email by a human". We had people who's job was solely just to take these emails and put the info in the database. We had a bunch of arbitrary safe-guards against data entry mistakes. Ticket prices above 10k or under 10 were flagged as suspicious and needed someone more senior to approve them. So this airline, headquartered somewhere with a 7 hour time difference from us, wants to do a 9 euro flight special for some holiday. There's a big marketing campaign, apparently lots of excited customers, etc. I don't know if you've connected the dots yet but 9 euros was less than our suspicious-data-threshold of 10. So about 95% of their available flights just disappeared. Searching for them would turn up a "no results" page. The only available flights were expensive ones excluded from the sale, usually codeshares with a larger airline for a more distant destination. The airline never notified us. It wasn't until we waltzed into the office 7 hours later that any one of us saw the problem. One of the DBAs wrote a quick "approve all suspicious entries" query and that was the end of it. Thankfully it was a 3 day sale. Afterwards, I remember the chosen method to prevent this from happening again was to change the low threshold for a suspicious price to 8.99 or below.
technothrasher 3y ago
I had my whole flight lost on me one day. I had just flown in to Philly and my connecting flight to Corning was at gate A7 (or whatever the terminal said). I proceeded to that gate and found the gate completely empty and nothing on the monitor. I waited until it was about ten minutes before boarding time and still crickets. I walked to another gate and asked the agent behind the counter about the flight. A bunch of tapping and they said there wasn't any record of that flight and that they couldn't help me. Uh, ok... I next went to the customer service desk, and the agent there did a bunch more tapping and finally said, "oh, here, the gate is H1". Where the heck is H1? "It's that brown building on the other side of the runway, you can see through the window". The hanger?? "Uh... yes." Huh? "<shrug>". I finally went out through security and into the ticketing line and said, "I don't know what is going on, but can I get a seat on the next flight going to Corning?" The agent had a heck of a time trying to cancel my ticket, as it kept telling them there was no ticket even though they could still bring it up on screen. Finally a supervisor figured it out. Somebody had cancelled the flight, moved everybody to other flights, and then killed the flight record. For some reason, my ticket was still booked on a now non-existent flight, and that confused the hell out of the system and made it give weird info (such as the flight was in the hanger, or that my ticket didn't exist). The supervisor had to re-create the flight, move my ticket off it, and then delete the flight again.

58 points by lawgimenez 3y ago

jacquesm 3y ago

rm -rf vv\ *

datadrivenangel 3y ago

Accidentally deleting prod is a rite of passage.

davidthewatson 3y ago
Been there, done that. The paradox is that every device I use interrupts me non-stop with modal dialogs ad nauseam. But the production delete? You said,"drop", we said, "gone". Either production database deletes need to adopt alert fatigue or the ops are already so fatigued that it doesn't matter.
- ilyt 3y ago
  I'm pretty annoyed by those prompts. Yes, fuck you, do what I fucking said, my backups work fine. Also frankly I think most software does it wrong, if it is possible it shouldn't ask before delete but just allow to undo the step easily. Then work can go fast and uninterrupted while still being resilient to mistakes. Obviously not possible everywhere but where it is it should be the default.
  - Xylakant 3y ago
    Cool that you have a working backup, but restores are what counts. And they’re not instantaneous in almost all cases. So asking before certain destructive actions is still a good practice.
  - darkwater 3y ago
    > Also frankly I think most software does it wrong, if it is possible it shouldn't ask before delete but just allow to undo the step easily. Obviously. When undoing is not easy or possible automatically, there is the "Are you really really sure?" dialog.
- Aerroon 3y ago
  >The paradox is that every device I use interrupts me non-stop with modal dialogs ad nauseam. Pop-up warnings that are too common are a problem itself. People get so used to accepting them immediately that they stop serving their purpose and just become an annoyance. With one exception: they can prevent misclicks.
  - davidthewatson 3y ago
    Right! That's why I used the term alert fatigue: https://en.wikipedia.org/wiki/Alarm_fatigue https://en.wikipedia.org/wiki/Alarm_fatigue The misclick point is beyond truth, particularly given the storm of marketing that is mobile web.
- taneliv 3y ago
  In a company where we chose to do production SQL shenanigans in the spirit of "move fast, break things", I developed a habit of starting those with "BEGIN;" and paying careful attention to the number of records modified before calling it a "COMMIT;". In case of unexpected numbers, immediate "ROLLBACK;" so as to not hog the resources and make the applications suffer. Even fatigued, "DELETE 1182900" tends to stick out, if you're used to, and expecting to, see "DELETE 1". ("BEGIN" appears to be PostgreSQLism, "START TRANSACTION" is more standard.)
moralestapia 3y ago
Definitely! There's probably some mystical mechanism making sure newbies go through it. I consider myself an "experienced" developer, yet on the first day on my current gig I dropped a '--recursive' flag in an AWS command and deleted a bunch of stuff I shouldn't have. ¯\_(ツ)_/¯
- _boffin_ 3y ago
  I’m sure some processes and procedures were updated on that day and the next.
  - Zobat 3y ago
    At some time we learned about delete locks in Azure and promptly created those for all production resources. Have had/heard of enough actual/almost "killed production" stories to understand their value. A bit cumbersome when you actually need to delete something but so worth it for the mental health of the team.
    - bombcar 3y ago
      The key with delete locks and similar is it HAS to be something out-of-band; if the normal way to delete is via a website, the delete lock removal should be API or command line, etc A simple --max-really-delete added to a command line is subject to "replay attacks" on yourself when you use CTRL+R.
pjmlp 3y ago
Yeah, raises hand. Even though I take all care to differenciate environments, always cross check, we had to live drop a table while on a meeting, and due to ongoing conversation the wrong window was selected, and multitasking ended up causing a major distraction.
Moru 3y ago
I sort of did that at my first web-job. I thought it was a bit unhandy that the customers had to be mailed a link to the admin interface, why not just put a link to the login on the front page? A couple of days later I learned why. There's always a good reason somewhere, you just don't know why yet. The reason? Well, the admin interface was written in ASP. According to the guy that called me in panic said that ASP version was so very old it was extremely insecure. I didn't have access to the database so could not verify anything but the webserver logs had a lot of "Bobby tables" in them so I have a strong suspicion it wasn't all ASP's fault... Anyway, we ended up with the database disappearing in the night or containing random data until I got that phonecall asking why we were linking directly to the admin interface :-) They were very good at restoring the database though, just took an hour or so every morning until they found the reason.
briffle 3y ago
At a previous position a co-worker reached his rite of passage. Lots of connections, and not paying attention to which screen window/session he was in. After that, we then changed the bash prompts on all prod servers to be red. Never had another incident..
huehehue 3y ago
I dropped ~100 records from a join table thinking I was connected to the dev db. "No worries, we have backups", I thought. Turns out those were more like replication, copying over every change immediately (including deletes). In the end, fixed with a short script that cross-referenced tables and checked for missing relationships, but terrifying nonetheless.
nso 3y ago
Worked for a CRM SaaS. I once did a DELETE FROM users; WHERE... Open office. Boss rolled over on his chair and started saying "so..." and I dismissed him with a "deleted all users leave me alone". He rolled away, kinda like Homer in that GIF where he disappears into the hedge. Luckily it was back up in minutes, but a new phobia was unlocked.
- thedanbob 3y ago
  > and I dismissed him with a "deleted all users leave me alone" I think I'm going to start unplugging my desk phone when one of those moments happen to me. "Hey, there's something wrong with..." "YES I KNOW I WAS FIXING IT BEFORE YOU INTERRUPTED ME"

notjustanymike 3y ago

♫ Tale as old as time, True as it can be ♫

jan_Sate 3y ago

That's far worse than the gitlab incident.

secretsatan 3y ago

El Regs' "Who Me?" stories should be taken with a grain of salt

fnord77 3y ago
"Dear Penthouse" of IT
- rswail 3y ago
  Now that's a phrase I haven't heard for a while. I wonder what the 2023 equivalent is.
- FireBeyond 3y ago
  "... I never thought I'd be writing this letter to you."
123pie123 3y ago
the one I had published about a support call - saying their PC was not working, whilst fault finding I found out they was in the middle of a powercut - they also hurt themselves because it was dark/ no lights. was 100% true. There was loads of comments saying BS I never bothered writing about the person putting tipex on the screen to delete something!! (true) or the time another person swore blind that they've not touched the keyboard - dispite all the keys being in the wrong order (they took the keys off to clean because they spilt coffee on it and put them back in the wrong order) or the time I nearly wiped out 4 man years of code.....
- ilyt 3y ago
  > or the time another person swore blind that they've not touched the keyboard - dispite all the keys being in the wrong order (they took the keys off to clean because they spilt coffee on it and put them back in the wrong order) That reminds me when one user pissed one of our then-senior admins so they reordered his keyboard keys alphabetically. Or the fun we had where in old system user needed to come to admins to type their first password (that then they are supposed to change after) on account creation and one of admins had blank keycaps keyboard. Reaction was anywhere between "didn't even notice" to "bewilderment and panic"

hmry 3y ago

Happy ending, at least.

jnsie 3y ago

> the manager told him that he understood it wasn't his fault. He blamed the Senior DBA for poor supervision Would have been good management...had they not decided to replace said senior with the new hire junior DBA.

ysavir 3y ago
Agreed. Imagine the junior did this after the senior left and didn't have them to fall back on to restore things.

generalizations 3y ago

Poor senior DBA - another month, and he would've gotten away with forgetting about those backups.

hiyjnix 3y ago

Deleting prod is a rite of passage-- the junior DBA because senior that day.

vsareto 3y ago

One DBA was responsible for all backups for an airline? Why not make it a team effort?

rawgabbit 3y ago
Many years ago, I worked for a fortune 500 company that did billions in revenue. I was literally one of two DBAs for the entire company. Yes, we did have a revolving door of contractors and a team of "DBAs" in India. Did I ever let them touch production? No. I tasked them with monitoring and dealing with all the non prod environments. You have to realize in a big corp there are an infinite number of ways for things to fall apart. Much of it beyond your control. My goal was to minimize the blast radius. One time I took vacation for the first time in years. When I came back I was blamed for an outage while I was on holiday. Their reasoning was that they didn’t push any new code into production so it must be the database’s fault. After a day of troubleshooting, I learned the application code was not using a framework like Microsoft Linq to read the SQL output but was reading it raw. I also learned that the Windows admin had been pushing Windows updates into the servers. The issue was that a Windows update (not a SQL Server update) changed how timestamps were recorded. It was beyond stupid. Instead of yyyymmdd hhmmss like the application expected. It was now yyyymmdd hhmmss timezoneoffset +- hh. They now had some weird hour offset and the actual values changed. To fix the issue the windows update was rolled back. And the Windows admins were told never to touch the database servers without my permission. And the application was rewritten.

ilyt 3y ago

I got four eyes system to be implemented on our main loadbalances thanks to that roughly a decade - got hired and as junior managed to make change in CM that fucked them up... so seniors decided it's time to make it a bit more resilient

Zobat 3y ago

I've marked a few tens of thousands of records as not delivered to a customer, instead of as intended just one record. We had no development or test environment so development were done in production. The system lacked the necessary functions to correct errors so there were frequent updates/corrections made by writing SQL-queries directly towards the production database. The incident was when I should update a single record so the system would resend it to the customer, but ran the query before writing the WHERE-clause. Luckily there were working backups (incremental backups that ran every 15 min, almost like someone knew they might need them!) and no one had made any changes since the last one. We did a quick restore and let out a sigh of relief. We did eventually write a tool to do those updates (and deletes) we were doing multiple times a day and were able to hand that tool over to one of the users, saving us multiple five minute interruptions each day and boosting our productivity. Worth noting that this was part of a business that we bought from another company and we did transfer all customers to our own product with dev environments and no need to manually updating databases.

sowbug 3y ago
A sysadmin at one of my startups once mixed up c and x in a tar backup. Our source repository suddenly teleported back in time. We polled ourselves to see who'd updated most recently, and that guy's snapshot became commit #1 in the new repo. (Unfortunately this was the age of cvs, so the local copies were just source, not the history.)

slowhand09 3y ago

Lucky to have never done this... yet. But I did disable a major (and I do mean MAJOR) system for 20 minutes by misconfiguring an IP tables entry in Prod.

michaelmior 3y ago

This was over a decade ago, but out of paranoia, I found a way to use ROXTerm to colour code my terminals. That way whenever I connected to a production instance, things lit up red so I couldn't be confused about where I was. I don't do this anymore, but it was a fun little hack. https://michael.mior.ca/blog/coloured-ssh-terminals/ https://michael.mior.ca/blog/coloured-ssh-terminals/

s_dev 3y ago
Thanks for this -- I colour all my terminals manually with option+command+i and then select a dark colour where each shell has a colour coded purpose usually to indicate which machine it is. This link will help a lot!
kotaKat 3y ago
Ooh, my former hospital's EMR did something similar to this! In MEDITECH 6.x, your LIVE system instance was blue. TEST environment was green, and any management interface or screen on a server console that was RED was "this will fuck up LIVE" including giant ** WARNING ** LIVE ** WARNING ** LIVE ** MEDITECH warnings in the title bar for the window..

lelima 3y ago

If it ended bad, with no backups, Is there any legal implacations for the Senior DBA?

netsharc 3y ago

> Right on cue the phones started ringing. In every terminal in every airport around the world, the airline was suddenly unable to create, check or cancel tickets. That doesn't sound like they deleted a flight, how often do you hit a flight that's currently interesting to airports around the world? Of course there are exceptions, e.g. Emirates would route you via Dubai, so a flight Dubai to Frankfurt could have connecting passengers from a few dozen different originating countries, but not "every airport around the world". OK, come to think of it, maybe the inconsistency in the DB halted everything, e.g. the system saw a few hundred passengers booked on flight with ID 463523 but there's no record of a flight with that ID...

drra 3y ago

I doubt the sorry is true. It could only happen with a custom in-house system and any interconnected (and distributed worldwide several decades ago) airlines would use an external distribution system for creating and managing tickets.

squeaky-clean 3y ago

It's not deleting prod, but this reminds me of a funny story when we kind of prevented a low-cost airline from being able to sell any tickets for 8~ hours. We ran the booking websites for lots of airlines (Which was really just a frontend that connected to Sabre or Amadeus). Airline data standards were bad. They're probably still bad. You were lucky if you could get an airline's upcoming schedule or pricing rules in some format other than "manually typed into an email by a human". We had people who's job was solely just to take these emails and put the info in the database. We had a bunch of arbitrary safe-guards against data entry mistakes. Ticket prices above 10k or under 10 were flagged as suspicious and needed someone more senior to approve them. So this airline, headquartered somewhere with a 7 hour time difference from us, wants to do a 9 euro flight special for some holiday. There's a big marketing campaign, apparently lots of excited customers, etc. I don't know if you've connected the dots yet but 9 euros was less than our suspicious-data-threshold of 10. So about 95% of their available flights just disappeared. Searching for them would turn up a "no results" page. The only available flights were expensive ones excluded from the sale, usually codeshares with a larger airline for a more distant destination. The airline never notified us. It wasn't until we waltzed into the office 7 hours later that any one of us saw the problem. One of the DBAs wrote a quick "approve all suspicious entries" query and that was the end of it. Thankfully it was a 3 day sale. Afterwards, I remember the chosen method to prevent this from happening again was to change the low threshold for a suspicious price to 8.99 or below.

technothrasher 3y ago

I had my whole flight lost on me one day. I had just flown in to Philly and my connecting flight to Corning was at gate A7 (or whatever the terminal said). I proceeded to that gate and found the gate completely empty and nothing on the monitor. I waited until it was about ten minutes before boarding time and still crickets. I walked to another gate and asked the agent behind the counter about the flight. A bunch of tapping and they said there wasn't any record of that flight and that they couldn't help me. Uh, ok... I next went to the customer service desk, and the agent there did a bunch more tapping and finally said, "oh, here, the gate is H1". Where the heck is H1? "It's that brown building on the other side of the runway, you can see through the window". The hanger?? "Uh... yes." Huh? "<shrug>". I finally went out through security and into the ticketing line and said, "I don't know what is going on, but can I get a seat on the next flight going to Corning?" The agent had a heck of a time trying to cancel my ticket, as it kept telling them there was no ticket even though they could still bring it up on screen. Finally a supervisor figured it out. Somebody had cancelled the flight, moved everybody to other flights, and then killed the flight record. For some reason, my ticket was still booked on a now non-existent flight, and that confused the hell out of the system and made it give weird info (such as the flight was in the hanger, or that my ticket didn't exist). The supervisor had to re-create the flight, move my ticket off it, and then delete the flight again.