4 ms·

I once ran rm -rf in the production mysql data directory. Shit happens.

by chuhnk 11y ago

ivthreadp110 11y ago
When I worked at an ecommerce company the boss (who also was a programmer) did that too (on our master engine- not just website orders, but ebay, amazon, everything historical, inventory, RMAs.... etc etc... OOPS indeed... Yep- shit happens. *Recovered from it- was a hastle and shut down everything, but it was one of thoes whoops moments. Happens to the best of us.
zymhan 11y ago
I did that too. Destroyed out Zabbix database. Neither that Zabbix server, nor the other one monitoring the server I destroyed, could alert us that anything had gone wrong for over an hour. I finally realized it when I couldn't login... I was able to painstakingly rebuild the server after 9 hours without anyone noticing. To this day one of my biggest fuck ups and prouder accomplishments.
- postila 11y ago
  Are you saying that once you break something, you should break your monitoring as well, but do it very quickly since it may be too late? :)
  - zymhan 11y ago
    Yeah, if you're going to break your monitoring solution, you'd better shoot it in the head and vaporize the body.
cflat 11y ago
You are the human manifestation of "the chaos monkey"
- thusoy 11y ago
  For anyone that feel they might fall under the same category (I do!), represent: https://teespring.com/human-chaos-monkey https://teespring.com/human-chaos-monkey
Shizza 11y ago
I'm going to put that on a T-shirt
- thusoy 11y ago
  This close enough? https://teespring.com/human-chaos-monkey https://teespring.com/human-chaos-monkey
AdamJacobMuller 11y ago
Been there. Done that. No backups. Recovered all the data by using open file handles in /proc/. Not a fun two hours. Shit happens. Live and learn.
- marco-lavag 11y ago
  Have you written a blog post about that /proc/ trick? sounds like an interesting read
  - teraflop 11y ago
    Here's an article that explains it: http://archive09.linux.com/feature/58142 http://archive09.linux.com/feature/58142
  - AdamJacobMuller 11y ago
    I don't blog, but, what teraflop posted is basically correct.

I once ran rm -rf in the production mysql data directory. Shit happens.

by chuhnk 11y ago

I once ran rm -rf in the production mysql data directory. Shit happens.

When I worked at an ecommerce company the boss (who also was a programmer) did that too (on our master engine- not just website orders, but ebay, amazon, everything historical, inventory, RMAs.... etc etc... OOPS indeed... Yep- shit happens. *Recovered from it- was a hastle and shut down everything, but it was one of thoes whoops moments. Happens to the best of us.

zymhan 11y ago

I did that too. Destroyed out Zabbix database. Neither that Zabbix server, nor the other one monitoring the server I destroyed, could alert us that anything had gone wrong for over an hour. I finally realized it when I couldn't login... I was able to painstakingly rebuild the server after 9 hours without anyone noticing. To this day one of my biggest fuck ups and prouder accomplishments.

postila 11y ago
Are you saying that once you break something, you should break your monitoring as well, but do it very quickly since it may be too late? :)
- zymhan 11y ago
  Yeah, if you're going to break your monitoring solution, you'd better shoot it in the head and vaporize the body.

cflat 11y ago

You are the human manifestation of "the chaos monkey"

thusoy 11y ago
For anyone that feel they might fall under the same category (I do!), represent: https://teespring.com/human-chaos-monkey https://teespring.com/human-chaos-monkey

Shizza 11y ago

I'm going to put that on a T-shirt

thusoy 11y ago
This close enough? https://teespring.com/human-chaos-monkey https://teespring.com/human-chaos-monkey

AdamJacobMuller 11y ago

Been there. Done that. No backups. Recovered all the data by using open file handles in /proc/. Not a fun two hours. Shit happens. Live and learn.

marco-lavag 11y ago
Have you written a blog post about that /proc/ trick? sounds like an interesting read
- teraflop 11y ago
  Here's an article that explains it: http://archive09.linux.com/feature/58142 http://archive09.linux.com/feature/58142
- AdamJacobMuller 11y ago
  I don't blog, but, what teraflop posted is basically correct.