4 ms·

Oh, that interview question. “Tell me about something you broke in your last job"

by magd 10y ago

ckozlowski 10y ago
It's my favorite. =D
BookmarkSaver 10y ago
"The Internet."
swalsh 10y ago
I once changed a piece of code that was referenced by every page on the customer facing site (10's millions of visits a day) to use a new function that someone had previously written (and was called on 1 page in the site). I mistakenly didn't really look to closely at the implementation of the function, and didn't realize how badly it's caching strategy was designed. When this code was deployed it instantly caused a thundering herd on our cache servers bringing the site down for about ~40 seconds. I felt TERRIBLE about it.
plandis 10y ago
My worst was that one time I accidentally took down some services that were essential for order processing via an unrelated, large stress test. My test accidentally consumed a large amount of bandwidth saturating the links to a shared service. I did all this while sitting 2 feet from a print out of "The 8 fallacies of distributed systems". Bandwidth is indeed not infinite, can confirm.
rconti 10y ago
Paged for "$HOSTNAME is down!" Usual checks to access $HOSTNAME failed Rushed to office at 6am before important process was about to run that needed that host. Plugged in keyboard+monitor, dead screen, nothing. Physically power-cycled server. Stood in front of monitor+keyboard. It occurred to me it was taking longer than expected to show POST screen. About that time, I got a page saying $ACTUALHOSTNAME is down. Walk around to the back of the racks. The monitor cable had come detached from the cable extender that I plugged into the server. I had never plugged the monitor in at all, just the extension. The server wasn't down in the first place, it just lost a virtual interface, which I was paged for, and stupidly tested that virtual interface instead of the REAL name/IP. And then I raced to the office just so that I could cause an outage.
noisy_boy 10y ago
Due to youth, totally misplaced confidence and poor access rights regime, ran an untested script in production causing it to fork uncontrollably requiring reboot.

Oh, that interview question. “Tell me about something you broke in your last job"

by magd 10y ago

Oh, that interview question. “Tell me about something you broke in your last job"

ckozlowski 10y ago

It's my favorite. =D

BookmarkSaver 10y ago

"The Internet."

swalsh 10y ago

I once changed a piece of code that was referenced by every page on the customer facing site (10's millions of visits a day) to use a new function that someone had previously written (and was called on 1 page in the site). I mistakenly didn't really look to closely at the implementation of the function, and didn't realize how badly it's caching strategy was designed. When this code was deployed it instantly caused a thundering herd on our cache servers bringing the site down for about ~40 seconds. I felt TERRIBLE about it.

plandis 10y ago

My worst was that one time I accidentally took down some services that were essential for order processing via an unrelated, large stress test. My test accidentally consumed a large amount of bandwidth saturating the links to a shared service. I did all this while sitting 2 feet from a print out of "The 8 fallacies of distributed systems". Bandwidth is indeed not infinite, can confirm.

rconti 10y ago

Paged for "$HOSTNAME is down!" Usual checks to access $HOSTNAME failed Rushed to office at 6am before important process was about to run that needed that host. Plugged in keyboard+monitor, dead screen, nothing. Physically power-cycled server. Stood in front of monitor+keyboard. It occurred to me it was taking longer than expected to show POST screen. About that time, I got a page saying $ACTUALHOSTNAME is down. Walk around to the back of the racks. The monitor cable had come detached from the cable extender that I plugged into the server. I had never plugged the monitor in at all, just the extension. The server wasn't down in the first place, it just lost a virtual interface, which I was paged for, and stupidly tested that virtual interface instead of the REAL name/IP. And then I raced to the office just so that I could cause an outage.

noisy_boy 10y ago

Due to youth, totally misplaced confidence and poor access rights regime, ran an untested script in production causing it to fork uncontrollably requiring reboot.