4 ms·

Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / d

Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.

wccrawford 16y ago
I did it for the main test database, and STILL felt like crap. SO many headaches.
mathgladiator 16y ago
I once messed up a table that was responsible for a custom-distributed auto-increment field. If you roll that back and don't reset some servers, you have new data overwriting old data. Very bad stuff.
SkyMarshal 16y ago
> Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Or sent a test email to thousands of customers in your prod database encouraging them to use web check-in for their non-existent flight tomorrow. Yeah, did that five years ago, talk about heart-attack-inducing. Quickly remedied by sending a second email to the same test set, thankfully, but that's the kind of mistake you never forget.
- resdirector 16y ago
  I have a strict rule for myself: never use any curse words in any comments, variable names, dummy accounts etc.
  - jrockway 16y ago
    I have the opposite strict rule: use as many curse words in comments, variable names, and dummy accounts as possible. That way you'll find out quickly when someone else notices!
    - 1337p337 16y ago
      I recommend against that. :) A team I was on had a demo for a client, and a shaky database schema. I had used, as a test account, the username "MOTHERFUCKINYEAH" for the same purpose. The purging of this account caused a few 500 errors, and we almost lost the client, and although I didn't get fired, we were all shuffled around after that to less critical projects, and one of us actually got sent out of the state.
  - 1337p337 16y ago
    I don't think they're useful for variable names, but I use them (very sparingly) in comments. A great way to spot what needs refactoring is to find the piece that caused the most frustration to the author. Under a tight deadline, apologies to future maintainers can also be helpful in injecting some levity.
- lambda 16y ago
  Heck, you think that's heart attack inducing? How about the DreamHost case in which they typed the wrong year in their billing code; charged many of their users for an extra year of service, to the tune of $7.5 million: http://blog.dreamhost.com/2008/01/15/um-whoops/ http://blog.dreamhost.com/2008/01/15/um-whoops/ http://blog.dreamhost.com/2008/01/16/the-aftermath/ http://blog.dreamhost.com/2008/01/16/the-aftermath/ http://blog.dreamhost.com/2008/01/17/the-final-update/ http://blog.dreamhost.com/2008/01/17/the-final-update/
  - joelg87 16y ago
    Wow. That is a great read. “What’s the harm in keeping it flexible?” $7.5M in harm, that’s what! "flexibility" is rarely desired in programming!
1337p337 16y ago
I suspect that this is why people tend to understand when GitHub goes down, and this allows them to be more open. All (or nearly all) of GitHub's users are tech-savvy, so we tend to understand the problems.
- haploid 16y ago
  It also doesn't hurt that every user's repositories are "backed up" to localhost. Sounds like the biggest impact was on non-github-users trying to read source or documentation on other people's projects moreso than actual users.
  - 1337p337 16y ago
    Definitely the important part. Linus Torvalds once joked: "Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it." Distributed SCMs like git are a great way to But GitHub's main value over something like, say, gitosis, is all of the metadata. This stuff is mainly useful to github users. Projects you're watching, pull requests, bug reports, fun (if only of dubious general use) graphs. That stuff doesn't come across the wire with the individual repos. The big (in terms of bytes) part, the "events" table, is helpful for keeping up with people and projects (if sometimes noisy). I use the RSS feed of that, though, so I didn't even notice the interruption until I heard about it here.
seldo 16y ago
My worst was discovering I had written a unique ID generator which was (due to me typing "==" instead of "!="), producing duplicate IDs -- and not only that, it was producing them at exponentially increasing rates -- and every duplicate ID was destroying an association in our database, making it unclear what records belonged to who. It was not a good day.
- pixdamix 16y ago
  Mine was for a French social networking site 4 years ago. They used to send mails everyday to say "hey look at the people who you might know". The links on the mail would automatically log the user on the website. When I sent the code live it took 2 days (and more than 50000 mails to found out that when I sent a mail to person Z about person Y the link logged in Z ON Y's account. [1] I knew it was a bad idea to automatically log on the target of the mail. But it was the policy. And it still the case as far as I know. And if you forgot your password don't worry It's stored in plaintext...
  - deleted 16y ago
    [deleted]
Jabbles 16y ago
Like when the google engineer added "/" to the list of bad URLs, thereby marking every single website (!) as possibly dangerous. http://googleblog.blogspot.com/2009/01/this-site-may-harm-your-computer-on.html http://googleblog.blogspot.com/2009/01/this-site-may-harm-yo... Now that would be a bad feeling.

wccrawford 16y ago

I did it for the main test database, and STILL felt like crap. SO many headaches.

mathgladiator 16y ago

I once messed up a table that was responsible for a custom-distributed auto-increment field. If you roll that back and don't reset some servers, you have new data overwriting old data. Very bad stuff.

SkyMarshal 16y ago

> Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Or sent a test email to thousands of customers in your prod database encouraging them to use web check-in for their non-existent flight tomorrow. Yeah, did that five years ago, talk about heart-attack-inducing. Quickly remedied by sending a second email to the same test set, thankfully, but that's the kind of mistake you never forget.

resdirector 16y ago
I have a strict rule for myself: never use any curse words in any comments, variable names, dummy accounts etc.
- jrockway 16y ago
  I have the opposite strict rule: use as many curse words in comments, variable names, and dummy accounts as possible. That way you'll find out quickly when someone else notices!
  - 1337p337 16y ago
    I recommend against that. :) A team I was on had a demo for a client, and a shaky database schema. I had used, as a test account, the username "MOTHERFUCKINYEAH" for the same purpose. The purging of this account caused a few 500 errors, and we almost lost the client, and although I didn't get fired, we were all shuffled around after that to less critical projects, and one of us actually got sent out of the state.
- 1337p337 16y ago
  I don't think they're useful for variable names, but I use them (very sparingly) in comments. A great way to spot what needs refactoring is to find the piece that caused the most frustration to the author. Under a tight deadline, apologies to future maintainers can also be helpful in injecting some levity.
lambda 16y ago
Heck, you think that's heart attack inducing? How about the DreamHost case in which they typed the wrong year in their billing code; charged many of their users for an extra year of service, to the tune of $7.5 million: http://blog.dreamhost.com/2008/01/15/um-whoops/ http://blog.dreamhost.com/2008/01/15/um-whoops/ http://blog.dreamhost.com/2008/01/16/the-aftermath/ http://blog.dreamhost.com/2008/01/16/the-aftermath/ http://blog.dreamhost.com/2008/01/17/the-final-update/ http://blog.dreamhost.com/2008/01/17/the-final-update/
- joelg87 16y ago
  Wow. That is a great read. “What’s the harm in keeping it flexible?” $7.5M in harm, that’s what! "flexibility" is rarely desired in programming!

1337p337 16y ago

I suspect that this is why people tend to understand when GitHub goes down, and this allows them to be more open. All (or nearly all) of GitHub's users are tech-savvy, so we tend to understand the problems.

haploid 16y ago
It also doesn't hurt that every user's repositories are "backed up" to localhost. Sounds like the biggest impact was on non-github-users trying to read source or documentation on other people's projects moreso than actual users.
- 1337p337 16y ago
  Definitely the important part. Linus Torvalds once joked: "Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it." Distributed SCMs like git are a great way to But GitHub's main value over something like, say, gitosis, is all of the metadata. This stuff is mainly useful to github users. Projects you're watching, pull requests, bug reports, fun (if only of dubious general use) graphs. That stuff doesn't come across the wire with the individual repos. The big (in terms of bytes) part, the "events" table, is helpful for keeping up with people and projects (if sometimes noisy). I use the RSS feed of that, though, so I didn't even notice the interruption until I heard about it here.

seldo 16y ago

My worst was discovering I had written a unique ID generator which was (due to me typing "==" instead of "!="), producing duplicate IDs -- and not only that, it was producing them at exponentially increasing rates -- and every duplicate ID was destroying an association in our database, making it unclear what records belonged to who. It was not a good day.

pixdamix 16y ago
Mine was for a French social networking site 4 years ago. They used to send mails everyday to say "hey look at the people who you might know". The links on the mail would automatically log the user on the website. When I sent the code live it took 2 days (and more than 50000 mails to found out that when I sent a mail to person Z about person Y the link logged in Z ON Y's account. [1] I knew it was a bad idea to automatically log on the target of the mail. But it was the policy. And it still the case as far as I know. And if you forgot your password don't worry It's stored in plaintext...
- deleted 16y ago
  [deleted]

Jabbles 16y ago

Like when the google engineer added "/" to the list of bad URLs, thereby marking every single website (!) as possibly dangerous. http://googleblog.blogspot.com/2009/01/this-site-may-harm-your-computer-on.html http://googleblog.blogspot.com/2009/01/this-site-may-harm-yo... Now that would be a bad feeling.