4 ms·

Demystifying Site Outages

30 points by mcfunley 14y ago

patdennis 14y ago
Good on them for taking the time to write such a detailed report. Shows respect for their users.
andrewcooke 14y ago
that was interesting, but it felt like two different articles rolled into one (a short non-technical and a long-but-shorter-than-that technical one). anyway, so alter table modify on a primary key (int to bigint) requires that the index be rebuilt? is that true for all databases? or was the problem something else?
- theatrus2 14y ago
  Since the allocation space is different, it's generally a rebuild. The only exception would be a DB storing integers as a variable length int - unusual.
DigitalSea 14y ago
This is the most detailed server outage response I have ever seen. Looks like Etsy understand the importance of accountability and I really appreciate the way they wrote the post. A short version for non-technical and short attention span minded people and a longer version for the nerds who like to know the intricacies of big sites like Etsy.
- gadders 14y ago
  And even the longer version was fairly understandable for non-technical folks, I'd have thought. Very well written. I always remember reading in a book once (Influence by Cialdini?) that the main signifier for whether a doctor gets sued or not is not whether they make mistakes, but how they handle them. 'fessing up honestly is the way to go.
tedunangst 14y ago
Without diving too far into technical details, suffice it to say that in order to support the special characters that some languages require to be complete, we needed to do an upgrade of the database server software. Oh, hi, I see you're using MySQL. MySQL's 1024 character (oh sorry, not characters, bytes, oopsie!) limit for indices was the bane of my existence. You have a mountain of code and tables using varchar(200) and an index spanning five such columns. Works with latin-1. Then somebody decides this new fangled utf-8 is the new hotness. hahahaha, no database index for you! Now you get to go back and decide if you can shrink said columns or get by with a one column index (mysql assumes 3 bytes, so one 200 char column is already more than half your allotment).
- seiji 14y ago
  The hilarious part is this is just Catastrophic MySQL Production Error Case #693, but people won't dare move away from MySQL. They will keep using it and building out new services based on it because there's such a large list of known failure cases. People feel safe knowing how things can fail rather than seeing something "just work" and not knowing why.
- Maxious 14y ago
  Why do you need to do a MySQL upgrade you say? "Up until MySQL 5.5 (released December of 2010) the UTF-8 support was severely hobbled." [missing some Japanese characters, including those used in many people's names, and accents] http://mzsanford.wordpress.com/2010/12/28/mysql-and-unicode/ http://mzsanford.wordpress.com/2010/12/28/mysql-and-unicode/
- unwind 14y ago
  On a different but related note, I found that sentence to be pretty ... strange. They go into lots of technical detail about servers, indices, users and tables and things, but that wording sounds as if they're explaining the world to a (small) child. The idea that there are "normal" (=US ASCII, I presume?) and "special" (=the rest) characters used in human languages is a bit 1980:s, or thereabouts, to me.
  - 14y ago
latchkey 14y ago
In 2004, when etsy.com was created, we all were pretty aware of the importance of specifying utf-8 when we setup systems. How could a company, that now has 120+ engineers listed on the about page, get to 80+ databases without fixing this issue?
- deleted 14y ago
  [deleted]

30 points by mcfunley 14y ago

Good on them for taking the time to write such a detailed report. Shows respect for their users.

that was interesting, but it felt like two different articles rolled into one (a short non-technical and a long-but-shorter-than-that technical one). anyway, so alter table modify on a primary key (int to bigint) requires that the index be rebuilt? is that true for all databases? or was the problem something else?

theatrus2 14y ago
Since the allocation space is different, it's generally a rebuild. The only exception would be a DB storing integers as a variable length int - unusual.

DigitalSea 14y ago

This is the most detailed server outage response I have ever seen. Looks like Etsy understand the importance of accountability and I really appreciate the way they wrote the post. A short version for non-technical and short attention span minded people and a longer version for the nerds who like to know the intricacies of big sites like Etsy.

gadders 14y ago
And even the longer version was fairly understandable for non-technical folks, I'd have thought. Very well written. I always remember reading in a book once (Influence by Cialdini?) that the main signifier for whether a doctor gets sued or not is not whether they make mistakes, but how they handle them. 'fessing up honestly is the way to go.

tedunangst 14y ago

Without diving too far into technical details, suffice it to say that in order to support the special characters that some languages require to be complete, we needed to do an upgrade of the database server software. Oh, hi, I see you're using MySQL. MySQL's 1024 character (oh sorry, not characters, bytes, oopsie!) limit for indices was the bane of my existence. You have a mountain of code and tables using varchar(200) and an index spanning five such columns. Works with latin-1. Then somebody decides this new fangled utf-8 is the new hotness. hahahaha, no database index for you! Now you get to go back and decide if you can shrink said columns or get by with a one column index (mysql assumes 3 bytes, so one 200 char column is already more than half your allotment).

seiji 14y ago
The hilarious part is this is just Catastrophic MySQL Production Error Case #693, but people won't dare move away from MySQL. They will keep using it and building out new services based on it because there's such a large list of known failure cases. People feel safe knowing how things can fail rather than seeing something "just work" and not knowing why.
Maxious 14y ago
Why do you need to do a MySQL upgrade you say? "Up until MySQL 5.5 (released December of 2010) the UTF-8 support was severely hobbled." [missing some Japanese characters, including those used in many people's names, and accents] http://mzsanford.wordpress.com/2010/12/28/mysql-and-unicode/ http://mzsanford.wordpress.com/2010/12/28/mysql-and-unicode/
unwind 14y ago
On a different but related note, I found that sentence to be pretty ... strange. They go into lots of technical detail about servers, indices, users and tables and things, but that wording sounds as if they're explaining the world to a (small) child. The idea that there are "normal" (=US ASCII, I presume?) and "special" (=the rest) characters used in human languages is a bit 1980:s, or thereabouts, to me.
- 14y ago

latchkey 14y ago

In 2004, when etsy.com was created, we all were pretty aware of the importance of specifying utf-8 when we setup systems. How could a company, that now has 120+ engineers listed on the about page, get to 80+ databases without fixing this issue?

deleted 14y ago
[deleted]