4 ms·
As I remember it, most of the downtime and hairpulling from mid-2009 to mid-2011 was due to EBS issues, wasn't it? Now, maybe that was Amazon's fault for selli
by raldi 14y ago
As I remember it, most of the downtime and hairpulling from mid-2009 to mid-2011 was due to EBS issues, wasn't it?
Now, maybe that was Amazon's fault for selling a flaky storage solution, or maybe it was our fault for not getting our databases the hell off of it, but yeah, it definitely wasn't Postgres's fault.
I'm gonna go page ketralnis, since he got his hands a lot dirtier than I did with respect to reddit's storage technologies.
- jedberg 14y agoA lot of it was EBS and also the replication system, which turned out to be breaking due to a bug in our own code when the whole thing slowed down due to EBS. As a side note, things have been much better, from what I hear, since they moved to local storage. Things work much better here at Netflix for the same reason. I haven't had much of a chance to play with the new Provisioned IOPS much, but from what I've seen that is a pretty solid storage solution.
- ketralnis 14y ago> A lot of it was EBS and also the replication system, which turned out to be breaking due to a bug in our own code when the whole thing slowed down due to EBS. This bug was a pretty rare case, but to be specific, here it is: 1. Databases for a particular data type are specified in the config file like: dbmaster, dbslave1, dbslave2, dbslave3 2. In Python this ends up just like that: a list where the 0th element is special. There's a "config list" (the list read out of the config file) and an "active list", which is those from the config file that have recently reported themselves as live. 3. Reads are sent to (sort of) random.choice(active_list) and writes are send to active_list[0] 4. But wait! If the master is "down" (the EBS backing him has slowed down so far as to mark him unresponsive) then active_list[0] may very well be a slave 5. Londiste now can't reconcile the slave database with the master database. There's a sequence conflict! The symptom would be that a replica would refuse to replicate, we'd detect the replication lag on that slave, and remove it from the active list. Then its load would fall, we'd get an alert, and we'd say "well crap" and rereplicate to it. In practise this didn't happen very often, I can only think of a few occasions off-hand, and generally the master's disk lag was so bad that a downed slave was the least of our problems.
- jedberg 14y ago> In practise this didn't happen very often, I can only think of a few occasions off-hand, and generally the master's disk lag was so bad that a downed slave was the least of our problems. It got progressively worse as time went on the EBS performance got worse. It was happening every couple of weeks after you left, and I think it got even worse after that. Luckily we had dug into the londiste internals and managed to figure out how to restore replication much faster, but that was still super painful. :) But you're right, the disk lag usually caused the immediate problem -- the out of sync slave was usually pretty easy to deal with, depending on how many got out of sync at once. It was just annoying and time consuming.
- redthrowaway 14y agoAs an aside brought on by the fraternal ballyhoo going on here, do the ex-reddit guys still keep in touch? It's kind of sad that Eric (and, I suppose, Alexis) are the only members of the old guard still kicking around there.