3 ms·

Did Reddit overcome its rather serious persistence-side issues? It seemed like the object-blob Postgres store was a disaster, then they brought in Cassandra wh

Did Reddit overcome its rather serious persistence-side issues? It seemed like the object-blob Postgres store was a disaster, then they brought in Cassandra which was initially used as a cache of sorts, but now is the primary point of contact for threads apparently? Reverse-engineering the persistence strategy from the open sourced code is more tedious and potentially inaccurate than I am comfortable with, so could a Reddit employee or code contributor provide some insight as to how those problems were worked past? There was Reddit downtime today (read-only mode), which is common. What's the usual cause of that sort of thing these days?

jedberg 14y ago
I can cover the rest in the talk, but I just want to comment that the Postgres object-blob store (as you describe it :) ) was not only NOT a disaster, but one of the strongest parts of the persistence layer.
- codewright 14y ago
  Then where did the growth pains that necessitated Cassandra come from? I'm trying to build up a mental genealogy of the engineering decisions made so that the knowledge isn't lost to future programmers. Most of you OG peeps have moved onto greener pastures.
  - keysersosa 14y ago
    ketralnis will likely remember this better, but by my recollection we switched to cassandra after we hit the limit of what we could do with memcachedb: http://blog.reddit.com/2010/01/why-did-we-take-reddit-down-for-71.html http://blog.reddit.com/2010/01/why-did-we-take-reddit-down-f... We were already using memcachedb as a persistent keystore, "caching" precomputed listings and comment threads. When it started becoming more trouble than it was worth to maintain, we decided to try cassandra as a beefier replacement.
    - ketralnis 14y ago
      It was capping memcachedb's performance that was the last straw, but it became pretty clear that we were going to similarly cap the performance of any disc that we could afford to put in a postgres write master and that we needed a solution that eliminated that bottleneck, as well as it as a single point of failure. Cassandra (or more generally the dynamo model) did that nicely.
  - jedberg 14y ago
    I'll cover that in the talk, since I'm sure others would like to hear the answer as well. Thanks for the suggestion! If I don't cover it sufficiently, you can hit me up for more info.
  - ketralnis 14y ago
    > Then where did the growth pains that necessitated Cassandra come from? Increased traffic? I don't think I understand the question
    - raldi 14y ago
      Mostly porn and ragecomics, wasn't it?
      - ketralnis 14y ago
        And advice animals
      - codewright 14y ago
        A little less porn these days, no?
    - codewright 14y ago
      >Increased traffic? I don't think I understand the question :| I was trying to determine what component broke first.
      - ketralnis 14y ago
        There's always some component "breaking". That's just how performance works, you're always bounded by some resource. An app like reddit has most of its backend tweaked or rewritten continuously as it becomes more important to find and alleviate some new bottleneck. If you do it right, you predict the next bottleneck before it becomes one but there will always be something. Maybe writes take too long. Why? You've capped the performance of the disc in your DB master (or more importantly of the most expensive single set of discs you can afford). Maybe adding app servers proportional to site-wide traffic stopped helping you keep up at some point. Turns out you're network bandwidth bound. Maybe requests are too slow, but apps aren't using much CPU. Why? Because you're network-latency bound. Hey, it turns out that some tight loop was doing single memache GETs when it could have been doing a single multi get. That last one is just a general performance bug, but that's just it. Every bottleneck that keeps you from just adding resources is a performance bug. Assuming your app is not infinitely fast, there's always something.
- raldi 14y ago
  As I remember it, most of the downtime and hairpulling from mid-2009 to mid-2011 was due to EBS issues, wasn't it? Now, maybe that was Amazon's fault for selling a flaky storage solution, or maybe it was our fault for not getting our databases the hell off of it, but yeah, it definitely wasn't Postgres's fault. I'm gonna go page ketralnis, since he got his hands a lot dirtier than I did with respect to reddit's storage technologies.
  - jedberg 14y ago
    A lot of it was EBS and also the replication system, which turned out to be breaking due to a bug in our own code when the whole thing slowed down due to EBS. As a side note, things have been much better, from what I hear, since they moved to local storage. Things work much better here at Netflix for the same reason. I haven't had much of a chance to play with the new Provisioned IOPS much, but from what I've seen that is a pretty solid storage solution.
    - ketralnis 14y ago
      > A lot of it was EBS and also the replication system, which turned out to be breaking due to a bug in our own code when the whole thing slowed down due to EBS. This bug was a pretty rare case, but to be specific, here it is: 1. Databases for a particular data type are specified in the config file like: dbmaster, dbslave1, dbslave2, dbslave3 2. In Python this ends up just like that: a list where the 0th element is special. There's a "config list" (the list read out of the config file) and an "active list", which is those from the config file that have recently reported themselves as live. 3. Reads are sent to (sort of) random.choice(active_list) and writes are send to active_list[0] 4. But wait! If the master is "down" (the EBS backing him has slowed down so far as to mark him unresponsive) then active_list[0] may very well be a slave 5. Londiste now can't reconcile the slave database with the master database. There's a sequence conflict! The symptom would be that a replica would refuse to replicate, we'd detect the replication lag on that slave, and remove it from the active list. Then its load would fall, we'd get an alert, and we'd say "well crap" and rereplicate to it. In practise this didn't happen very often, I can only think of a few occasions off-hand, and generally the master's disk lag was so bad that a downed slave was the least of our problems.
ketralnis 14y ago
> Did Reddit overcome its rather serious persistence-side issues? Mostly by getting off of EBS > It seemed like the object-blob Postgres store was a disaster Goodness no > then they brought in Cassandra which was initially used as a cache of sorts, but now is the primary point of contact for threads apparently? I don't know what "primary contact for threads" means, Cassandra doesn't really do IPC. Some datatypes are in Postgres (Links, Accounts) and some datatypes are in Cassandra (Votes, Saves/Hides). > There was Reddit downtime today (read-only mode), which is common. What's the usual cause of that sort of thing these days? I can't speak for today now that EBS is mostly out of the loop, but it used to be EBS issues on a DB master

jedberg 14y ago

I can cover the rest in the talk, but I just want to comment that the Postgres object-blob store (as you describe it :) ) was not only NOT a disaster, but one of the strongest parts of the persistence layer.

codewright 14y ago
Then where did the growth pains that necessitated Cassandra come from? I'm trying to build up a mental genealogy of the engineering decisions made so that the knowledge isn't lost to future programmers. Most of you OG peeps have moved onto greener pastures.
- keysersosa 14y ago
  ketralnis will likely remember this better, but by my recollection we switched to cassandra after we hit the limit of what we could do with memcachedb: http://blog.reddit.com/2010/01/why-did-we-take-reddit-down-for-71.html http://blog.reddit.com/2010/01/why-did-we-take-reddit-down-f... We were already using memcachedb as a persistent keystore, "caching" precomputed listings and comment threads. When it started becoming more trouble than it was worth to maintain, we decided to try cassandra as a beefier replacement.
  - ketralnis 14y ago
    It was capping memcachedb's performance that was the last straw, but it became pretty clear that we were going to similarly cap the performance of any disc that we could afford to put in a postgres write master and that we needed a solution that eliminated that bottleneck, as well as it as a single point of failure. Cassandra (or more generally the dynamo model) did that nicely.
- jedberg 14y ago
  I'll cover that in the talk, since I'm sure others would like to hear the answer as well. Thanks for the suggestion! If I don't cover it sufficiently, you can hit me up for more info.
- ketralnis 14y ago
  > Then where did the growth pains that necessitated Cassandra come from? Increased traffic? I don't think I understand the question
  - raldi 14y ago
    Mostly porn and ragecomics, wasn't it?
    - ketralnis 14y ago
      And advice animals
    - codewright 14y ago
      A little less porn these days, no?
  - codewright 14y ago
    >Increased traffic? I don't think I understand the question :| I was trying to determine what component broke first.
    - ketralnis 14y ago
      There's always some component "breaking". That's just how performance works, you're always bounded by some resource. An app like reddit has most of its backend tweaked or rewritten continuously as it becomes more important to find and alleviate some new bottleneck. If you do it right, you predict the next bottleneck before it becomes one but there will always be something. Maybe writes take too long. Why? You've capped the performance of the disc in your DB master (or more importantly of the most expensive single set of discs you can afford). Maybe adding app servers proportional to site-wide traffic stopped helping you keep up at some point. Turns out you're network bandwidth bound. Maybe requests are too slow, but apps aren't using much CPU. Why? Because you're network-latency bound. Hey, it turns out that some tight loop was doing single memache GETs when it could have been doing a single multi get. That last one is just a general performance bug, but that's just it. Every bottleneck that keeps you from just adding resources is a performance bug. Assuming your app is not infinitely fast, there's always something.
raldi 14y ago
As I remember it, most of the downtime and hairpulling from mid-2009 to mid-2011 was due to EBS issues, wasn't it? Now, maybe that was Amazon's fault for selling a flaky storage solution, or maybe it was our fault for not getting our databases the hell off of it, but yeah, it definitely wasn't Postgres's fault. I'm gonna go page ketralnis, since he got his hands a lot dirtier than I did with respect to reddit's storage technologies.
- jedberg 14y ago
  A lot of it was EBS and also the replication system, which turned out to be breaking due to a bug in our own code when the whole thing slowed down due to EBS. As a side note, things have been much better, from what I hear, since they moved to local storage. Things work much better here at Netflix for the same reason. I haven't had much of a chance to play with the new Provisioned IOPS much, but from what I've seen that is a pretty solid storage solution.
  - ketralnis 14y ago
    > A lot of it was EBS and also the replication system, which turned out to be breaking due to a bug in our own code when the whole thing slowed down due to EBS. This bug was a pretty rare case, but to be specific, here it is: 1. Databases for a particular data type are specified in the config file like: dbmaster, dbslave1, dbslave2, dbslave3 2. In Python this ends up just like that: a list where the 0th element is special. There's a "config list" (the list read out of the config file) and an "active list", which is those from the config file that have recently reported themselves as live. 3. Reads are sent to (sort of) random.choice(active_list) and writes are send to active_list[0] 4. But wait! If the master is "down" (the EBS backing him has slowed down so far as to mark him unresponsive) then active_list[0] may very well be a slave 5. Londiste now can't reconcile the slave database with the master database. There's a sequence conflict! The symptom would be that a replica would refuse to replicate, we'd detect the replication lag on that slave, and remove it from the active list. Then its load would fall, we'd get an alert, and we'd say "well crap" and rereplicate to it. In practise this didn't happen very often, I can only think of a few occasions off-hand, and generally the master's disk lag was so bad that a downed slave was the least of our problems.

ketralnis 14y ago

> Did Reddit overcome its rather serious persistence-side issues? Mostly by getting off of EBS > It seemed like the object-blob Postgres store was a disaster Goodness no > then they brought in Cassandra which was initially used as a cache of sorts, but now is the primary point of contact for threads apparently? I don't know what "primary contact for threads" means, Cassandra doesn't really do IPC. Some datatypes are in Postgres (Links, Accounts) and some datatypes are in Cassandra (Votes, Saves/Hides). > There was Reddit downtime today (read-only mode), which is common. What's the usual cause of that sort of thing these days? I can't speak for today now that EBS is mostly out of the loop, but it used to be EBS issues on a DB master