10 ms·

Why Reddit was down for 6 hours

234 points by meghan 16y ago

tedjdziuba 16y ago
> We could make some speculation about the disks possibly losing writes when Postgres flushed commits to disk, but we have no proof to determine what happened. If you read between the lines, this says that EBS lies about the result of fsync(), which is horrifying.
- thaumaturgy 16y ago
  ...I was wondering if I was going to be the only one that caught that. There's not enough information to know if that's even possible or not, or how it might be possible, but as far as I know there aren't any known postgres issues that would cause that. I would be extremely reluctant to store data on a network that sometimes didn't actually store the data without raising an error anywhere.
- jrmg 16y ago
  Most consumer /hard drives/ lie about the result of fsync, as a 'performance optimization'. It's generally possible to fsync, then cut the power before the data is physically on the disk.
  - lsc 16y ago
    Yeah. but even I don't use consumer hard drives in production. (Honestly, I don't know 100% that the 'enterprise' drives are that much better, but I'd guess they'd lie less... I switched because consumer drives tend to hang RAIDs when they fail, while 'enterprise' stuff fails clean)
    - omh 16y ago
      Enterprise hard drives might lie about fsync with caching turned on, but they generally let you disable caching and behave correctly. Of course that completely shafts your performance, but hopefully you were expecting that.
    - wulczer 16y ago
      There's a section in the Postgres docs about fsync and lying hard drives: http://developer.postgresql.org/pgdocs/postgres/wal-reliability.html http://developer.postgresql.org/pgdocs/postgres/wal-reliabil... There's also a Postgres contrib module called pg_test_fsync that tests various fsync modes on your hard drive, and if you get too high results, you can strongly suspect that the dist is lying.
    - CrLf 16y ago
      Enterprise hardware lies about fsync, because even if you loose power, there is a little battery on the RAID controller that's enough to flush the cache to disk or to keep it for hours until the machine is powered up again. When the battery goes bad, the write cache is disabled automatically. On bigger hardware, like SAN storage arrays, the (redundant) batteries keep the whole thing running for a while after the loss of power.
      - omh 16y ago
        You could consider that the whole battery backup etc. means that it isn't lying about fsync. It says that it's been permanently written, and then it makes sure that is has. It might not have been burnt to spinning metal, but the system as a whole will ensure that it's permanent.
      - lsc 16y ago
        "enterprise" disk is just that; it's just disk. I've not seen a disk with an onboard battery. You can put a raid controller in front of your disks, with or without battery backed cache. (most raid cards have an optional battery module.) But, even many gold plated 'corporate' systems I've seen omit the battery, as it's usually not cheap. My personal opinion is that raid controllers are of little value without battery backed cache. I just use MD in front of 'enterprise sata' drives, as without doubling my total cost for storage, I can't get a raid card, as far as I can tell, which is better than MD, and really, if I was to double my storage cost, simply buying twice as many spindles would get me better bang for my buck, I think, than a hardware raid card that cost that much.
- moe 16y ago
  If you read between the lines, this says that EBS lies about the result of fsync(), which is horrifying. Jump to conclusions much? Do you know if reddit had fsync enabled in first place?
  - khafra 16y ago
    If you read between the lines, "if you read between the lines" almost always means "if you make an educated jump to conclusions."
    - moe 16y ago
      So where is the education in jumping from "we have no proof to determine what happened" to an outrageous claim about EBS? It's very common to turn off fsync on a database for performance reasons. It's far less common to have a network block device driver (which tend to be designed with intermittent outages in mind) lie about fsync.
- patrickgzill 16y ago
  You can actually configure postgres to not do an fsync() after each transaction; instead it gets written to disk, which on Linux means written to the disk caches in RAM, then Linux flushes to disk when it does the rest of the disk writes (anywhere from 5 to 30 seconds are set as default on different versions of Linux).
  - tedjdziuba 16y ago
    Why would you want to do that? This isn't MongoDB, the real world cares about durability.
    - patrickgzill 16y ago
      I am not sure why I was downvoted - I wasn't suggesting as a course of action... it gives a large speed increase and I was suggesting that possibly this was done as a performance measure.
- _delirium 16y ago
  A comment from an ex reddit employee linked elsewhere in this HN discussion (http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l6ykx http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...) seems to confirm that: More recently we also discovered that these disks [EBS volumes experiencing performance degradation] will also frequently report that a disk transaction has been committed to hardware but are flat-out lying.
duck 16y ago
Then, something really bad happened. Something which made the earlier outage a comparative walk in the park. Murphy's law on St. Patrick's Day. Doesn't get any better than that.
- jedberg 16y ago
  I didn't even get a chance to have a Guinness today. :(
  - bryanh 16y ago
    At this rate, a nice, tall Guinness is probably just what the doctor ordered.
rgrieselhuber 16y ago
Great writeup. I'd love to hear other people's experience with regards to workarounds when / if EBS goes down (switching over to RDS for a short time, etc.). The comment about moving to local storage was interesting. Isn't the local storage on EC2 instances extremely limited (like 10-20GB?)
- ktsmith 16y ago
  Assuming "local storage" is synonymous with "instance storage" it's 160GB to 1690GB. http://aws.amazon.com/ec2/instance-types/ http://aws.amazon.com/ec2/instance-types/
  - rgrieselhuber 16y ago
    That makes more sense, thanks.
  - gregburek 16y ago
    So, I assume instances reddit uses have instance storage root volumes instead of EBS root vols. I've always assumed the persistence of ebs AMIs was a plus without a downside. Why would you opt for instance-storage AMIs instead of ebs root volume EC2 AMIs?
    - ktsmith 16y ago
      Being that EBS booting only became an option in December 2009 I would not be surprised if Reddit had not migrated their instances to that boot/storage method. They acknowledged in the last two years they hadn't even had time to move one of their databases from a single EBS volume to striped EBS volumes.
      - jedberg 16y ago
        We're currently in the process of replacing every one of our hosts with new OS versions. As we do this we are in fact going to the EBS based instances. Those instances actually show the same problems, but they aren't too bad, because once you boot them, you don't need the root vol that much (that's what the instance storage is for).
A1kmm 16y ago
Amazon claims: "Each storage volume is automatically replicated within the same Availability Zone. This prevents data loss due to failure of any single hardware component". They make it sound like they are already providing RAID or something similar; however, the fact that things like this happen to Reddit, who have built their own RAID on top of Amazon's already replicated volumes, show that reliability is not a good reason to go with AWS.
- Montagist 16y ago
  Yea I always kind of assumed they'd something like that - RAIDed everything, but then again I think that may be the difference between using one of their DBs and running your own atop EBS. EBS hasn't an actual file-system, right? Or does it? I think it's more raw, and I guess more "use at your own risk" apparently. Damn Amazon, not a good look. Also, why aren't dudes sending you apology fruit baskets or refunding money or something?
- misterbwong 16y ago
  Disk access was the single reason we couldn't go with AWS offerings. The speed/reliability of EBS just isn't where we need it to be for our database servers. I don't blame amazon for this-it's just a drawback of the their choice to go fully shared tenant and virtual.
- parasubvert 16y ago
  EBS isn't really RAIDed, it's virtualized block storage with replicas. The issue Reddit experienced wasn't drive failure, though, it was network degradation. The solution is to deploy redundant replicas in different availability zones (and/or regions, if you can). Reddit unfortunately wasn't built for that. This isn't really any different from an on-premise application. An availability zone by definition implies "shared network hardware". Using multiple is what you do when you want redundancy.
  - snewman 16y ago
    How do you know the issue was network degradation? Is this written up somewhere?
X-Istence 16y ago
I always love seeing a good technical post-mortem of what went wrong and how it could be fixed in the future... I'm currently working on building a backend service that has to scale massively as well, and it has been a fun challenge trying to understand exactly where things can go wrong and how wrong they can go...
deleted 16y ago
[deleted]
naner 16y ago
http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l6ykx http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d... A former employee is not quite as nice to Amazon.
- brianwillis 16y ago
  Assuming for a minute that Amazon deserves as much blame as ketralnis is heaping on here, why would the Reddit guys be so reluctant to point this out? Professionalism? Kindheartedness? Even professionalism and mutual respect have limits. The community loves both the site and the admins, but there are limits to the patience of users, and those limits are being tested by these outages. I would think the Reddit guys would be happy to have a scapegoat to direct the community's rage towards.
  - FiReaNG3L 16y ago
    Because the whole site is hosted on Amazon, and it would be non-trivial to move from it. They also may or may not be receiving special pricing due to their size?
  - idlewords 16y ago
    Blaming a third party lacks class. The Reddit guys made the decision to rely heavily on EBS, and it came back to bite them. They show a lot of character by taking responsibility for an outage they had very little control over.
    - phire 16y ago
      Also, its not a great idea to badmouth someone who is currently providing you with a service and actively working on improving it for you (according to the blog post.)
      - masklinn 16y ago
        Then again, according to the comments Amazon is mostly actively promising to work on improving it, and has been for a year with little to show for it.
kowsik 16y ago
EBS storage aside, they are down to 3 guys? yikes
- mttwrnr 16y ago
  They have been granted a lot more help from Conde Nast. They're in the process of hiring four more developers. http://blog.reddit.com/2011/03/so-long-and-thanks-for-all-postcards.html http://blog.reddit.com/2011/03/so-long-and-thanks-for-all-po...
  - kowsik 16y ago
    devops and all, the ratio's still staggering for the number of hits that reddit gets.
- Raphael 16y ago
  Just one programmer, spladug, on the job for 4 months.
- brown9-2 16y ago
  Idle speculation is a bit useless, but it really makes you wonder why so many of their team have left recently, doesn't it? I wonder if the stress of keeping up with the increased load (and subsequent downtime) became too much for some of them.
marcamillion 16y ago
Wow...they sound like they are really beating themselves up over it. I know the community can be demanding, but that just seems stressful.
bryanh 16y ago
On that note, I have been meaning to ask HN (even if nothing more than an exercise)... If you had to run a site like Reddit, what would you do?
- phire 16y ago
  Most importantly, I wouldn't let the staffing levels get this low. At this point in its life, reddit should have 6-12 programmers/system administrators + a few support staff, compared with the 3 they have at the moment. That way they won't be agonizing over the choice between devoting their resources to keeping reddit running in the short term, or to moving reddit away from EC2 for long term stability.
  - stingraycharles 16y ago
    "Most importantly, I wouldn't let the staffing levels get this low." How would you pay for that?
    - phire 16y ago
      Reddit has never been low on money. According to one of their old developers, the staffing issues have always been political, not financial. http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l7ajp?context=1 http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...
      - _delirium 16y ago
        Ah yes, the telltale sign of a large bureaucracy: people spending $150 of one kind of money to avoid spending $100 of a different kind of money (in this case, payroll budget versus tech-infrastructure budget).
- lsc 16y ago
  Well, I'm a SysAdmin/hardware guy myself, so obviously, I'd buy my own hardware and run it. Once a month I'd spin up my off-site backups (that would be on ec2) to make sure I had somewhere to go if the shit hit the fan at my co-lo. Of course, I've spent my life dealing with infrastructure, so running a rack or two (or ten) servers is going to be a lot cheaper/easier for me than it would be for someone without that experience. The economics for people unwilling to gain that experience will depend entirely on scale; e.g. do they have enough servers that the lower running cost of owning hardware would pay for someone to manage that sort of thing. (and yes, hiring other people has overhead in and of itself.) Generally, as much as possible, I avoid building complexity. I find that having a single point of failure with a backup that can be manually brought in to place (such as an asynchronously replicated database) is quite often more reliable than fancy home-made SAN solutions. In general, you need to be /very careful/ of complex redundant systems. In fact, I approach it somewhat like crypto. As much as possible, I don't build it myself. use well-used open-source tools with well-known failure modes. The other thing to think of is failure domains. Sure, think about single points of failure, but more importantly, think about what goes down if that single point fails. For instance, in my current setup, each xen host is a single point of failure... but one going down won't take down anything else. I've seen other people design similar systems with improvised SAN setups, thinking "oh, if one node dies, I'll boot the guests on another!" the problem is that if that san goes down, everyone is toast, while if one of my hosts goes down, we're talking maybe 1/40th of my customers who are out of action. I've seen drbd setups where a guest mirrored itself locally and to a second server... It sounded like a great idea, but the system turned out to be less reliable than my dumb local storage setup, as weirdness in drbd and how drbd dealt with disk and network issues would cause lockups that were much more frequent than the hardware failures that would take down whole nodes in my local disk setup. In fact, I have a very strong suspicion, born of hard experience and smoking pagers, of SANs that cost less than mid-sized bay-area condos. And I'm pretty cheap, so that means local storage for me. There has been a lot of activity in that field lately, so I'm very carefully exploring it again, but I certainly wouldn't count on some homemade san as being more reliable than local disk. (to be fair, my experience lies with "expensive SANs, used expensive SANs, and homemade SANs." and my only good experience was with the first of those. I don't have a lot of experience with low-cost commercial SANs.) Morever, I'm pretty suspicious of disk over the network schemes, even when the expensive "real" SANs are involved. NFS is the only scheme I really trust; it's older than I am. It has problems, but we know about all of them. Overall, I think NFS handles network blips much better than any of the block device over the network schemes I've used. And you do see network blips. The network simply isn't as reliable as your sata cable, and the block subsystem isn't designed to deal with devices that are temporarily unavailable. I've seen a lot of 'clever' redundant setups built by people who are much smarter than I am... quite often, their setup ends up becoming less reliable than my "dumb" systems.
jamwt 16y ago
I know it's not exactly in vogue these days to tout the merits of bare hardware, but.. after all the VPS hubbub over the last couple of years, the best progression for your website still seems to be: 1. No traction? Just put it anywhere, 'cause frankly, it doesn't matter. Cheapest reputable VPS possible. Let's say, Linode. 2. Scaling out, high concurrency and rapid growth? DEDICATED hardware from a QUALITY service provider--use rackspace, softlayer et al. Have them rack the servers for you and you'll still get ~3 hour turnarounds on new server orders. That's plenty fast for most kinds of growth. No inventory to deal with, and with deployment automation you're really not doing much "sysadmin-y" work or requiring full timers that know what Cisco switch to buy. 3. Technology megacorp, top-100 site? Staff up on hardcore net admin and sysadmin types, colocate first, and eventually, take control of/design the entire datacenter. I simply don't understand why so many of these high-traffic services continue to rely on VPSes for phase 2 instead of managed or unmanaged dedicated hosting. The price/concurrent user is competitive or cheaper for bare metal. Most critically, it's insanely hard to predictably scale out database systems with high write loads when you have unpredictable virtualized (or even networked) I/O performance on your nodes.
- jedberg 16y ago
  reddit actually is a top 100 site, but we don't have nearly the need to host our own datacenter or co-locate. If we do make a move, it will be to #2. I don't want to hire people to be hands on -- I'd rather outsource that and let someone else pay to have spare capacity laying around.
  - jamwt 16y ago
    Fair enough; 100 was sort of arbitrary, but there is some metric when you're sort of at google scale and cutting down on power costs or achieving absolute minimum latencies between datacenters has a meaningful impact on the bottom line. Draw a line somewhere, and yeah, Reddit is probably on the other side.
    - phire 16y ago
      Reddit has always amazed me with what they can do with extremely limited resources. But it looks like that attitude has finally caught up with them, especially since they are down to just 3 technical staff (from 5 last week), and two of them are brand new.
deleted 16y ago
[deleted]
PaulHoule 16y ago
I had two machines running in east-1 last night and one of them went down around the same time reddit did. The other one made it through the night O.K. EBS problems do seem to be the biggest reliability problem in EC2 right now. The most common symptom is that a machine goes to 100% CPU use and 'locks up'. Stopping the instance and restarting usually solves the problem. The events also appear to be clustered in time. I've had instances go for a month with no problems, then it happens 6 times in the next 24 hours. My sites are small, but one of them runs VERY big batch jobs periodically that take up a lot of RAM and CPU. Being able to rent a very powerful machine for a short time to get the batch job done without messing up the site is a big plus.
jwcacces 16y ago
This is why you don't outsource your bread and butter, people! If you want to outsource who makes your lunch, fine, but if your whole business is requests in, data out, you do not put the responsibility of storing your data in someone else's hands. I get it, Amazon EBS is cheap. But at the end of the day you've got to make sure it's your fingers on the pulse of those servers, not someone else who's priorities and vigilance may not always line up with yours. (also the cloud is dumb)
- tomkarlo 16y ago
  It's all a continuum. You could also build your own servers, or design specialized boards with dedicated processors optimized to your application. Everyone is going to choose a point on the continuum and each will have tradeoffs.
- nkohari 16y ago
  You're still outsourcing if you go with a managed dedicated hosting service, or even if you buy hardware and colocate it. Even if you owned the datacenter and the entire backbone, you're still banking on everyone else not fucking up their end of the connection.
  - jwcacces 16y ago
    Yeah, but at least you can take direct action when your people fuck up.

234 points by meghan 16y ago

> We could make some speculation about the disks possibly losing writes when Postgres flushed commits to disk, but we have no proof to determine what happened. If you read between the lines, this says that EBS lies about the result of fsync(), which is horrifying.

thaumaturgy 16y ago
...I was wondering if I was going to be the only one that caught that. There's not enough information to know if that's even possible or not, or how it might be possible, but as far as I know there aren't any known postgres issues that would cause that. I would be extremely reluctant to store data on a network that sometimes didn't actually store the data without raising an error anywhere.
jrmg 16y ago
Most consumer /hard drives/ lie about the result of fsync, as a 'performance optimization'. It's generally possible to fsync, then cut the power before the data is physically on the disk.
- lsc 16y ago
  Yeah. but even I don't use consumer hard drives in production. (Honestly, I don't know 100% that the 'enterprise' drives are that much better, but I'd guess they'd lie less... I switched because consumer drives tend to hang RAIDs when they fail, while 'enterprise' stuff fails clean)
  - omh 16y ago
    Enterprise hard drives might lie about fsync with caching turned on, but they generally let you disable caching and behave correctly. Of course that completely shafts your performance, but hopefully you were expecting that.
  - wulczer 16y ago
    There's a section in the Postgres docs about fsync and lying hard drives: http://developer.postgresql.org/pgdocs/postgres/wal-reliability.html http://developer.postgresql.org/pgdocs/postgres/wal-reliabil... There's also a Postgres contrib module called pg_test_fsync that tests various fsync modes on your hard drive, and if you get too high results, you can strongly suspect that the dist is lying.
  - CrLf 16y ago
    Enterprise hardware lies about fsync, because even if you loose power, there is a little battery on the RAID controller that's enough to flush the cache to disk or to keep it for hours until the machine is powered up again. When the battery goes bad, the write cache is disabled automatically. On bigger hardware, like SAN storage arrays, the (redundant) batteries keep the whole thing running for a while after the loss of power.
    - omh 16y ago
      You could consider that the whole battery backup etc. means that it isn't lying about fsync. It says that it's been permanently written, and then it makes sure that is has. It might not have been burnt to spinning metal, but the system as a whole will ensure that it's permanent.
    - lsc 16y ago
      "enterprise" disk is just that; it's just disk. I've not seen a disk with an onboard battery. You can put a raid controller in front of your disks, with or without battery backed cache. (most raid cards have an optional battery module.) But, even many gold plated 'corporate' systems I've seen omit the battery, as it's usually not cheap. My personal opinion is that raid controllers are of little value without battery backed cache. I just use MD in front of 'enterprise sata' drives, as without doubling my total cost for storage, I can't get a raid card, as far as I can tell, which is better than MD, and really, if I was to double my storage cost, simply buying twice as many spindles would get me better bang for my buck, I think, than a hardware raid card that cost that much.
moe 16y ago
If you read between the lines, this says that EBS lies about the result of fsync(), which is horrifying. Jump to conclusions much? Do you know if reddit had fsync enabled in first place?
- khafra 16y ago
  If you read between the lines, "if you read between the lines" almost always means "if you make an educated jump to conclusions."
  - moe 16y ago
    So where is the education in jumping from "we have no proof to determine what happened" to an outrageous claim about EBS? It's very common to turn off fsync on a database for performance reasons. It's far less common to have a network block device driver (which tend to be designed with intermittent outages in mind) lie about fsync.
patrickgzill 16y ago
You can actually configure postgres to not do an fsync() after each transaction; instead it gets written to disk, which on Linux means written to the disk caches in RAM, then Linux flushes to disk when it does the rest of the disk writes (anywhere from 5 to 30 seconds are set as default on different versions of Linux).
- tedjdziuba 16y ago
  Why would you want to do that? This isn't MongoDB, the real world cares about durability.
  - patrickgzill 16y ago
    I am not sure why I was downvoted - I wasn't suggesting as a course of action... it gives a large speed increase and I was suggesting that possibly this was done as a performance measure.
_delirium 16y ago
A comment from an ex reddit employee linked elsewhere in this HN discussion (http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l6ykx http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...) seems to confirm that: More recently we also discovered that these disks [EBS volumes experiencing performance degradation] will also frequently report that a disk transaction has been committed to hardware but are flat-out lying.

duck 16y ago

Then, something really bad happened. Something which made the earlier outage a comparative walk in the park. Murphy's law on St. Patrick's Day. Doesn't get any better than that.

jedberg 16y ago
I didn't even get a chance to have a Guinness today. :(
- bryanh 16y ago
  At this rate, a nice, tall Guinness is probably just what the doctor ordered.

rgrieselhuber 16y ago

Great writeup. I'd love to hear other people's experience with regards to workarounds when / if EBS goes down (switching over to RDS for a short time, etc.). The comment about moving to local storage was interesting. Isn't the local storage on EC2 instances extremely limited (like 10-20GB?)

ktsmith 16y ago
Assuming "local storage" is synonymous with "instance storage" it's 160GB to 1690GB. http://aws.amazon.com/ec2/instance-types/ http://aws.amazon.com/ec2/instance-types/
- rgrieselhuber 16y ago
  That makes more sense, thanks.
- gregburek 16y ago
  So, I assume instances reddit uses have instance storage root volumes instead of EBS root vols. I've always assumed the persistence of ebs AMIs was a plus without a downside. Why would you opt for instance-storage AMIs instead of ebs root volume EC2 AMIs?
  - ktsmith 16y ago
    Being that EBS booting only became an option in December 2009 I would not be surprised if Reddit had not migrated their instances to that boot/storage method. They acknowledged in the last two years they hadn't even had time to move one of their databases from a single EBS volume to striped EBS volumes.
    - jedberg 16y ago
      We're currently in the process of replacing every one of our hosts with new OS versions. As we do this we are in fact going to the EBS based instances. Those instances actually show the same problems, but they aren't too bad, because once you boot them, you don't need the root vol that much (that's what the instance storage is for).

A1kmm 16y ago

Amazon claims: "Each storage volume is automatically replicated within the same Availability Zone. This prevents data loss due to failure of any single hardware component". They make it sound like they are already providing RAID or something similar; however, the fact that things like this happen to Reddit, who have built their own RAID on top of Amazon's already replicated volumes, show that reliability is not a good reason to go with AWS.

Montagist 16y ago
Yea I always kind of assumed they'd something like that - RAIDed everything, but then again I think that may be the difference between using one of their DBs and running your own atop EBS. EBS hasn't an actual file-system, right? Or does it? I think it's more raw, and I guess more "use at your own risk" apparently. Damn Amazon, not a good look. Also, why aren't dudes sending you apology fruit baskets or refunding money or something?
misterbwong 16y ago
Disk access was the single reason we couldn't go with AWS offerings. The speed/reliability of EBS just isn't where we need it to be for our database servers. I don't blame amazon for this-it's just a drawback of the their choice to go fully shared tenant and virtual.
parasubvert 16y ago
EBS isn't really RAIDed, it's virtualized block storage with replicas. The issue Reddit experienced wasn't drive failure, though, it was network degradation. The solution is to deploy redundant replicas in different availability zones (and/or regions, if you can). Reddit unfortunately wasn't built for that. This isn't really any different from an on-premise application. An availability zone by definition implies "shared network hardware". Using multiple is what you do when you want redundancy.
- snewman 16y ago
  How do you know the issue was network degradation? Is this written up somewhere?

X-Istence 16y ago

I always love seeing a good technical post-mortem of what went wrong and how it could be fixed in the future... I'm currently working on building a backend service that has to scale massively as well, and it has been a fun challenge trying to understand exactly where things can go wrong and how wrong they can go...

deleted 16y ago

[deleted]

naner 16y ago

http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l6ykx http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d... A former employee is not quite as nice to Amazon.

brianwillis 16y ago
Assuming for a minute that Amazon deserves as much blame as ketralnis is heaping on here, why would the Reddit guys be so reluctant to point this out? Professionalism? Kindheartedness? Even professionalism and mutual respect have limits. The community loves both the site and the admins, but there are limits to the patience of users, and those limits are being tested by these outages. I would think the Reddit guys would be happy to have a scapegoat to direct the community's rage towards.
- FiReaNG3L 16y ago
  Because the whole site is hosted on Amazon, and it would be non-trivial to move from it. They also may or may not be receiving special pricing due to their size?
- idlewords 16y ago
  Blaming a third party lacks class. The Reddit guys made the decision to rely heavily on EBS, and it came back to bite them. They show a lot of character by taking responsibility for an outage they had very little control over.
  - phire 16y ago
    Also, its not a great idea to badmouth someone who is currently providing you with a service and actively working on improving it for you (according to the blog post.)
    - masklinn 16y ago
      Then again, according to the comments Amazon is mostly actively promising to work on improving it, and has been for a year with little to show for it.

kowsik 16y ago

EBS storage aside, they are down to 3 guys? yikes

mttwrnr 16y ago
They have been granted a lot more help from Conde Nast. They're in the process of hiring four more developers. http://blog.reddit.com/2011/03/so-long-and-thanks-for-all-postcards.html http://blog.reddit.com/2011/03/so-long-and-thanks-for-all-po...
- kowsik 16y ago
  devops and all, the ratio's still staggering for the number of hits that reddit gets.
Raphael 16y ago
Just one programmer, spladug, on the job for 4 months.
brown9-2 16y ago
Idle speculation is a bit useless, but it really makes you wonder why so many of their team have left recently, doesn't it? I wonder if the stress of keeping up with the increased load (and subsequent downtime) became too much for some of them.

marcamillion 16y ago

Wow...they sound like they are really beating themselves up over it. I know the community can be demanding, but that just seems stressful.

bryanh 16y ago

On that note, I have been meaning to ask HN (even if nothing more than an exercise)... If you had to run a site like Reddit, what would you do?

phire 16y ago
Most importantly, I wouldn't let the staffing levels get this low. At this point in its life, reddit should have 6-12 programmers/system administrators + a few support staff, compared with the 3 they have at the moment. That way they won't be agonizing over the choice between devoting their resources to keeping reddit running in the short term, or to moving reddit away from EC2 for long term stability.
- stingraycharles 16y ago
  "Most importantly, I wouldn't let the staffing levels get this low." How would you pay for that?
  - phire 16y ago
    Reddit has never been low on money. According to one of their old developers, the staffing issues have always been political, not financial. http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l7ajp?context=1 http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...
    - _delirium 16y ago
      Ah yes, the telltale sign of a large bureaucracy: people spending $150 of one kind of money to avoid spending $100 of a different kind of money (in this case, payroll budget versus tech-infrastructure budget).
lsc 16y ago
Well, I'm a SysAdmin/hardware guy myself, so obviously, I'd buy my own hardware and run it. Once a month I'd spin up my off-site backups (that would be on ec2) to make sure I had somewhere to go if the shit hit the fan at my co-lo. Of course, I've spent my life dealing with infrastructure, so running a rack or two (or ten) servers is going to be a lot cheaper/easier for me than it would be for someone without that experience. The economics for people unwilling to gain that experience will depend entirely on scale; e.g. do they have enough servers that the lower running cost of owning hardware would pay for someone to manage that sort of thing. (and yes, hiring other people has overhead in and of itself.) Generally, as much as possible, I avoid building complexity. I find that having a single point of failure with a backup that can be manually brought in to place (such as an asynchronously replicated database) is quite often more reliable than fancy home-made SAN solutions. In general, you need to be /very careful/ of complex redundant systems. In fact, I approach it somewhat like crypto. As much as possible, I don't build it myself. use well-used open-source tools with well-known failure modes. The other thing to think of is failure domains. Sure, think about single points of failure, but more importantly, think about what goes down if that single point fails. For instance, in my current setup, each xen host is a single point of failure... but one going down won't take down anything else. I've seen other people design similar systems with improvised SAN setups, thinking "oh, if one node dies, I'll boot the guests on another!" the problem is that if that san goes down, everyone is toast, while if one of my hosts goes down, we're talking maybe 1/40th of my customers who are out of action. I've seen drbd setups where a guest mirrored itself locally and to a second server... It sounded like a great idea, but the system turned out to be less reliable than my dumb local storage setup, as weirdness in drbd and how drbd dealt with disk and network issues would cause lockups that were much more frequent than the hardware failures that would take down whole nodes in my local disk setup. In fact, I have a very strong suspicion, born of hard experience and smoking pagers, of SANs that cost less than mid-sized bay-area condos. And I'm pretty cheap, so that means local storage for me. There has been a lot of activity in that field lately, so I'm very carefully exploring it again, but I certainly wouldn't count on some homemade san as being more reliable than local disk. (to be fair, my experience lies with "expensive SANs, used expensive SANs, and homemade SANs." and my only good experience was with the first of those. I don't have a lot of experience with low-cost commercial SANs.) Morever, I'm pretty suspicious of disk over the network schemes, even when the expensive "real" SANs are involved. NFS is the only scheme I really trust; it's older than I am. It has problems, but we know about all of them. Overall, I think NFS handles network blips much better than any of the block device over the network schemes I've used. And you do see network blips. The network simply isn't as reliable as your sata cable, and the block subsystem isn't designed to deal with devices that are temporarily unavailable. I've seen a lot of 'clever' redundant setups built by people who are much smarter than I am... quite often, their setup ends up becoming less reliable than my "dumb" systems.

jamwt 16y ago

I know it's not exactly in vogue these days to tout the merits of bare hardware, but.. after all the VPS hubbub over the last couple of years, the best progression for your website still seems to be: 1. No traction? Just put it anywhere, 'cause frankly, it doesn't matter. Cheapest reputable VPS possible. Let's say, Linode. 2. Scaling out, high concurrency and rapid growth? DEDICATED hardware from a QUALITY service provider--use rackspace, softlayer et al. Have them rack the servers for you and you'll still get ~3 hour turnarounds on new server orders. That's plenty fast for most kinds of growth. No inventory to deal with, and with deployment automation you're really not doing much "sysadmin-y" work or requiring full timers that know what Cisco switch to buy. 3. Technology megacorp, top-100 site? Staff up on hardcore net admin and sysadmin types, colocate first, and eventually, take control of/design the entire datacenter. I simply don't understand why so many of these high-traffic services continue to rely on VPSes for phase 2 instead of managed or unmanaged dedicated hosting. The price/concurrent user is competitive or cheaper for bare metal. Most critically, it's insanely hard to predictably scale out database systems with high write loads when you have unpredictable virtualized (or even networked) I/O performance on your nodes.

jedberg 16y ago
reddit actually is a top 100 site, but we don't have nearly the need to host our own datacenter or co-locate. If we do make a move, it will be to #2. I don't want to hire people to be hands on -- I'd rather outsource that and let someone else pay to have spare capacity laying around.
- jamwt 16y ago
  Fair enough; 100 was sort of arbitrary, but there is some metric when you're sort of at google scale and cutting down on power costs or achieving absolute minimum latencies between datacenters has a meaningful impact on the bottom line. Draw a line somewhere, and yeah, Reddit is probably on the other side.
  - phire 16y ago
    Reddit has always amazed me with what they can do with extremely limited resources. But it looks like that attitude has finally caught up with them, especially since they are down to just 3 technical staff (from 5 last week), and two of them are brand new.

deleted 16y ago

[deleted]

PaulHoule 16y ago

I had two machines running in east-1 last night and one of them went down around the same time reddit did. The other one made it through the night O.K. EBS problems do seem to be the biggest reliability problem in EC2 right now. The most common symptom is that a machine goes to 100% CPU use and 'locks up'. Stopping the instance and restarting usually solves the problem. The events also appear to be clustered in time. I've had instances go for a month with no problems, then it happens 6 times in the next 24 hours. My sites are small, but one of them runs VERY big batch jobs periodically that take up a lot of RAM and CPU. Being able to rent a very powerful machine for a short time to get the batch job done without messing up the site is a big plus.

jwcacces 16y ago

This is why you don't outsource your bread and butter, people! If you want to outsource who makes your lunch, fine, but if your whole business is requests in, data out, you do not put the responsibility of storing your data in someone else's hands. I get it, Amazon EBS is cheap. But at the end of the day you've got to make sure it's your fingers on the pulse of those servers, not someone else who's priorities and vigilance may not always line up with yours. (also the cloud is dumb)

tomkarlo 16y ago
It's all a continuum. You could also build your own servers, or design specialized boards with dedicated processors optimized to your application. Everyone is going to choose a point on the continuum and each will have tradeoffs.
nkohari 16y ago
You're still outsourcing if you go with a managed dedicated hosting service, or even if you buy hardware and colocate it. Even if you owned the datacenter and the entire backbone, you're still banking on everyone else not fucking up their end of the connection.
- jwcacces 16y ago
  Yeah, but at least you can take direct action when your people fuck up.