4 ms·
Thankfully, no. :) But yeah, right now we're shutting everything down to try and avoid possible data corruption. Once they restore service, hopefully we'll be
by jedberg 16y ago
Thankfully, no. :)
But yeah, right now we're shutting everything down to try and avoid possible data corruption. Once they restore service, hopefully we'll be able to come back quickly.
- thirtysixred 16y agoSHUT. DOWN. EVERYTHING. Good plan though. :p
- retube 16y agoReminds me of that scene in Jurassic Park
- retube 16y agoGood lord. What's with all the DVs??
- retube 16y agoHa. HN really on it's high horse today.
- eogas 16y agoI know right? I made a hilarious joke a little while ago and jedberg yelled at me and everyone downvoted me. And I'll be very surprised if this comment doesn't get downvoted to hell too. EDIT: I also simply greeted jedberg, and a bunch of people thought that was a good reason to downvote. Do people think there's an imminent influx of redditors, and that they have to dissuade them from becoming HNers? I don't think that's the case. EDIT: Fuckin' called it.
- rmoriz 16y agoPresident Madagascar is doin' cloud biz, too?
- jedberg 16y agoMy original comment was "We're going Madagascar on the servers." Then I remembered I was on HN, not reddit. :)
- ILIKECAKE 16y agoHey man.. some of us would have got it :)
- davidreiss666 16y agoYou don't need to flee the country yet.
- JonnieCache 16y ago>Then I remembered I was on HN, not reddit. Right now we may as well be on reddit.
- pvh 16y agoHey Jedberg, if you guys aren't already rolling your own, check out fdr's WAL-E tool. It bounces postgres write-ahead logs off S3 and goes great with the new PG9 replication. https://github.com/heroku/WAL-E https://github.com/heroku/WAL-E
- ghotli 16y agoThanks for this. I had designed and partially implemented this exact same thing. Do you know of this running in production anywhere?
- stef25 16y agoAmazon is really not being kind to you guys; I sort of hope you'll find an alternative solution fast!
- acangiano 16y agoIf I was Rackspace, I'd be at Reddit/Wired's headquarters already.
- pbhjpbhj 16y agoDidn't Jedburg say that they could reduce the failure by spending with Amazon. I wonder if Rackspace really want this particular traffic burden. It seems that if Reddit choose not to pay for the load they need then you get lot's of bad press for it ... perhaps I'm seeing it wrong. Rubbish analogy: Kinda like if I was doing a haulage business and you called out for a wheelbarrow to carry some elephants, then when the barrows broke we got bad press despite. If you'd paid for a heavy animal transport package ... OK it's all going wrong, you get the idea.
- jedberg 16y agoNo, I said that we have spent all we can, and at this point we need development. However, in this case, the outage is not because of any issues with our setup, but with Amazon.
- pbhjpbhj 16y ago>"we have spent all we can" So is it a financial constraint with Amazon? Would you be suffering the same sorts of outages regardless of the technology on the backend or does AWS basically suck?
- rm445 16y agoYou have the wrong end of the stick, because you're missing the history of the story. Reddit have a weird budget when it comes to staffing costs versus operating costs due to their parent company's policies as a media comapny - so they have a decent budget but are massively understaffed. Statements like the one you're quoting are in that context. Let's say you have an unlimited operating budget - you can come up with all kinds of wonderful plans for massive redundancy and zero downtime. But you can't make that happen if you're not allowed to hire any engineers or sysadmins! As far as I'm aware reddit are paying Amazon mucho dinero but still having irredeemable problems with the storage product, EBS. They are stuck on an unreliable service without the manpower to move off. That's the story, as far as I can piece together from comments here and on reddit.
- deleted 16y ago[deleted]
- jedberg 16y agoI'm here all the time. :)
- hysterix 16y agoSup jedberg, I obviously don't have nearly the level of knowledge with the intricacies of reddit, but coming from a strictly "business" standpoint, the amount of downtime reddit receives due to amazon issues is astounding. Perhaps it's time to look for alternatives? Anyway, thanks for your time.
- blhack 16y agoWill you guys ever do something like this: http://cache-scale.appspot.com/c/www.reddit.com/ http://cache-scale.appspot.com/c/www.reddit.com/ but officially supported (and paid for) by you?