4 ms·
Let me update this: we've been working all day on moving Trello to AWS, a project which is now underway. If all goes well (fingers crossed, etc) we'll be up and
by spolsky 14y ago
Let me update this: we've been working all day on moving Trello to AWS, a project which is now underway. If all goes well (fingers crossed, etc) we'll be up and running on AWS very soon. There are still other services in the flooded New York data center (FogBugz, Kiln, Copilot) and we are making plans to physically move servers to a new data center as quickly as possible if the flooded data center falls over, but a physical move of 5 racks could take a whole day so I hope that doesn't happen.
Separately, Stack Exchange, which is in the same data center, is running off of its hot backup in Oregon.
We would have liked to have completely redundant data centers for FogBugz on Demand and Kiln on Demand to avoid even a few hours of downtime, but because those services rely heavily on giving every customer their own SQL database, there is almost no reasonable way to get fast failover to a different data center. We can do it with Stack Exchange because there are only a couple of hundred databases. We've been building a SAN solution which will make it possible to hot swap out to another datacenter, one day, but that project is not complete.
- deleted 14y ago[deleted]
- drewcrawford 14y ago> but because those services rely heavily on giving every customer their own SQL database, there is almost no reasonable way to get fast failover to a different data center Why did you say you could do fast failover to LA in 2007? [1] It was the case then that every customer had their own SQL database too. I'm not trying to be snarky, your team is obviously putting forth some heroic effort. I'm just an affected customer that is curious why the question seems to have two mutually exclusive answers. [1] http://webcache.googleusercontent.com/search?q=cache:lHEK939AKiEJ:www.joelonsoftware.com/items/2007/07/09.html+&cd=5&hl=en&ct=clnk&gl=us&client=safari http://webcache.googleusercontent.com/search?q=cache:lHEK939...
- guiambros 14y agoI'd be curious to see if there's any update to FBOD architecture. The idea of using self-hosted servers running SQL Server and IIS already sounded eccentric 5 years ago, but given the number of options today, it'd be hard to still justify such a monolithic solution. In all fairness, Joel recognized in the article that the initial decision was based on their previous experience with Microsoft -- and AWS was still incipient in 2007 -- so it made sense at the time. I doubt this is still the case. Even less if you consider the costs of Sandy - duplicated servers, migration, hours lost, unhappy clients, etc.
- jmillikin 14y agoWhy would you have needed fast failover in this case? Sandy's trajectory was predictable enough that even a day-long migration could have been completed with time to spare.
- patrickgzill 14y agoISCSI over DRBD for Linux, seems to be one solution. I know of a highly available VOIP company that uses this in production. And maybe this sounds snarky, but AWS is your "making it better" solution? Seriously?
- notatoad 14y agoIf aws is up and their DC is down, in what way is that not better?
- patrickgzill 14y agoSorry for not being clear - I was referring to AWS' not so stellar uptime.
- donavanm 14y agoExcept for this event. Where pretty much every DC in NY/NJ is compromised, yet no blips in us-east.
- derobert 14y agoProbably because Amazon US-East is in Northern VA, not NY/NJ. http://aws.amazon.com/about-aws/globalinfrastructure/ http://aws.amazon.com/about-aws/globalinfrastructure/ NoVA didn't get hit that hard by Sandy, unlike NY/NJ. So data centers around here didn't tend to have problems, the few that I have equipment in didn't even loose utility power.