11 ms·

Trello has moved to AWS

124 points by gecko 14y ago

Killah911 14y ago
Would love to hear about what it took to transition to AWS. Any chance Joel would divulge some of the driving factors (reliability being one)? How about cost? Unlike my paltry webapps, trello gets a lot of activity and no current revenue stream, so I'd love to know more on what the cost difference is between running your own vs in the cloud. How long did it take to move over? Are the old servers being used as backups in the event Amazon has their very rare aws outage? I know that's a lot of questions, but heck it would be an article I'd love to read and something I believe would be very relevant to the community as well!
- thedufer 14y ago
  > Are the old servers being used as backups in the event Amazon has their very rare aws outage? AWS is essentially acting as a backup of the old servers, as the datacenter is in danger of losing power. That said, the cost, reliability, etc. had been carefully thought out, albeit as a somewhat longer-term project than it ended up being. More info will certainly wait at least until the current state of things is cleared up - Fogbugz and Kiln are still in the datacenter.
  - Killah911 14y ago
    Thanks for the info. I know things must be pretty crazy right now. Quite literally in the trenches. Just goes to show what an amazing team behind an amazing product can do. Kudos!
rurounijones 14y ago
Be interesting to see how this was migrated, method, what had to be changed, lessons learned etc. Would also be great to see a follow-on report after a month or so of lessons learned from hosting on AWS vs own infrastructure. I will also be keeping an eye out to see if it remains as responsive.
- Killah911 14y ago
  What would be really interesting is if they thought of a Hybrid-Cloud approach. I recall Joel talking about adding a new server every week due to Trello's ever growing popularity during his presentation at Startup School. Being able to spool up on-demand should really help with performance or for times when the datacenter is flooded. But wouldn't it be more cost effective to run it on your hardware during off-peak times and spool up some new servers on AWS when performance starts to lag? Disclaimer: I'm a Trello fanboy, it's like having container classes in STL, only more visual!
  - gecko 14y ago
    Technically, it is running hybrid now, albeit for somewhat silly reasons: the mail server is still at Peer1.
    - trimbo 14y ago
      Yeah it's hard to migrate email given the anti-spam measures ISPs use like IP reputation. Good luck. Maybe a quick migration to Sendgrid for that system? (Edit: or Amazon SES, of course)
gecko 14y ago
There's a little bit more information in the last status post, plus a picture of Fog Creek cofounder Michael Pryor hauling 5-gallon jugs of diesel up 17 flights of stairs. (We have other employees down there, too, and I know Stack Exchange's NYC employees were excited to help out, so they're probably there by now, too.) http://status.fogcreek.com/2012/10/diesel-bucket-brigade-maintains-services.html http://status.fogcreek.com/2012/10/diesel-bucket-brigade-mai...
- ivany 14y ago
  Are there any safety issues associated with hauling a bucket of diesel up a staircase? I know it's not as volatile as gasoline but it's still not exactly the most fire-safe substance. Are they literally 5-gallon pails, or are they sealed containers?
  - gecko 14y ago
    Are there any safety issues associated with hauling a bucket of diesel up a staircase? It's flammable and slippery. Wear boots, and I suggest this is a good day to quit smoking. I know it's not as volatile as gasoline but it's still not exactly the most fire-safe substance. Are they literally 5-gallon pails, or are they sealed containers? They are open buckets we store the cleaning supplies for the fish tank in. And also now gasoline.
    - meaty 14y ago
      A million times this. Diesel is extremely dangerous stuff. I nearly blew myself up a few years ago by not cleaning out a diesel tank with detergent properly that I was welding. Diesel residue vapourised and went boom. Fortunately there wasn't enough in it to create a large explosion but it required new underpants. To add to that, my only car accident was due to having diesel on my shoes. My foot slipped off the brake and I rear ended some poor guy.
      - js2 14y ago
        After Hurricane Andrew, my father and I needed to siphon gasoline from one of our cars to use in our generator. Lacking a proper container, we siphoned the gas into a 5 gallon plastic water cooler bottle. The gasoline promptly ate through the bottle's seam and spilled out, dissolving a good size chunk of our asphalt driveway. Lesson learned: use only approved storage containers for gasoline.
citricsquid 14y ago
Not entirely related, but regarding the backup fuel situation: how long does 1 "bucket" of fuel keep the power going for and what is the cost of 1 bucket of fuel?
- gecko 14y ago
  This information is from earlier today, so I don't know how accurate it is, but: - They burn 44 gallons per hour - The tank adjacent to the generator holds ~400 gallons - Gas is currently VERY expensive, but I don't have a real number for you at the moment.
  - jspthrowaway 14y ago
    Diesel, on-highway in New England, is averaging $4.205/gal[1] (probably more in the city, especially more in a disaster situation). Applying your information to the above question and assuming they can get just shy of five gallons per bucket, it's probably a fair estimate that each bucket carries more than $20 of diesel fuel and accounts for seven minutes of generator time. So, about $3 per minute or 5.1 cents every second. Really puts things in perspective. [1]: http://www.eia.gov/petroleum/gasdiesel/ http://www.eia.gov/petroleum/gasdiesel/
    - lmm 14y ago
      I got a similar feeling at my last^2 job when we had a client run a superbowl ad, for which they were paying a million dollars. That works out as something like $30/millisecond.
    - lsllc 14y ago
      FYI: #2 heating oil and diesel are pretty much the same thing, the only differences being dyes in the heating oil (since it's often taxed differently) and additives in diesel (for example in the winter to stop it waxing up in the cold). Not sure about sulfur content differences when comparing #2 to the low sulfur diesels you see, but I doubt a few hours of burning higher sulfur fuel will do anything. Also, #2 heating oil is cheaper than diesel in New England, it's about $3.70/gal.
rdl 14y ago
"Trello is in the process of being moved to Amazon AWS where it will not be affected by further data center issues." I guess that's called the power of positive thinking?
- sturadnidge 14y ago
  Oh, the irony. Interesting they didn't move to Azure though.
  - dagw 14y ago
    Unlike their other products trello isn't .Net based. It runs on a node.js/mongoDB/Redis (and I'm guessing Linux) stack . My understanding was that they're using Trello as a test platform for playing with and evaluating different new technologies.
    - smiler 14y ago
      You can run Linux VMs on Azure.... http://www.windowsazure.com/en-us/pricing/calculator/?scenario=virtual-machines http://www.windowsazure.com/en-us/pricing/calculator/?scenar...
      - bdavisx 14y ago
        "Let us redefine progress to mean that just because we can do a thing, it does not necessarily mean we must do that thing."
- joezydeco 14y ago
  ..or the power of a really long night with very sore muscles.
- atesti 14y ago
  So they kind of move some business away from Peer 1 after all this bucket brigade job? Not nice. I hope they move Trello back to "real" servers and not this AWS, once the datacenter is stabilized. I think that AWS is overrated and overpriced and it's a big asset of FogCreek to still use real data centers, after all they started this thing before AWS even existed. I don't know if StackExchange or FogBugz would run so good on these AWS-servers with bad IO. It's great though if they now have the chance to quickly overflow Trello to AWS in case of problems or high demands and buy a new correctly-priced, powerful server to put at Peer 1 a week later!
modarts 14y ago
Unrelated to the post, but I hope you guys are all doing okay out there. I have a huge amount of respect and appreciation for the level of commitment you have for your customers, despite these horrific circumstances. Stay safe!
dancryer 14y ago
Did they move to AWS to ensure that their services go down more efficiently next time there's a hint of a natural disaster?
- zwily 14y ago
  You do know that AWS did not go down during Sandy, right? /me knocks on wood
  - Xylakant 14y ago
    Well, but it's not like AWS didn't have its share of downtime in the last couple of weeks. Back then the recommendation of the arm-chair-admins was to move to dedicated hardware you own. Oh the irony.
    - pyre 14y ago
      Not necessarily. If they only have their setup in a single AWS datacenter, and it gets hit by a hurricane, then you still have issues. The real issue is geographic redundancy, regardless of using AWS or dedicated hardware.
      - Xylakant 14y ago
        With the last round of failures some of the people affected reported that they couldn't start up instances in other datacenters since those were overloaded. Still, I agree - the underlying problem is cross-datacenter redundancy but that's a very difficult problem to tackle and might just not be economically feasible.
Tyrannosaurs 14y ago
I'd be interested to know whether there has been a significant surge in demand for AWS since Sandy hit. Having the cloud as your back up makes a lot of sense but if everyone did it could it cope with the additional demand? Sure the AWS infrastructure is massive but I'm guessing the load that would be dumped on it if a significant chunk of New York's web hosting was suddenly moved across would be massive.
mgamer 14y ago
I wonder, how technically can you move something like Trello to AWS in such short period of time? Do you think they had a plan prepared for such contingency?
ludovicurbain 14y ago
And there goes my respect for fogCreek. Moving to AWS because of an outage sounds very dumb to me, considering AWS itself is quite unreliable and strangely expensive (expenses that could be directed towards a better solution). I guess they simply decided it wasn't their job, still... what a fail solution
mgkimsal 14y ago
Haven't we seen more AWS outage issues than "east coast shutdown"-level storms over the past, oh, say, 5 years? EDIT: Thinking now this may be a temporary move, not a permanent one. As such, it probably makes sense to have a tested process to be able to move between cloud/dedicated/whatever as quickly and painlessly as possible.
- acdha 14y ago
  We've seen many people who failed to read AWS' documentation and put all of their services in one data center. If you follow Amazon's guidelines and used multiple AZs (i.e. if you use RDS check that box), you've had very little (~20 minutes) downtime; if you went multi-region it's even less. It's not like this is a well-hidden secret - head over to AWS's whitepaper section http://media.amazonwebservices.com/AWS_Cloud_Best_Practices.pdf http://media.amazonwebservices.com/AWS_Cloud_Best_Practices.... “Be a pessimist when designing architectures in the cloud” http://media.amazonwebservices.com/AWS_Web_Hosting_Best_Practices.pdf http://media.amazonwebservices.com/AWS_Web_Hosting_Best_Prac... “As the AWS web hosting architecture diagram in this paper shows, we recommend that you deploy EC2 hosts across multiple Availability Zones to make your web application more fault-tolerant.”
  - mgkimsal 14y ago
    But... didn't we also have some comments before about how Amazon's own control panel systems are all in one data center? If Amazon themselves don't 'get this right', perhaps it's because it's too hard, and they need to take steps to make this easier (or indeed automatic, and charge a premium)?
    - jimwalsh 14y ago
      Those comments were in the wrong, and they were posted because parts of the control panel stopped working during the last outage. This was actually due to the throttling of the API, as mentioned in their write up. AWS is resilient enough so long as you dont park all of your environment in the same AZ/Region.
  - 14y ago
amalag 14y ago
The real question is "is it permanent?"

124 points by gecko 14y ago

Would love to hear about what it took to transition to AWS. Any chance Joel would divulge some of the driving factors (reliability being one)? How about cost? Unlike my paltry webapps, trello gets a lot of activity and no current revenue stream, so I'd love to know more on what the cost difference is between running your own vs in the cloud. How long did it take to move over? Are the old servers being used as backups in the event Amazon has their very rare aws outage? I know that's a lot of questions, but heck it would be an article I'd love to read and something I believe would be very relevant to the community as well!

thedufer 14y ago
> Are the old servers being used as backups in the event Amazon has their very rare aws outage? AWS is essentially acting as a backup of the old servers, as the datacenter is in danger of losing power. That said, the cost, reliability, etc. had been carefully thought out, albeit as a somewhat longer-term project than it ended up being. More info will certainly wait at least until the current state of things is cleared up - Fogbugz and Kiln are still in the datacenter.
- Killah911 14y ago
  Thanks for the info. I know things must be pretty crazy right now. Quite literally in the trenches. Just goes to show what an amazing team behind an amazing product can do. Kudos!

rurounijones 14y ago

Be interesting to see how this was migrated, method, what had to be changed, lessons learned etc. Would also be great to see a follow-on report after a month or so of lessons learned from hosting on AWS vs own infrastructure. I will also be keeping an eye out to see if it remains as responsive.

Killah911 14y ago
What would be really interesting is if they thought of a Hybrid-Cloud approach. I recall Joel talking about adding a new server every week due to Trello's ever growing popularity during his presentation at Startup School. Being able to spool up on-demand should really help with performance or for times when the datacenter is flooded. But wouldn't it be more cost effective to run it on your hardware during off-peak times and spool up some new servers on AWS when performance starts to lag? Disclaimer: I'm a Trello fanboy, it's like having container classes in STL, only more visual!
- gecko 14y ago
  Technically, it is running hybrid now, albeit for somewhat silly reasons: the mail server is still at Peer1.
  - trimbo 14y ago
    Yeah it's hard to migrate email given the anti-spam measures ISPs use like IP reputation. Good luck. Maybe a quick migration to Sendgrid for that system? (Edit: or Amazon SES, of course)

gecko 14y ago

There's a little bit more information in the last status post, plus a picture of Fog Creek cofounder Michael Pryor hauling 5-gallon jugs of diesel up 17 flights of stairs. (We have other employees down there, too, and I know Stack Exchange's NYC employees were excited to help out, so they're probably there by now, too.) http://status.fogcreek.com/2012/10/diesel-bucket-brigade-maintains-services.html http://status.fogcreek.com/2012/10/diesel-bucket-brigade-mai...

ivany 14y ago
Are there any safety issues associated with hauling a bucket of diesel up a staircase? I know it's not as volatile as gasoline but it's still not exactly the most fire-safe substance. Are they literally 5-gallon pails, or are they sealed containers?
- gecko 14y ago
  Are there any safety issues associated with hauling a bucket of diesel up a staircase? It's flammable and slippery. Wear boots, and I suggest this is a good day to quit smoking. I know it's not as volatile as gasoline but it's still not exactly the most fire-safe substance. Are they literally 5-gallon pails, or are they sealed containers? They are open buckets we store the cleaning supplies for the fish tank in. And also now gasoline.
  - meaty 14y ago
    A million times this. Diesel is extremely dangerous stuff. I nearly blew myself up a few years ago by not cleaning out a diesel tank with detergent properly that I was welding. Diesel residue vapourised and went boom. Fortunately there wasn't enough in it to create a large explosion but it required new underpants. To add to that, my only car accident was due to having diesel on my shoes. My foot slipped off the brake and I rear ended some poor guy.
    - js2 14y ago
      After Hurricane Andrew, my father and I needed to siphon gasoline from one of our cars to use in our generator. Lacking a proper container, we siphoned the gas into a 5 gallon plastic water cooler bottle. The gasoline promptly ate through the bottle's seam and spilled out, dissolving a good size chunk of our asphalt driveway. Lesson learned: use only approved storage containers for gasoline.

citricsquid 14y ago

Not entirely related, but regarding the backup fuel situation: how long does 1 "bucket" of fuel keep the power going for and what is the cost of 1 bucket of fuel?

gecko 14y ago
This information is from earlier today, so I don't know how accurate it is, but: - They burn 44 gallons per hour - The tank adjacent to the generator holds ~400 gallons - Gas is currently VERY expensive, but I don't have a real number for you at the moment.
- jspthrowaway 14y ago
  Diesel, on-highway in New England, is averaging $4.205/gal[1] (probably more in the city, especially more in a disaster situation). Applying your information to the above question and assuming they can get just shy of five gallons per bucket, it's probably a fair estimate that each bucket carries more than $20 of diesel fuel and accounts for seven minutes of generator time. So, about $3 per minute or 5.1 cents every second. Really puts things in perspective. [1]: http://www.eia.gov/petroleum/gasdiesel/ http://www.eia.gov/petroleum/gasdiesel/
  - lmm 14y ago
    I got a similar feeling at my last^2 job when we had a client run a superbowl ad, for which they were paying a million dollars. That works out as something like $30/millisecond.
  - lsllc 14y ago
    FYI: #2 heating oil and diesel are pretty much the same thing, the only differences being dyes in the heating oil (since it's often taxed differently) and additives in diesel (for example in the winter to stop it waxing up in the cold). Not sure about sulfur content differences when comparing #2 to the low sulfur diesels you see, but I doubt a few hours of burning higher sulfur fuel will do anything. Also, #2 heating oil is cheaper than diesel in New England, it's about $3.70/gal.

rdl 14y ago

"Trello is in the process of being moved to Amazon AWS where it will not be affected by further data center issues." I guess that's called the power of positive thinking?

sturadnidge 14y ago
Oh, the irony. Interesting they didn't move to Azure though.
- dagw 14y ago
  Unlike their other products trello isn't .Net based. It runs on a node.js/mongoDB/Redis (and I'm guessing Linux) stack . My understanding was that they're using Trello as a test platform for playing with and evaluating different new technologies.
  - smiler 14y ago
    You can run Linux VMs on Azure.... http://www.windowsazure.com/en-us/pricing/calculator/?scenario=virtual-machines http://www.windowsazure.com/en-us/pricing/calculator/?scenar...
    - bdavisx 14y ago
      "Let us redefine progress to mean that just because we can do a thing, it does not necessarily mean we must do that thing."
joezydeco 14y ago
..or the power of a really long night with very sore muscles.
atesti 14y ago
So they kind of move some business away from Peer 1 after all this bucket brigade job? Not nice. I hope they move Trello back to "real" servers and not this AWS, once the datacenter is stabilized. I think that AWS is overrated and overpriced and it's a big asset of FogCreek to still use real data centers, after all they started this thing before AWS even existed. I don't know if StackExchange or FogBugz would run so good on these AWS-servers with bad IO. It's great though if they now have the chance to quickly overflow Trello to AWS in case of problems or high demands and buy a new correctly-priced, powerful server to put at Peer 1 a week later!

modarts 14y ago

Unrelated to the post, but I hope you guys are all doing okay out there. I have a huge amount of respect and appreciation for the level of commitment you have for your customers, despite these horrific circumstances. Stay safe!

dancryer 14y ago

Did they move to AWS to ensure that their services go down more efficiently next time there's a hint of a natural disaster?

zwily 14y ago
You do know that AWS did not go down during Sandy, right? /me knocks on wood
- Xylakant 14y ago
  Well, but it's not like AWS didn't have its share of downtime in the last couple of weeks. Back then the recommendation of the arm-chair-admins was to move to dedicated hardware you own. Oh the irony.
  - pyre 14y ago
    Not necessarily. If they only have their setup in a single AWS datacenter, and it gets hit by a hurricane, then you still have issues. The real issue is geographic redundancy, regardless of using AWS or dedicated hardware.
    - Xylakant 14y ago
      With the last round of failures some of the people affected reported that they couldn't start up instances in other datacenters since those were overloaded. Still, I agree - the underlying problem is cross-datacenter redundancy but that's a very difficult problem to tackle and might just not be economically feasible.

Tyrannosaurs 14y ago

I'd be interested to know whether there has been a significant surge in demand for AWS since Sandy hit. Having the cloud as your back up makes a lot of sense but if everyone did it could it cope with the additional demand? Sure the AWS infrastructure is massive but I'm guessing the load that would be dumped on it if a significant chunk of New York's web hosting was suddenly moved across would be massive.

mgamer 14y ago

I wonder, how technically can you move something like Trello to AWS in such short period of time? Do you think they had a plan prepared for such contingency?

ludovicurbain 14y ago

And there goes my respect for fogCreek. Moving to AWS because of an outage sounds very dumb to me, considering AWS itself is quite unreliable and strangely expensive (expenses that could be directed towards a better solution). I guess they simply decided it wasn't their job, still... what a fail solution

mgkimsal 14y ago

Haven't we seen more AWS outage issues than "east coast shutdown"-level storms over the past, oh, say, 5 years? EDIT: Thinking now this may be a temporary move, not a permanent one. As such, it probably makes sense to have a tested process to be able to move between cloud/dedicated/whatever as quickly and painlessly as possible.

acdha 14y ago
We've seen many people who failed to read AWS' documentation and put all of their services in one data center. If you follow Amazon's guidelines and used multiple AZs (i.e. if you use RDS check that box), you've had very little (~20 minutes) downtime; if you went multi-region it's even less. It's not like this is a well-hidden secret - head over to AWS's whitepaper section http://media.amazonwebservices.com/AWS_Cloud_Best_Practices.pdf http://media.amazonwebservices.com/AWS_Cloud_Best_Practices.... “Be a pessimist when designing architectures in the cloud” http://media.amazonwebservices.com/AWS_Web_Hosting_Best_Practices.pdf http://media.amazonwebservices.com/AWS_Web_Hosting_Best_Prac... “As the AWS web hosting architecture diagram in this paper shows, we recommend that you deploy EC2 hosts across multiple Availability Zones to make your web application more fault-tolerant.”
- mgkimsal 14y ago
  But... didn't we also have some comments before about how Amazon's own control panel systems are all in one data center? If Amazon themselves don't 'get this right', perhaps it's because it's too hard, and they need to take steps to make this easier (or indeed automatic, and charge a premium)?
  - jimwalsh 14y ago
    Those comments were in the wrong, and they were posted because parts of the control panel stopped working during the last outage. This was actually due to the throttling of the API, as mentioned in their write up. AWS is resilient enough so long as you dont park all of your environment in the same AZ/Region.
- 14y ago

amalag 14y ago

The real question is "is it permanent?"