4 ms·
Being on AWS is also easy to explain to customers about downtimes - AWS was down and customers are pretty understanding in that case and don't demand why you ar
by manojlds 7y ago
Being on AWS is also easy to explain to customers about downtimes - AWS was down and customers are pretty understanding in that case and don't demand why you aren't multi AZ etc ( of course YMMV based on sensitivity of your business)
- jen_h 7y agoYep. I call it the "AWS Chicken Pox Party." We got lucky this time, RDS, ELB, EC2 and Lightsail instances all in US-East-1 across multiple accounts and no issues (knock-on-wood and understand that we'll get it the next time). Especially happy as I had a two-day running neural net training job running and it's still going, that would have been depressing. Phew.
- kahnjw 7y agoWhy don't you just checkpoint the model every n steps? NNs fail for a myriad of reasons, you can easily reduce risk by routinely saving state.
- jen_h 7y agoAfter my last oom-party, I now have it checkpointing every 1000 steps (way too often, I think, but there's plenty of disk), but I just really really want it to complete a full run. ;)
- freehunter 7y agoKind of like the "no one ever got fired for recommending IBM", if you have significant downtime on Linode or Hetzner, people are going to ask "why weren't you on AWS?!". If you're on AWS and AWS goes down, you get to skip that question entirely. You were already using the logical choice, you don't need to defend anything. If Netflix can go down when AWS goes down, so can your app. AWS outages impact so much of the Internet that people will just accept it. Of course here I am with a site on Heroku which uses AWS... impacted by the AWS outage... fielding questions about why I didn't pick AWS if Heroku suffers outages like this. Can't please them all.