9 ms·
AWS Fault Injection Simulator
- jedberg 6y agoThis is exciting! I've been asking for this for years. There is only so much you can do by implementing your chaos on instance. Having access to a lower level of infrastructure will be huge! Especially in a world of serverless services where we can't implement on instance nearly as easily.
- hrpnk 6y agoAWS provided recently a library [1] for running chaos experiments [2], which look like an evolution of the work their own employee did before [3]. I wonder if the FIS is just a re-packaging and proper integration of these or something more. [1] https://github.com/amzn/awsssmchaosrunner https://github.com/amzn/awsssmchaosrunner [2] https://aws.amazon.com/blogs/opensource/building-resilient-services-at-prime-video-with-chaos-engineering/ https://aws.amazon.com/blogs/opensource/building-resilient-s... [3] https://github.com/adhorn/chaos-ssm-documents https://github.com/adhorn/chaos-ssm-documents
- discodave 6y agoChaos Runner, or anything else that you run on your EC2 instances (or in your ECS containers) requires you to install some kind of agent or daemon. This is non-optimal as it is work in and of itself, it is also hard to correctly simulate things like the network, or various AWS services being down/unreachable. If you run the agent/daemon on your production stack, then it's a potential vector for misconfiguration or attack. But if you don't run the agent/daemon in production, then it's another way in which your test stack diverges from production! I saw various PR/FAQs related to Chaos engineering while I worked in both EC2 and the AWS developer tools org. I've been gone over a year now, but I would bet that FIS does something at the EC2 Network level so that you don't have to install stuff on your instances or containers.
- gadders 6y agoJust let me setup your cloud instances. I guarantee loads of random failures.
- atonse 6y agoInattention-to-detail as a service? I love it! Hire me to do a half-assed job so your infrastructure can randomly fail to make it more resilient.
- gadders 6y agoYeah, I'm applying to YC next year. Let me know if you want in as a co-founder.
- dijksterhuis 6y agoI'm game of you need a hwlping hand,
- atonse 6y agoI would let you know but I can’t be bothered to write an official email. Inattention to detail and all.
- bithavoc 6y agos/us-east-1/Fault Injection Simulator/g
- dgrin91 6y agoThis was the good laugh I needed today. Thank you.
- brodouevencode 6y agoHad the same thought but with kinesis
- jedberg 6y agoAs funny as this is, Chaos Engineering is not just random failures. It's causing known failures to see what happens. They may be a surprise to the service or even the engineers running the service, but someone should know what's happening so it can be turned off and so things can be monitored and recorded and hypotheses can be proposed and tested.
- stingraycharles 6y agoDoesn’t that kind of defeat the point of chaos engineering, that is, making the process unsupervised and triggering failure scenarios that were not yet known? I always considered the simplest form of chaos engineering to be a process that randomly kills processes on a server, and see what happens. If, on the other hand, you mean that it’s important that the “chaos” aspects needs and on/off button and that someone needs to be managing that, then I agree. :)
- jedberg 6y agoYeah, that second thing. :) The chaos itself should be random, but the time and place should not be.
- oblio 6y agoNow that there are other regions there, is there any reason, except for legacy, to use us-east-1? New features are generally ported quite fast to the other regions, so this doesn't seem like a valid reason in 2020.
- SilasX 6y agoctrl+f ... Nope, no mention of Netflix’s Chaos Monkey. https://en.wikipedia.org/wiki/Chaos_engineering#Chaos_Monkey https://en.wikipedia.org/wiki/Chaos_engineering#Chaos_Monkey
- jon-wood 6y agoHowever the very first sentence is "AWS Fault Injection Simulator is a fully managed chaos engineering service".
- SilasX 6y agoI know -- still no mention of the state of the art or what product it might replace.
- nolaspring 6y agoIn Dr Vogel's keynote presentation he brought up Netflix and Chaos Monkey specifically as precursors to this service.
- SilasX 6y agoIt would be better if that context carried over to this document.
- laumars 6y agoThat would be superfluous to the content in which most people would refer to that document for. Amazon already state it’s for Chaos Engineering, they don’t need to list other similar projects on their own product page.
- gitbisect 6y agoWhy would it? AWS was doing Chaos Engineering long before it was even called Chaos Engineering and well before Chaos Monkey at Netflix. Most historical coverings of Chaos Engineering trace back to Jesse Robbins at AWS pulling power cables from racks to simulate failure and running GameDays in the early 2000s. See: https://en.wikipedia.org/wiki/Jesse_Robbins#Contributions_to_web_operations_and_devops https://en.wikipedia.org/wiki/Jesse_Robbins#Contributions_to... https://medium.com/the-cloud-architect/chaos-engineering-ab0cc9fbd12a https://medium.com/the-cloud-architect/chaos-engineering-ab0...
- qz2 6y agoI need this now.
- deeblering4 6y agoLooks cool, but maybe wait until after the new year before blowing up production on the team :)
- agentwalker 6y agoLooks interesting - what stands out to you?
- arafsheikh 6y agoCompute + DB should cover a lot of use cases, but it’ll be great to have support for more AWS services. DNS (Route53) issues are a pain to deal with for example.
- tpetry 6y agoWhat problems does route53 create? Is the api not available? Or the dns servers returning ko data?
- twothamendment 6y agoA route 53 propagation issue caused a bunch of our web servers to be unable to connect to the read replicas of our RDS cluster. The cluster scaled in a RR, but the DNS change took hours to propagate. The web servers kept trying to connect to the replica that didn't exist. Our workaround was to put the IP address in the hosts files - not an ideal setup, but it got the job done.
- pbw 6y agoDoes it mention pricing? I couldn't find it.
- jrott 6y agoI didn't see it either. Hoping it's semi-reasonable because I'd like to be able to really use it.
- the_arun 6y agoMe neither. I would expect AWS to provide this service for free. It is like paying a price to design our systems to ensure they are reliable even if a zone/region goes down managed by AWS. Well - they might call this shared responsibility :)
- jacques_chester 6y ago> Me neither. I would expect AWS to provide this service for free. A truly cunning pricing scheme would be to strike a Coasian Bargain: "we inject faults for free! If you want us to stop, you'll need to pay extra". That would mean the cost would fall mostly on those who are unable to avoid paying it.
- enw 6y agoThe cost would fall one those who are unable to design for fault-tolerance.
- jacques_chester 6y agoOr who are unable to, due to existing constraints. The charitable name for this lucrative segment of the market is "Enterprise".
- jedberg 6y agoThey never talk about pricing until a service launches.
- fnord77 6y agowhat is with the amazon spam today? are their SDRs trying to boost their numbers for the year?
- DavidSJ 6y agoAmazon frequently releases a bunch of new cloud services all on the same day.
- karlding 6y agoIt's AWS re:Invent [0], where they typically announce new product offerings. [0] https://reinvent.awsevents.com/ https://reinvent.awsevents.com/
- fnord77 6y agodidn't realize this. and I went to re:invent last year!
- frakkingcylons 6y agoA new service announcement isn’t spam.
- sa46 6y agoAre there similar offerings for other clouds?
- aynsof 6y agoGremlin offers a chaos engineering service which I believe can be used across multiple clouds.
- agilob 6y agoJust use Oracle...
- chii 6y agoyou want to simulate faults, not have actual faults...
- csears 6y agoAre there any tools that simulate control plane failures for AWS or other cloud providers? This looks great, but it seems to only work on compute resources. Many AWS outages result in APIs failing or returning errors, which should be possible to simulate. I'd like to create experiments where EC2 instances won't spin up in a particular AZ, EBS disks can't be reattached, Route53 zones can't be updated, IAM changes don't take effect, etc.
- aynsof 6y agoThere's no tool that I'm aware of, but you could implement hard failures using SCPs. That will cause an error when the blocked API call is called. It won't, however, give you latency or any nuance in failure rates.
- juliend2 6y agoLooks like that guy's tweet was heard: https://twitter.com/cperciva/status/1292260921893457920 https://twitter.com/cperciva/status/1292260921893457920 But AWS made it more configurable, and not in a single region.