8 ms·
For those saying "Chaos Engineering", first off, the poster is well aware of Chaos Engineering. He's an AWS Hero and the founder of Tarsnap. Secondly, this wo
by jedberg 6y ago
For those saying "Chaos Engineering", first off, the poster is well aware of Chaos Engineering. He's an AWS Hero and the founder of Tarsnap.
Secondly, this would help make CE better. I actually asked Amazon for an API to do this ten years ago when I was working on Chaos Monkey.
I asked for an API to do a hard power off of an instance. To this day, you can only do a graceful power off. I want to know what happens when the instance just goes away.
I also asked for an API to slow down networking, set a random packet drop rate, EBS failures, etc. All of these things can be simulated with software, but it's still not exactly the same as when it happens outside the OS.
Basically I want an API where I can torture an EC2 instance to see what happens to it, for science!
- exdsq 6y agoDumb question here from a CE beginner but can’t you have a Docker image for that service and turn it off?
- paranoidrobot 6y agoOS-level 'turn off' options don't replicate what happens when you yank power on a rack of equipment. Pretty much every option you have from the OS will let caches flush, will have in-progress writes complete. Yank the power and none of that will happen. It'll let you actually see what level of fibbing your OS and hardware are telling you. Oh you got a result from that flush to disk? It completed? Are you sure? Really really sure? Lets find out...
- jrockway 6y agoThe main result of this testing is that you'll find bugs in all the abstraction layers that you can't control. You'll then be paranoid but unable to take any action. Have you ever seen the source code that's running on your SSD's CPU? Nope. And it's probably not a small program. My guess is that it works well when everything is OK, and fails catastrophically 0.0001% of the time when everything isn't OK. But you'll never know unless you try failing catastrophically a million times. Did the vendor even try failing catastrophically one million times before they started manufacturing (or at least sent the batch to retailers)? Did they do that, fail 1 time, and mark the bug closed as non-reproducible? I have no idea! Maybe everything is actually great. Or maybe someone else will be oncall the week you hit that one in a million failure case. Or maybe it will happen to your competitors instead of you! Without testing, all we have is hope.
- alexchamberlain 6y agoEven with testing, it’s still just hope; to quote Buzz Lightyear, it’s hoping with style.
- landryraccoon 6y agoI think this is too pessimistic. If all you do is test your error detection and backup recovery mechanism (of course you have backups right)? Then that’s an advantage. Let’s say a hard power off causes data corruption that can’t be fixed. For lots of applications downtime is better than corruption, so in that case you at least will be able to test when you should take the system down and recover from a known good backup.
- paranoidrobot 6y ago> You'll then be paranoid but unable to take any action. I disagree. While you can't control the bugs in the hardware or drivers or whatever, you can definitely ensure that your application detects corrupted data and can at least warn that something is wrong rather than starting back up and silently serving from corrupted data. It also ensures that a node suddenly disappearing without a graceful "I'm leaving" message, and potentially with open transactions or other operations can be handled by the clients and nodes that ARE still around.
- jrockway 6y agoYou've hit on one of my big pet peeves -- I am going to add redundancy and erasure codes at the application layer because the disk could catch on fire or be sucked up by a tornado, so why do I need a bunch of unaudited software adding extra redundancy and erasure codes behind my back? I get it for desktop users... they want to open some benchmark they downloaded and get a higher number than their friends (RAID-0-esque sharding data across individual flash chips), and they also don't have access to software that can add error correction codes (because the Windows installer can't recognize an SSD with 4 flash chips as 4 drives in a RAID array)... so the disk has to do it itself. But why is it a thing in datacenters? Just so you can install Linux+ext2+MySQL and call it a day? That seems crazy to me when better storage software exists.
- IgorPartola 6y agokill -9 on the VM process from the host?
- jedberg 6y agoIf you're using Docker, yes, you can get a lot more options in testing by poking "from the outside", but that still doesn't test what happens when your docker host just dies.
- dotancohen 6y agoThat's not a dumb question at all. I'm not the GP, but I assume it is because the GP would like to run his "production" setup (or even production setup itself) under such circumstances. So far as I remember, the origin of the idea came from the Chaos Monkey which would sabotage certain services in a production environment to ensure than everything was redundant and fail safe.
- deleted 6y ago[deleted]
- aloknnikhil 6y ago> I asked for an API to do a hard power off of an instance. To this day, you can only do a graceful power off. I want to know what happens when the instance just goes away. Wouldn't just running "halt -f" do the same?
- jedberg 6y agoIt would be really close yes, but not exactly the same as ripping the power cord out. It still gives the OS a hint that shutdown is coming.
- zyamada 6y agoPossibly, but how can we be 100% sure without the ability to compare behavior? If I were following this line of research I'd still want to know if there's any difference in the nature of a failure when it comes from within the OS (possibly simulated by half -f) and the situation the parent OP pointed out where the instance just goes poof without sending any kind of signal to the OS itself.
- BillinghamJ 6y agoThough of course then the trouble is, if AWS is simulating specific behaviour, it won't be exactly the same as real problems when they occur. It is a bit better, but hard to say how much better. I'd think the key on this is being able to simulate very specific partial-failure conditions. e.g. specific packet loss, loss of connections to EBS, etc. Just turning machines off I expect wouldn't be that valuable.
- jsjohnst 6y ago> Though of course then the trouble is, if AWS is simulating specific behaviour, it won't be exactly the same as real problems when they occur. If Amazon simulated power failure, or network cable disconnect, or potentially even corrupt writes to disk, I personally feel it would be indistinguishable from the event really happening.
- 6y ago
- hiyer 6y agoToday you can use spot block instances for this. They are guaranteed to die off after your chosen time block of 1-6 hours.
- jedberg 6y agoAs far as I know it still sends a graceful shutdown at the end of the time block. It sends exactly the same signal as if you use the shutdown API, which is the same as pressing the power button on the front of the machine.
- toredash 6y agoCorrect!
- recuter 6y agoWouldn't a simple cron that disconnects the networking on an instance randomly and/or pegs all the cores be equivalent?
- jedberg 6y agoThere are lots of ways to get a very close experience to a power pull, but nothing you can do from within the VM is quite the same as doing it outside the VM.
- recuter 6y agoI believe you but I'm curios as to the subtle difference. To my mind simply turning off the networking, physically cutting a cable, or hardware spontaneously combusting are not discernible events to an outside observer. What am I missing?
- thejj100100 6y agoTasks that are writing to disk, network outage wouldn't stop the task?
- simonebrunozzi 6y ago> Basically I want an API where I can torture an EC2 instance to see what happens to it, for science! And one day there will be PETA [0] for EC2 instances! [0]: https://www.peta.org/ https://www.peta.org/
- asdff 6y agoI wonder how much you would have to pay Amazon for them to send a tech down to the datacenter and pull the plug on your running node
- yjftsjthsd-h 6y agoIt's AWS; surely half of the work would be finding the server that happens to be running your instance, anonymously sitting in a datacenter with thousands of other servers. There's also the question of how many other customers' instances are located on the same hardware.
- pojzon 6y agoDid you try to setup an on premis euqalyptus cloud for that ? Eucalyptus has an api compliant with AWS api?
- jimnotgym 6y agoI really wish the world could agree on a better term than 'on premis' for 'not in the cloud'. At least 'on premises' is descriptive of the situation, although a bit clunky. On premises doesn't make any sense at all. 'Premis' is not the singular of 'premises', 'premises' is the singular of itself! I prefer 'on site', or 'self hosted' myself
- schoen 6y agoI think it's a plurale tantum https://en.wikipedia.org/wiki/Plurale_tantum https://en.wikipedia.org/wiki/Plurale_tantum That is, it seems to be grammatically plural but not have a distinctive singular. 4. pl. A piece of real estate; a building and its adjuncts; as, to lease premises; to trespass on another's premises. [1913 Webster]
- breatheoften 6y agoFascinating! The mere existence of such an api would be an interesting source of problems when used accidentally/via buggy code ... I wonder the degree to which this functionality would end up becoming relied on as a "in worst case hard kill things to recover" behavior that folks utilize for bad engineering reasons as opposed to good ones ...
- giancarlostoro 6y agoI would assume said code should never be part of actual deployments or only part of unit tests, maybe an external project even.
- agravier 6y agoOh you, sweet summer child...
- drjasonharrison 6y agoTotally. I have added wifi power outlets to installations because process restart, OS restart, and hardware restart did not get a system back up and running. Yes it was a startup, and yes it was remotely installed hardware.
- tacLog 6y agoMight be a bit strange as you are saying this is a example of bad planning but my employer was discussing this today. We had a hard time finding options, what did you end up deploying or did you develop your own in house?
- satya71 6y agoI think localstack[1] gets you lot of this. [1] https://localstack.cloud/ https://localstack.cloud/
- nbadg 6y agoI was just about to suggest localstack. I use it religiously in personal projects; can't recommend it enough. I haven't started telling it to induce errors yet, but it definitely has that capability. And if you're running it in docker, some of the network stuff can be simulated that way as well.
- MrBuddyCasino 6y agoToo bad its a resource hog though. Startup time is really bad, using it via Testcontainers takes patience.
- dmurray 6y ago> For those saying "Chaos Engineering", first off, the poster is well aware of Chaos Engineering. He's an AWS Hero and the founder of Tarsnap. Yeah, but did he win the Putnam?
- hitekker 6y agoFor reference: https://news.ycombinator.com/item?id=35079 https://news.ycombinator.com/item?id=35079
- Terretta 6y agoSame thread where Drew from "getdropbox.com" says he also has a "sync and backup done right" idea...
- OJFord 6y agoBrilliant! https://news.ycombinator.com/item?id=35103 https://news.ycombinator.com/item?id=35103
- swyx 6y agoi like how the guy recovered with "Just the once, though, huh?" 2007 HN was highly entertaining
- andyjohnson0 6y agoFurther context for those of us not in North America: "Putnam" refers to a undergrad mathematics competition. https://en.wikipedia.org/wiki/William_Lowell_Putnam_Mathematical_Competition https://en.wikipedia.org/wiki/William_Lowell_Putnam_Mathemat...
- jedberg 6y agoHe did, but that particular accolade didn't seem relevant. :)
- deleted 6y ago[deleted]
- peterwwillis 6y ago> Basically I want an API where I can torture an EC2 instance to see what happens to it, for science! # cat > dropme.sh <<'EOFILE' #!/bin/sh set -eu read -r SLEEP tmp=`mktemp` ; tmp6=`mktemp` iptables-save > $tmp ip6tables-save > $tmp6 for t in iptables ip6tables ; do for c in INPUT OUTPUT FORWARD ; do $t -P $c DROP ; done $t -t nat -F ; $t -t mangle -F ; $t -F ; $t -X done sleep "$SLEEP" iptables-restore < $tmp ip6tables-restore < $tmp6 rm -f $tmp $tmp6 EOFILE # chmod 755 dropme.sh # ncat -k -l -c ./dropme.sh $(ifconfig eth0 | grep 'inet ' | awk '{print $2}') 12345 & # echo "60" | ncat -v $(ifconfig eth0 | grep 'inet ' | awk '{print $2}') 12345 If you're lucky the existing connections won't even die, but the box will be offline for 60 seconds.
- blaser-waffle 6y agoNot a bad script, but it's basically going "shields up!", blocking all traffic, and then taking them down. Definitely useful, but won't test for the hard stuff, like if my DB was in the middle of something and then died, scrambling my data. What's that gonna look like on reboot?
- eru 6y agoNearly a decade ago I was working on Xen (or rather XenServer). We had lots of fun implementing these kinds of wonky devices for the virtual machines that would randomly drop network packets or fail to read the hark disk in arbitrarily bad ways.
- DonHopkins 6y agoI want a stress testing feature where I can accidentally misconfigure an instance or resource somehow, then when I get an unexpectedly $10,000 higher bill at the end of the month, I can declare that I was only testing, and I don't have to pay it!