11 ms·
30 Lines of Code Blew Up a 27-Ton Generator
- Nextgrid 6y agoI wonder why they needed to actually destroy a relatively expensive piece of equipment, plus spend time & money on all the orchestration around this test. They clearly knew they were capable of reprogramming the control logic remotely (otherwise this idea for a test would've gone nowhere) and any engineer could've predicted this obvious outcome without the need for a real-world test.
- daly 6y agoBecause business people are not engineers. Telling a business person they have a software problem is meaningless. Showing them losing a LOT of money because they ignored it gets attention. Ask any hospital about ransomware. Our local hospital is still running Windows 7 and older versions. Everybody loves network connected new cars. Nobody thinks about the fact that they are "fly by wire", using software to control everything. Who thinks about changing the battery controller to cause a battery fire? Or controlling the acceleration to maximum? Or setting off the airbags? Or... Use your imagination. What could go wrong in a 2 ton computer on wheels? Especially one that has a camera and software that can recognize "soft targets" (i.e. people). It really should be illegal to have network connected cars. Any software security engineer knows that. But you can't convince a business person.
- onion2k 6y agoNobody thinks about the fact that they are "fly by wire", using software to control everything. Who thinks about changing the battery controller to cause a battery fire? Or controlling the acceleration to maximum? Or setting off the airbags? Or... Automotive engineers think about those things. When we drive our cars we're trusting them to have thought about those things, just as we're trusting that the engineer who designed the wheel nuts did his work well enough to stop the wheels coming off, or the engineer who designed the steering linkage considered how we might need to go round corners reliably.
- deleted 6y ago[deleted]
- MaxBarraclough 6y ago> Automotive engineers think about those things Not always. Hacking a vehicle's brakes is no longer the stuff of sci-fi, it's already happened in a poorly-designed Internet-enabled jeep, see [0]. (As an aside, I have no idea why they thought it was acceptable to do their proof-of-concept demonstration while driving at speed on public roads.) [0] https://www.wired.com/2015/07/hackers-remotely-kill-jeep-highway/ https://www.wired.com/2015/07/hackers-remotely-kill-jeep-hig...
- macintux 6y agoThat story made me angry. There’s no way they could have known precisely what would happen when they executed that hack; software has bugs (the whole point of these exercises) and complex interactions between systems cause all sorts of weird behavior.
- leetcrew 6y agosetting aside unforeseen interactions between complex systems, the article seems to say that they deliberately disabled the brakes on a public road, while the jeep was moving! that goes well past a fun prank for a story. irresponsible on the journalist's part too to consent to this. the point could have been made just as well on a private track.
- userbinator 6y ago(As an aside, I have no idea why they thought it was acceptable to do their proof-of-concept demonstration while driving at speed on public roads.) To cause even more outrage.
- Silhouette 6y agoAutomotive engineers think about those things. If that were a reliable safeguard, we wouldn't already have several examples about the behaviour of connected vehicles being compromised remotely in one way or another. Since unfortunately we do have such examples, evidently whatever is going on within the engineering groups at the auto manufacturers is not sufficient to ensure our safety and security in relation to these new generations of connected vehicles.
- DoingIsLearning 6y ago> It really should be illegal to have network connected cars. Any software security engineer knows that. Everybody loves network connected new cars. Nobody thinks about the fact that they are "fly by wire" I fully agree that automotive and medical equipment/IT could greatly benefit from not having security as an afterthought. However, I don't know any vehicle with connectivity (other than that Jeep Cherokee controversy), which does not have safety critical CAN/FlexRay buses segregated from user facing 'infotainment' systems. What that means is that the network bus in which your 'compromised' infotainment system is able to operate is completely separate from Engine, ABS, AEB, ESP, Airbags etc. The solutions vary but there is usually a physical gateway that prevents a passthrough MITM attacks, so for example you cannot simple send a message frame from your infotainment pretending to be an Emergency braking request to your ABS system.
- Silhouette 6y agoThe problem is that there is an increasing overlap between physical vehicle control systems, cabin controls and displays, and remote communications. For example, how do you design a system that can automatically call for help in the event of an accident without giving that system access to the sensors that also trigger airbag deployment and similar safety features? How do you build "self-driving" automation (or even less ambitious driver aids that are already in widespread use) without relying on sensors observing the environment around the vehicle, which may be subject to interference or deliberate deception?
- DoingIsLearning 6y agoAll the use cases you described are already implemented in today's cars, bus segregation doesn't mean zero communication. Gateways can allow for certain messages to pass-through, the point is that they preserve the garantee that certain safety-critical messages can only originate from within a specific network. It's basically a crude MITM mitigation.
- TeMPOraL 6y ago> All the use cases you described are already implemented in today's cars, bus segregation doesn't mean zero communication. That's the problem, though. Once bidirectional communication exists, you can pretty much guarantee there's an exploitable hole in it that can be used to break the firewall.
- MaxBarraclough 6y ago> It really should be illegal to have network connected cars. The important systems should not be permitted to be connected to the Internet. If an Internet-enabled entertainment system is fully isolated from the systems that really matter, there's no problem.
- lvturner 6y agoEven then you could create random full volume sharp noise to be emitted from said entertainment system (or perhaps a blood curdling scream) I could see that causing some secondary serious problems. Could potentially amplify its impact by combining it with the GPS to ensure it occurs at a high speed and in busy traffic too.
- DoingIsLearning 6y agoAnd that is why fail-safe is important. Most older infotainment systems have a physical power switch which is a hard interrupt to power-off. It is in my view a design mistake to have full touch-screen interfaces on infotainment.
- hutzlibu 6y agoMost older system of any kind had a physical power switch. And I still want that. To be sure, that when I turn off a device, it is really off.
- sriram_malhar 6y agoIt could just be a soft commanding voice that says, "this is not an emergency. Please pull over to one side of the road and get out of the car and wait". Just that will cause enough chaos and accidents.
- Teever 6y agoThere is an insane bug in many human minds that is easily exploited with a soft commanding voice: https://en.wikipedia.org/wiki/Strip_search_phone_call_scam https://en.wikipedia.org/wiki/Strip_search_phone_call_scam
- im3w1l 6y agoYes exactly, people instinctively deny to themselves inconvenient things if they can find even a flimsy out. So sometimes you need a spectacular demonstration to pierce the psychological defenses.
- hobby-coder-guy 6y ago> Telling a business person they have a software problem is meaningless. Why do you think this? It isn’t true.
- _the_inflator 6y ago> What could go wrong in a 2 ton computer on wheels I don't see it this way. Right now, there are so many people in cars that should not drive. Alcohol, drugs, lacking skills - to me, computers add to safety. There will always be a risk. I trust computers more in this case.
- verroq 6y agoThe point is that automation causes us to cede control to the machines which can be hacked, or worse abused. It is a transfer of power to whoever controls the machines. This centralising power can be abused in ways people can only imagine. Of course, people will give up what’s left of their control and power in the name of “safety and security”, they’ve been doing that since 9/11. So much so that we have the term post-9/11 to remind them to toe the line (and to kneel and take off their shoes). Imagine for instance there is a bush fire and your centrally controlled car decides that the fire is fake news and prevents you from leaving since it’ll cause congestion on the roads. No different from Twitter banning the NY post story. Maybe they decide NY Post headquarters is not a real place and refuse to navigate the car there. Yeah we should implement some kind of social credit system so that people who do drugs can’t drive or take public transport. They’ve got that in the modern utopia of China right now.
- marcosdumay 6y agoEasily hacked computers don't add safety anywhere. They only add random failure and the possibility of large scale disasters. If you want to push the safety related tasks into the computer, you have the obligation to make them hard to hack. And on the case of cars, that includes not plugging them into the internet.
- Silhouette 6y agoThey only add random failure and the possibility of large scale disasters. And the latter is the really big danger with autonomous vehicles. The same consistency that means they aren't vulnerable to as much random variation as a human driver could also become a weakness. Yes, a drunk or some idiot on their phone or an unwitting patsy whose drink was spiked at the bar can cause a fatal crash. But they can't simultaneously cause 500,000 fatal crashes by doing the exact same stupid thing 500,000 times at once or by being the victim of the exact same crime 500,000 times at once.
- aeturnum 6y agoIf you're threat modeling, you need to know what you're dealing with. The equipment obviously wasn't designed with the goal of being able to destroy itself (even though it's possible). If equipment starts failing, identifying the source could be critical in limiting damage, and the best way to be able to recognize this kind of attack is do it and record the result.
- C1sc0cat 6y agoAnd any "engineer" would know you have to back test any simulation - and predicting the exact failure modes is hard. Having a real world failure under controlled conditions - would show where failure points are and possible fixes.
- rkangel 6y agoWe can theoretically understand concepts in the abstract, but sometimes more is needed for us to take them fully on board. The article mentions the people responsible not grasping the full scale of the problem. Giving them the visceral image of the potential destruction might help them take on board the risk emotionally as well as intellectually.
- mschuster91 6y ago> I wonder why they needed to actually destroy a relatively expensive piece of equipment, plus spend time & money on all the orchestration around this test. Because uneducated people (like Congressmen and executives) tend to believe stuff more when they can see it with their own eyes. It's like, in terms of information security, putting up sensitive information on walls that can be seen from the outside. Almost no one sees a problem with that because they believe it takes expensive equipment... until you go, take a 600mm+ tele lens and prove to them that their sensitive information can be stolen for under 200$ worth of camera rental fees. Or, remember the immigration debate in 2015? The picture of three year old Alan Kurdi on the beach and its consequences in politics? Never underestimate the power of images and videos.
- B_ryJenkins 6y agoits kind of bitter-sweet to say, but i would have to agree that congressmen are generally "uneducated".
- tpxl 6y agoMost people are uneducated in most things.
- eeZah7Ux 6y ago1) This is clearly a stunt meant to impress people. Information security is black magic and deals with unknown risk. People hate to think about unknown risks. 2) It's very difficult to justify increasing security budget every time some infosec people says so - especially if their salary depends on such budget.
- jdsully 6y agoI can guarantee they wouldn't have got the article in wired without the physical demonstration.
- gcheong 6y agoBecause there is nothing like a real-world test to verify if your ideas and predictions are correct and to reveal other things you may have not considered.
- MisterTea 6y agoThis was obviously a grant money grab. Anyone with a basic working knowledge of electrical machinery and mains power will understand this and would apply to any engineer working in power generation. It's a huge no duh.
- Lwrless 6y agoThe title reminds me of "How many engineers does it take to change a light bulb?"
- dgellow 6y agoWhat's your answer?
- cute_boi 6y agoMany
- leetcrew 6y agoone engineer can do it in about four hours. three and a half hours to research the ideal replacement bulb, twenty five minutes to set up the ladder safely, and five minutes to replace the bulb itself. source: I'm an engineer and I replaced a lightbulb this week. the real problem arises when you have multiple engineers replacing a lightbulb. the time needed to decide on a replacement bulb and agree that the ladder setup is safe is quadratic in n, where n is the number of engineers. it still only takes five minutes to replace it though :)
- xivzgrev 6y agoDidn’t read article but had some fun imagining what happened Code: While true { Get_current_demand() If current_demand > current_supply then increase_supply() } Post Mortem Eng mgr: where is the code to check the capacity of the supply? Eng: we considered it but the PM said it was a p1. Increasing supply to meet demand was the MVP solution PM: our extensive market research indicated that customers would come nowhere close to capacity constraints. In fact our contracts limit the amount of energy they get. But client X went way beyond that, then the generator blew up
- Cthulhu_ 6y agoThe TL;DR as I understood it is that they paused the synchronization with the (60 hz) grid for just a little bit, then reconnected it; running out of sync with the grid caused a sudden backlash, making the generator tear itself apart from the sudden shock.
- retSava 6y agoYeah. It's meant not to connect until sufficiently in sync. Running out of sync apparently results in the grid nudging the generator into sync, but here the discrepancy was so large the nudge became a percussive encouragement of significant enough magnitude for the generator to permanently take a seat and reconsider. (that sentence was fun to write)
- kn100 6y agoIt's useful to imagine this way: When a motor is forced to spin by some external influence, drawing more current out of it on the other end means it has to spin with more torque in order to maintain the same voltage. Here, the generators are connected together to supply power in unison. They must all be in phase with each-other otherwise the out of phase generator will find itself fighting with the other generators, and as beautifully demonstrated here will very violently lose the fight. You could describe this as a very cruel demonstration of destructive interference!
- 6y ago
- hgo 6y agoThe article seems interesting, but man, I can't read that font. Anybody has a good plugin for firefox that can help me?
- Freak_NL 6y agoReader mode is what you want, and it is a built-in Firefox feature. You should see an icon on the right of your address bar. On GNU/Linux Ctrl+Alt+R activates it.
- tleb_ 6y agouBlock Origin allows you blocking font requests so it uses the fallback. I have it turned on by default and it's only an issue with icon fonts.
- PhantomGremlin 6y agoI got tired of struggling against unreadable fonts. So now Firefox is set to present me with Lucida Grande for all fonts, viz. serif, sans-serif, and monospace. I don't allow pages to choose their own fonts. I don't think I'm missing much in general, but I do switch to Safari in special situations.
- nullc 6y agoDisable web fonts: about:config and set gfx.downloadable_fonts.enabled to false. Readable pages and bonus: two RCE vulnerabilities avoided so far.
- gruez 6y ago>Readable pages doesn't this cause random pages that use fontawesome (and other glyphicons) to fail?
- nullc 6y agoCauses some little useless icons to display as missing-character boxes. I've never found it to make pages unusable, or even particularly worse: those icons are often pretty inscrutable. (presumably pages that became unusable would have ADA problems...)
- pedrocr 6y agoSeems like the repost detector failed somehow: https://news.ycombinator.com/item?id=24903625 https://news.ycombinator.com/item?id=24903625 Same exact link 6 hours before.
- detaro 6y agonot quite - this here is actually an older submission that mods boosted through the second chance pool.
- yodelshady 6y agoThe incident itself is over a decade old and available under FOI, so it's possibly fair to file it under "things that make the front page from time to time". It's certainly not news.
- SagelyGuru 6y ago"...and millions of dollars in federal funds he’d spent developing the attack ...." Sounds like rather expensive 30 lines of code.
- londons_explore 6y agoThis is the difference between "Bob the engineer says the bridge will fall down if you cut this cable", Vs "Bob the engineer cuts cable to demonstrate how easy it is to knock bridge down". We should trust our engineers, and not require them to demonstrate something at great cost to get the attention of people who make decisions...
- harry8 6y agoWhich engineers to trust? The ones saying Y2K needed billions of investment or cars all crash with planes falling from the sky as we usher in the post-technology age Jan 1, 2000. Remember that? Some people still believe them that we headed it off successfully, everywhere on earth in every industry, regardless of how little was spent, including nothing at all... You can't trust engineers anymore than you can trust lawyers (which lawyers?) or Doctors (Second opinions ever much?) Or indeed scientists, managers and politicians. The question to ask is "Do you have good reason to find this convincing." When what is at stake has sufficient value proving the obvious is indeed, also correct, beyond doubt has very real benefits.
- hinkley 6y agoThis is the same kind of thinking that has people questioning quarantine protocols (in general, not just this one). A precautionary measure that prevents a worst case scenario doesn't suddenly become stupid because the scenario it was designed to prevent does not happen. We hedge our bets all the time, and strategy people who don't hedge will end up driving busloads of people off a cliff sooner or later. Reckless gambling has no place in policy decisions. We also had some pretty fair warning from industries that use long-term thinking. Banking software was already breaking in 1969 (30 year mortgage logic). For many industries this was a long emergency. Even by a cynical view that these were pork projects that didn't 'need to get done', so what? We do these all the time. The cash infusions prop up industries or divisions, and often serve as cover for attacking deferred maintenance. Over beers you can complain about how stupid it is to have to get things done this way, but at work you take the money and run as far as you possibly can. What bothers me most, I think, is that if you've worked in software for very long at all, you should know all of this. Politics and proxies come up if you work at a mature company, or one that matures while you're there. (On December 31st 1999, I watched the New Year's celebrations for Tokyo and Australia. When they didn't blow up, I agreed to make plans involving alcohol for the evening. By the time Europe didn't blow up, I figured we were gonna be okay, so Happy New Year to me.)
- mwill 6y agoThis is an excerpt from Andy Greenberg's book Sandworm, which I recommend, I grabbed it on audible earlier this year and thoroughly enjoyed it
- baybal2 6y agoWait a second, and think how real is the chance of protection relays having an Internet connection.
- Rychard 6y agoI would imagine the chance is higher than you think. Being able to remotely monitor and manage such systems may lead to big cost-savings, if it enables the same number of employees to monitor multiple locations. Seeing as how many tech companies routinely cut corners on security, one could easily imagine a non-tech company doing the same in an attempt to maximize profits.
- Animats 6y agoIt's not just remote monitoring. Protective relaying involves some disconnects sending signals to distant disconnects to tell them not to disconnect. The idea is that when you have a sudden overload like a line down, you want to trip the breaker closest to the trouble, and not others elsewhere upstream. Without this, small disruptions take out too much of the grid. See NYC blackout of 1965.[1] This is why large transmission networks require some capability for fast transmission of data between circuit breakers. Not much data. But it's so much cheaper to send it over the Internet than to have separate microwave repeaters or separate cables or gear to send data over high-tension power lines. [1] https://en.wikipedia.org/wiki/Northeast_blackout_of_1965 https://en.wikipedia.org/wiki/Northeast_blackout_of_1965
- netflixandkill 6y agoDirectly? Low. Indirectly through serial concentrators and RTUs and Scada access, almost 100%. The FERC / NERC security standards that fell out of this event are largely a bad joke, but had the benefit of at least requiring North American utilities to do something at all. Any transmission operator of size and most generation operations effectively require real time Scada for anything of meaningful capacity and they're sure not paying someone to sit next to it reading numbers off over a phone. They may not be IP addressed individually but to put it in perspective, I had a communication engineer tell me with a straight face that the fiber mux I telneted into over their local wifi wasn't internet connected, and thus the protection relays on the 500 kv substation the next building over were serial only -- over said fiber cables.
- anotheryou 6y ago"You could imagine it happening to a machine in an actual plant, and it would be terrible" He still hasn't fully grasped it yet or else he'd have said: "You could imagine it happening to all machines in all plants at the same time, and it would be terrible"
- X6S1x6Okd1st 6y agoGiven that the failure mode seems to rely on force applied to a specific generator from other generators this particular attack could only take down some percentage of all running generators at any given time correct?
- anotheryou 6y agoI think it's not unlikely that a very high percentage of these protective relay is from the same manufacturer running the same "operating system" with the same vulnerability. With that it doesn't matter if it's the same hardware model and also not which generator sits behind it. From the article it sounds like that force would have killed any generator.
- X6S1x6Okd1st 6y agoWhere is that force coming from though?
- anotheryou 6y agothe grid? you're thinking of this?: https://news.ycombinator.com/item?id=24911509 https://news.ycombinator.com/item?id=24911509
- Veserv 6y agoSure, but the OC was clearly alluding to the fact that the real risk is a true mass attack. If you could do anything to every generator, what is the worst thing you could do? If you use this technique you could probably blow out some percentage of generators. Would that take down the grid? For how long? How hard would it be to recover? We already know power grids can be taken down by cyberattacks, just look at the cyberattack Russia did against Ukraine's power grid [1]. The main difference is that they were probably not trying to permanently take down their grid. If you could do something that would destroy enough capacity to keep the grid down for months the casualties would be immense. There would be immediate consequences like the loss of heating and cooling resulting in exposure deaths, but these pale in comparison to losses of fresh water, fuel production and thus food transport, fertilizer production and thus food production, manufacturing and thus goods, cargo port throughput and thus shipping. Modern society in the developed world utterly depends on our power grid. [1] https://www.wired.com/story/russia-ukraine-cyberattack-power-grid-blackout-destruction/ https://www.wired.com/story/russia-ukraine-cyberattack-power...
- helsinkiandrew 6y agoIsn't this what Stuxnet did to thousands of Iran's Nuclear centrifuges? Has this demonstration had any effect on policy anywhere?
- marcosdumay 6y agoThere was some oil pipe blown due to a software attack before Stuxnet. If those changed anything, it was on some place I can't see.
- beambot 6y agoCIA plot led to huge blast in Siberian gas pipeline: https://www.telegraph.co.uk/news/worldnews/northamerica/usa/1455559/CIA-plot-led-to-huge-blast-in-Siberian-gas-pipeline.html https://www.telegraph.co.uk/news/worldnews/northamerica/usa/... Leaked extracts in yesterday's Washington Post describe how the operation caused "the most monumental non-nuclear explosion and fire ever seen from space" in the summer of 1982. "In order to disrupt the Soviet gas supply, its hard currency earnings from the West, and the internal Russian economy, the pipeline software that was to run the pumps, turbines and valves was programmed to go haywire after a decent interval, to reset pump speeds and valve settings to produce pressures far beyond those acceptable to pipeline joints and welds," Mr Reed writes
- Lucasoato 6y agoAre you sure? https://skeptics.stackexchange.com/questions/44212/did-the-cia-blow-up-a-siberian-pipeline-in-1982 https://skeptics.stackexchange.com/questions/44212/did-the-c...
- beambot 6y agoNot sure... Just remember the lore being reported. As with all things cloak & dagger, take it with a grain of salt.
- marcosdumay 6y agoI dunno. It's good to know that it's contested, but all the intelligence organs (from both sides) denying a hacking took place is how things always go. They denied that the Stuxnet was a cyberweapon too, for years after all the details were public known. That said, the argument that there wouldn't be a computer controlled pipeline in 1982 is kinda strong. If I were interested enough on the history, I'd start looking from there.
- Aaronstotle 6y agoI picked up his book from the library and it's been a good read. Makes me frustrated at just how little the higher ups in the U.S. Government seem to care about cyber-security. They'll learn the hard way I guess
- xbar 6y agoOh, you should see my one-liner.
- Animats 6y agoIt wasn't a "secret experiment". It was in the trade press for the power industry long ago.
- dosman33 6y agoI think this is the right thing to post here: https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25
- MisterTea 6y agoThere is nothing interesting here. So called "Hackers" simply told the generators relay to close out of phase causing the generator and grid currents to violently crash into each other. This means the magnetic fields of the rotor and armature were not aligned. Obviously the grid has much larger generators backing it so the little diesel gen set rotor was no match for the now grid energized armature. The armature field tries to violently pull the rotor into phase resulting in catastrophic physical forces on the generator mechanics. After a few of these devastating blows the diesel engine's crank shaft, connecting rod or perhaps the flywheel coupler violently fails. The only code needed would be to completely ignore line and gen set phasing or flip them 180 to really let the bolts fly. Any generator will succumb to this no matter the prime mover. They must be in sync with the grid or other operating generators before connected across the line/bus. This is power house practice 101 level stuff.
- airbreather 6y agoSo they messed with the synchroniser and the synch check relays, on a reciprocating deisel. I design such systems and it sort of reads ok, but a few key details not right or not explained - those size of generators grid connected will have reverse power, pole slip, instantaneous over current electrical protection as well, AVR excitation limiting, plus overspeed relays which would not have let this happen. (gas turbines traditionally have a hydraulic control oil relief overspeed "bolt" that comes out with centrifugal force and no software could mess with, these days electronic replacements are coming in vogue, but...) As for synch check, rule of thumb is two independant systems, one hard wired, three overall if you can swing it because often one of the multifunction protection relays will have that function available and you wire it and/or code it in line. But I have seen big reciprocating diesels go into synch at 30-60 degrees out of phase and if they hold without tripping the generator protection then you might get a brief flicker of the lights, some stress on the crankshaft that who knows what it does long term, but some of them were old and had it done to them many times. Yeah, so they did it, waste of a good machine for known outcome if ask me, very wasteful with no need. And it is a very particular kind of attack, if you knew it was a problem it could be protected against for hundreds or thousands of generators literally in 24 hours or less - hardwired synch check which can be achieved with a voltage relay at it's simplest. So I am not seeing anything in that excerpt other than highighting bad design practices.
- inetknght 6y ago> Yeah, so they did it, waste of a good machine for known outcome if ask me, very wasteful with no need. And yet there are plenty of people who don't understand that software problems can cause permanent damage to hardware or people. Or they think of it in the abstract. I don't think it's wasteful to demonstrate. It was definitely costly in this case. I think the author could have done with a smaller demonstration. But there is something to be said for grandeur. > So I am not seeing anything in that excerpt other than highighting bad design practices. Indeed. I think that was exactly the point of the demonstration.
- redis_mlc 6y agoThere's 2 reasons why security researchers go beyond a theoretical demonstration into a destructive one: 1) Security researchers have an intense need to show their exploits actually work in case they're accused of not providing proof. If you go to Defcon, most of the presenters spend half their talk on showing use of the exploit, no matter how tedious it is to the audience. (When I supervise IT security audits, I have to tell pentesters just to document what they find instead of making changes, which introduces more problems and wastes time and effort. I've had auditors find a configuration file with passwords on a production server that uses them, then try to present a report on how they logged into each one. Duh, of course they work.) 2) This is related to #1, but business people, especially across silos in an organization, may not fully comprehend an exploit unless there is obvious proof. I've seen laughable levels of denial by otherwise intelligent people once they get into management. As in, "Yeah you showed it to me on the screen, but I'll need to confirm it with other sources" who then look at the same screen. Pro tip 1: by default AWS AMI's are not encrypted. (The reason is that the main list of AMIs doesn't use your KMS keys.) So if you have a compliance program that requires encryption at rest, get cracking! Pro tip 2: All theoretical exploits have shown to be practical ones later. Stop deceiving yourself.