8 ms·
Engineer admits he wiped 456 Cisco WebEx VMs from AWS after leaving
- nixgeek 6y agoBiggest question for me would be why the employee still had access so long after terminating employment with Cisco. A common piece of auditor evidence across many compliance frameworks is whether employees have access proportionate to their role (which is naturally highly subjective), but also proving that access is revoked when employees leave the company. This seems like an outright failure on Cisco’s part. Hopefully they’ve learned from this and put effort into enhancing their identity governance situation.
- drdeadringer 6y agoAbout 5 years ago I had [contractor] job offers for Cisco and Google. I was negatively impressed about Cisco; I still don't see that changing any time soon. I say this without rosy glasses about Google.
- chromedev 6y agoI knew someone that worked at WebEx in their FedRAMP environment performing vulnerability remediation, and most of the work was outsourced to China because the founders are from China. They posted screenshots of a conversation with their teammate saying that because of the convoluted process to install packages, they had concerns that they were installing some sort of backdoor. They reported this and had investigators constantly checking out their LinkedIn profile for months.
- gruez 6y agoThis paragraph is baffling: >According to a court document, Ramesh is in the US on an H-1B visa and has a green card application pending. "Although he and his employer recognize that his guilty plea in this case may have immigration consequences, up to and including deportation, his employer … is willing to work with him regarding the possibility of his remaining in the country and continuing to work for the company," the document [PDF] says. Why would you re-hire someone who quit and wiped your servers?
- Traster 6y agoHis current employer is actually stitchfix (not Cisco)
- swatkat 6y agoI believe his current employer (Stitch Fix) is willing to keep him employed, not his ex-employer (Cisco).
- supercanuck 6y agoBecause his skills are valued higher than his cost.
- treeman79 6y agoWell he’s good at saving money on VM’s
- olliej 6y agoI can’t recall the deal with h1bs, but green card applications require that you haven’t committed any acts of moral turpitude, which would easily include this. All work visas have a no criminal charges rule, so if this is a criminal case I believe being found guilty puts him in the area of instant visa revocation
- harlanji 6y agoCriminal vs. civil, afaik the former is gov. vs. ____. Moral terpiude, deployed a project to GCP and it deleted VMs—not my idea of a moral failure. Generally I’d not fire people for honest mistakes no matter the cost. You have to pay it anyway, and now the person has hopefully learned and probably feels honored to work hard to be better.
- saidajigumi 6y agoThis article leaves more questions than it answers. Room-elephant number one: access being available after an employee has left is bad. That access remaining five months later is beyond the pale, unless the real story is that the employee created a backdoor. Barring a backdoor, there are further serious questions about the employee retaining this access, presumably without any employer-provided and controlled hardware (e.g. laptop, yubikey, or what-have-you). Room-elephant number two: motive. The reported facts naively summarize as "oops, ex-employee blew up some stuff in prod, caused problems". <meme>But whyyyyy??</meme> There's no indication of specifics, and seeming denials of some obvious guesses: attempts at hacking (e.g. data exfiltration for profit, which are denied), ransomware, revenge, or anything else that would explain this behavior. Further confounding everything is the bit where the new employer's response to these revelations is apparently "shrug".
- drdeadringer 6y agoI too am confused about motive. Timing aside, I myself would have to have Malicious Hate in my heart, or some ethical//moral equivalent in my brain, to do active big-cost "fire in the hole" damage on to a former employer.
- mmazing 6y agoI've had to juggle personal and professional AWS accounts for a while, I could see someone being confused about which account they were on and accidentally wiping out some stuff. Who knows though.
- tootie 6y agoI worked in consulting up until Covid. When I got laid off, my employer locked me out of every corporate system within 15 minutes. But every client who gave me VPN, AWS or other credentials didn't get notified.
- sangnoir 6y agoInteresting angle. I wonder if the perp was employed by Cisco directly or was a contractor and Cisco wasn't informed when he changed employers.
- viraptor 6y agoI can't find any place explicitly saying this was done maliciously. A theory: this could also be a really bad accident where (for example) he works without a company provided computer and didn't clean out his old AWS profiles from previous company - ended up deleting resources from the wrong account.
- henearkr 6y agoEspecially 5 months after being fired. Steam should have had time to cool off.
- rantwasp 6y agoi don't think he did this on purpose. it's probably the classic "ran with wrong credentials" story of one really unlucky negligent fellow.
- paranoidrobot 6y agoWhile yes, this is a problem for several reasons (They should've taken care to clean any company IP off the laptop). But the biggest, as others have already said is - why wern't his credentials revoked after leaving? I can understand this at smaller companies, but Cisco has no excuse - they have enough people around that there's surely multiple people who's job it is to ensure that credentials are tied to a person, and that after a person leaves they're all revoked on anything approaching a production/customer-facing environment.
- TheCondor 6y agoThat seems like a lot of resources to delete... It’s “possible” but I’m not sure they I find it plausible. Is there some magic terraform or cloud formation that just destroys everything? And a Cisco engineer with devops like experience wouldn’t at least sniff around before running it? Doesn’t seem realistic. Cisco should wear this too though, this is shockingly negligent. The only reason I can think of suggests a lot more problems and likely noncompliance with regulations and standards I’m sure they claim to comply with.
- 6y ago
- deleted 6y ago[deleted]
- TwoBit 6y agoWith that kind of sloppy security in place, Cisco is going to be easily ransomwared.
- chromedev 6y agoThey already are. Based on someone I know, they were warned by their coworkers at WebEx that much of the work was outsourced to China and they had concerns that they were installing backdoors.
- waste_monk 6y agoAh yes, Cisco, well known for their good security practices. Such as this galaxy-brain manoeuvre: https://twitter.com/RedTeamPT/status/1110843396657238016 https://twitter.com/RedTeamPT/status/1110843396657238016
- kstrauser 6y agoI left one job to switch to another. A year later, my new job started using a cloud-based task management app. When I went to sign in, my 1Password auto-filled the credentials I'd used for the same app at my previous job, and there I was looking at all of my old employer's current projects and other confidential info. I called my old boss (who I got along with just fine), told him what happened, and asked him to please cut off my access immediately. When you leave a job, it's in your own best interest to make sure that all of your access is removed. It's a lot harder for them to blame unexpected happenings on you if you can't even log into the thing. (Not that this happened here. I just wanted to point out a gotcha you might not have thought about.) If you find out that they missed something, report it to them immediately and keep that paper trail demonstrating your good intentions toward them. Then hound them about it until they get around to fixing the situation. And for the love of God, don't ever, EVER log in "just to look around". Absolutely no good can come of that.
- andi999 6y agoHow would you know that they didnt disable you accounts without trying to log in.
- steve_adams_86 6y agoI think they meant ‘don’t log in, discover you have access, then browse and take no further action’. You can try to log in, you just need to inform someone immediately if you have access when you shouldn’t. That’s my take anyway.
- samus 6y agoThat's what to do when it happens accidentally. It would also make sense to do so immediately after you left. They already made sure that you left behind all relevant keys, documents, and your work computer. It would be natural and in both parties' interest to ensure that all other credentials have been revoked as well in time.
- andi999 6y ago
- blinkingled 6y agoThis feels like may be he at some point discovered that his AWS access to Cisco account was intact and got curious about it and maybe even did some harmless things. Then while playing around with GCP he managed to run something that deletes stuff (Terraform maybe) but the credentials used were that of Cisco AWS account which wasn't what he intended - clearly just deleting stuff is not the smart thing to do when it will be recorded against his AWS credentials. I think he is pleading guilty to unauthorized access which was intentional - but not to the deletion which was unintended.
- nervlord1 6y agoErmmm yeah that's a well deserved deportation right there
- turowicz 6y agoIt also means there was no orchestration for the VMs. System should have recreated them 1 by 1 on a health check triggered restore action.
- gregoriol 6y agoThis is a really scary situation for an employee: maybe this case was a mistake or maybe it was bad intentions, but imagine you are leaving a company and your access should be closed, but is actually not, then your account still active without your knowledge might get hacked... and you could be responsible?
- eithed 6y agoPhhhh, 6 months... I'd still have access to my first company's Google local directory console almost 14 years after I've left it (this is despite numerous messages on my part to remove my access) if Google were not to remove the listing altogether