6 ms·

We had a prod case where a server was being flooded with requests, and a downstream server kept falling over. We figured it was an attack of some sort and inves

We had a prod case where a server was being flooded with requests, and a downstream server kept falling over. We figured it was an attack of some sort and investigated, eventually traced it back to a computer inside our own network (we're a big computer, five floors of computers). It had an open file share, containing some Delphi books and from which we got the computer name too. So we walked over to the Delphi team's side, and kept yelling the computer name until some dude said "Hey, that's me!" Turns out he was running a test-case, in an infinite loop until it worked (because that's how test cases worked), and he thought he was pointed at QA, but he somehow had it set up to target Prod. Our job was done at that point, we left the rest to management (who made sure he didn't get fired but didn't do it again).

Seattle3503 5y ago
I'm surprised employees have sufficient access to prod to make this mistake.
- more_corn 5y ago
  I've done security reviews for a dozen companies. This sort of thing is startlingly common. Every single company I've reviewed is doing something that in retrospect should have been obvious. I try to tell people: "You don't need AI security, you need a checklist." Colonial Pipeline reused passwords, shared passwords, used the same password for all VPN users, failed to rotate it when people left. (that's 4 insanely basic violations of password security). ANY human who did a security review would have caught that. Even an intern who knew nothing and furiously googled "information security review" on the bus on the way in to kick off the review. (no disrespect to interns in over their heads, my point is they didn't prioritize security so they didn't get security) Capital One used an admin privileged instance profile attached to a publicly accessible admin interface for a security tool (which tool, by the way, had no need of admin credentials). They were hit by an SSRF vuln and leaked their admin credentials. They also failed to alert of unexpected use of those credentials (try it, use of admin credentials is rare enough you won't have a lot of noise) failed to alert on large outbound connection (this one is subtle, but worth doing if you can figure it out) Equifax failed to apply security updates regularly (just turn on automatic security updates. People suck at chores) Failed to deploy a SIEM, failed to conduct periodic security reviews, failed to put capable security people in place. The above are not my clients, just public reports to illustrate that everyone can benefit from a security review to catch the obvious errors.
- EvanKnowles 5y ago
  They shouldn't have, a lot went wrong here.
niij 5y ago
Doesn't sound like a management failure to me. It sounds like there should be separate vlans for QA/test and Production to prevent this very thing (or potentially something more malicious like the spread of ransomware).
- EvanKnowles 5y ago
  I'd say to say "Yeah, this was a long time ago"... but this could probably still happen.
that_guy_iain 5y ago
One of the issues with Knights Capital was that they forgot about a server running an old bit of code and shut down all the new ones which just sent all the data to the old server which was causing all the problems. Not keeping track of that server was very expensive.

Seattle3503 5y ago

I'm surprised employees have sufficient access to prod to make this mistake.

more_corn 5y ago
I've done security reviews for a dozen companies. This sort of thing is startlingly common. Every single company I've reviewed is doing something that in retrospect should have been obvious. I try to tell people: "You don't need AI security, you need a checklist." Colonial Pipeline reused passwords, shared passwords, used the same password for all VPN users, failed to rotate it when people left. (that's 4 insanely basic violations of password security). ANY human who did a security review would have caught that. Even an intern who knew nothing and furiously googled "information security review" on the bus on the way in to kick off the review. (no disrespect to interns in over their heads, my point is they didn't prioritize security so they didn't get security) Capital One used an admin privileged instance profile attached to a publicly accessible admin interface for a security tool (which tool, by the way, had no need of admin credentials). They were hit by an SSRF vuln and leaked their admin credentials. They also failed to alert of unexpected use of those credentials (try it, use of admin credentials is rare enough you won't have a lot of noise) failed to alert on large outbound connection (this one is subtle, but worth doing if you can figure it out) Equifax failed to apply security updates regularly (just turn on automatic security updates. People suck at chores) Failed to deploy a SIEM, failed to conduct periodic security reviews, failed to put capable security people in place. The above are not my clients, just public reports to illustrate that everyone can benefit from a security review to catch the obvious errors.
EvanKnowles 5y ago
They shouldn't have, a lot went wrong here.

niij 5y ago

Doesn't sound like a management failure to me. It sounds like there should be separate vlans for QA/test and Production to prevent this very thing (or potentially something more malicious like the spread of ransomware).

EvanKnowles 5y ago
I'd say to say "Yeah, this was a long time ago"... but this could probably still happen.

that_guy_iain 5y ago

One of the issues with Knights Capital was that they forgot about a server running an old bit of code and shut down all the new ones which just sent all the data to the old server which was causing all the problems. Not keeping track of that server was very expensive.