7 ms·

> Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more

by sevg 3y ago

> Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian products (Jira, Confluence, and Bitbucket). > The threat actor also attempted to access a console server in our new, and not yet in production, data center in São Paulo. All attempts to gain access were unsuccessful. To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway. They didn't have to go this far. It would have been really easy not to. But they did and I think that's worthy of kudos.

readyplayernull 3y ago
> The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway. Aha, the old replace-your-trusted-hardware trick.
- zitterbewegung 3y ago
  Manufacturers have had security vulnerabilities for hardware to the point that the firmware on device couldn’t be trusted to be replaced so they said to get new hardware so it’s not a bad strategy.
- AzzyHN 3y ago
  In a corporate environment, standard procedure when an employee's computer gets infected is to re-image it. Even if it was a stupid virus that was immediately caught, the potential risk of undetected malware running amuck is just too high. Now imagine, instead of Steve from HR's laptop, it's one of Cloudflare's servers.
syncsynchalt 3y ago
Honestly I wish we'd had an excuse/reason to do an org-wide prod creds refresh like this at some places I've been. You find some scary things when you go looking for how exactly some written-by-greybeard script is authenticating against your started-in-1990s datastore.
barkingcat 3y ago
I think they did have to do that far though. Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in at the centre of a new Meet-Me room (https://en.wikipedia.org/wiki/Meet-me_room https://en.wikipedia.org/wiki/Meet-me_room) and having persistent access to key switches there. Cloudflare datacentres tend to be at the hub of insane amounts of data traffic. The fact that the attacker knew how valuable a "pre-production" data centre is means that cloudflare probably realized themselves that it would be a 100% game over if someone managed to get a foot hold there before the regular security systems are set up. It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up. Also remember, at the beginning of data centre builds, all switches/equipment have default / blank root passwords (admin/admin), and all switch/equipment firmware are old and full of exploits (you either go into each one and update the firmware one by one or hook them up to automation for fleet wide patching) Imagine that this exploit is taking place before automation services had a chance to patch all the firmware ... that's a "return all devices to make sure the manufacturer ships us something new" event.
- vasco 3y ago
  What I think they meant is customers would keep paying them. And they are right, one just has to look at Okta, Solarwinds and other providers that have been owned, not done half of this and somehow are still in business. Everyone whistles to the side and pretends they shouldn't switch vendors, rotate all creds, cycle hardware, because it saves lots of work and this stuff falls under "reasonable oopsie" to the general public, when in fact there should be rules about what to do in the event of a breach that should be much stricter. So they do some partial actions to "show work" in case of lawsuits and keep going. The old engineers leave, new ones come in, and now you have systems who are potentially owned for years to come. It takes some honesty and good values by someone in the decision-making to go ahead with such a comprehensive plan. This is sad because it should be tablestakes, as you say correctly, but having seen many other cases, I think although they did "the expected", it's definitely above and beyond what peers have done.
- swyx 3y ago
  do manufacturers share some of the cost of this kind of security related return or is this a straight up "pay twice for the same thing" financial hit?
tptacek 3y ago
This is why old secops/corpsec security hands are so religious about tabletop exercises, and what's so great about BadThingsDaily† on Twitter. Being prepared to do this kind of credential rotation takes discipline and preparation and, to be frank, most teams don't make that investment, including a lot of really smart, well-resourced ones. If Cloudflare is in a position where their security team can make a call to rotate every secret and reimage every machine, and then that happens in some reasonable amount of time, that's pretty impressive. † https://twitter.com/badthingsdaily?lang=en https://twitter.com/badthingsdaily?lang=en
- akira2501 3y ago
  It'd be more impressive if they actually got all the credentials. It's good that you think you can absorb a complicated security task, it's useless if you have no way to test or verify this action.
  - swyx 3y ago
    yes but this is a nice #2. not many fortune 500s would 1) even know they were breached and 2) if they were breached, have the breach be so contained.
schainks 3y ago
Having seen the small number of DEFCON talks that I've seen, I would have absolutely gone that far.
orenlindsey 3y ago
Cloudflare is showing how to correctly respond to attacks. Other companies should take note.
ldoughty 3y ago
The nuclear response to compromise should be the standard business practice. It should be exceptional to deviate from it. If you assume that they only accessed what you can prove they accessed, you've left a hole for them to live in. It should require a quorum of people to say you DON'T need to do this. Of course, this is ideal world. I'm glad my group is afforded the time to implement features with no direct monetary or user benefit.
deleted 3y ago
[deleted]

by sevg 3y ago

readyplayernull 3y ago

> The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway. Aha, the old replace-your-trusted-hardware trick.

zitterbewegung 3y ago
Manufacturers have had security vulnerabilities for hardware to the point that the firmware on device couldn’t be trusted to be replaced so they said to get new hardware so it’s not a bad strategy.
AzzyHN 3y ago
In a corporate environment, standard procedure when an employee's computer gets infected is to re-image it. Even if it was a stupid virus that was immediately caught, the potential risk of undetected malware running amuck is just too high. Now imagine, instead of Steve from HR's laptop, it's one of Cloudflare's servers.

syncsynchalt 3y ago

Honestly I wish we'd had an excuse/reason to do an org-wide prod creds refresh like this at some places I've been. You find some scary things when you go looking for how exactly some written-by-greybeard script is authenticating against your started-in-1990s datastore.

barkingcat 3y ago

I think they did have to do that far though. Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in at the centre of a new Meet-Me room (https://en.wikipedia.org/wiki/Meet-me_room https://en.wikipedia.org/wiki/Meet-me_room) and having persistent access to key switches there. Cloudflare datacentres tend to be at the hub of insane amounts of data traffic. The fact that the attacker knew how valuable a "pre-production" data centre is means that cloudflare probably realized themselves that it would be a 100% game over if someone managed to get a foot hold there before the regular security systems are set up. It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up. Also remember, at the beginning of data centre builds, all switches/equipment have default / blank root passwords (admin/admin), and all switch/equipment firmware are old and full of exploits (you either go into each one and update the firmware one by one or hook them up to automation for fleet wide patching) Imagine that this exploit is taking place before automation services had a chance to patch all the firmware ... that's a "return all devices to make sure the manufacturer ships us something new" event.

vasco 3y ago
What I think they meant is customers would keep paying them. And they are right, one just has to look at Okta, Solarwinds and other providers that have been owned, not done half of this and somehow are still in business. Everyone whistles to the side and pretends they shouldn't switch vendors, rotate all creds, cycle hardware, because it saves lots of work and this stuff falls under "reasonable oopsie" to the general public, when in fact there should be rules about what to do in the event of a breach that should be much stricter. So they do some partial actions to "show work" in case of lawsuits and keep going. The old engineers leave, new ones come in, and now you have systems who are potentially owned for years to come. It takes some honesty and good values by someone in the decision-making to go ahead with such a comprehensive plan. This is sad because it should be tablestakes, as you say correctly, but having seen many other cases, I think although they did "the expected", it's definitely above and beyond what peers have done.
swyx 3y ago
do manufacturers share some of the cost of this kind of security related return or is this a straight up "pay twice for the same thing" financial hit?

tptacek 3y ago

This is why old secops/corpsec security hands are so religious about tabletop exercises, and what's so great about BadThingsDaily† on Twitter. Being prepared to do this kind of credential rotation takes discipline and preparation and, to be frank, most teams don't make that investment, including a lot of really smart, well-resourced ones. If Cloudflare is in a position where their security team can make a call to rotate every secret and reimage every machine, and then that happens in some reasonable amount of time, that's pretty impressive. † https://twitter.com/badthingsdaily?lang=en https://twitter.com/badthingsdaily?lang=en

akira2501 3y ago
It'd be more impressive if they actually got all the credentials. It's good that you think you can absorb a complicated security task, it's useless if you have no way to test or verify this action.
- swyx 3y ago
  yes but this is a nice #2. not many fortune 500s would 1) even know they were breached and 2) if they were breached, have the breach be so contained.

schainks 3y ago

Having seen the small number of DEFCON talks that I've seen, I would have absolutely gone that far.

orenlindsey 3y ago

Cloudflare is showing how to correctly respond to attacks. Other companies should take note.

ldoughty 3y ago

The nuclear response to compromise should be the standard business practice. It should be exceptional to deviate from it. If you assume that they only accessed what you can prove they accessed, you've left a hole for them to live in. It should require a quorum of people to say you DON'T need to do this. Of course, this is ideal world. I'm glad my group is afforded the time to implement features with no direct monetary or user benefit.

deleted 3y ago

[deleted]