6 ms·

I think they did have to do that far though. Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in

I think they did have to do that far though. Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in at the centre of a new Meet-Me room (https://en.wikipedia.org/wiki/Meet-me_room https://en.wikipedia.org/wiki/Meet-me_room) and having persistent access to key switches there. Cloudflare datacentres tend to be at the hub of insane amounts of data traffic. The fact that the attacker knew how valuable a "pre-production" data centre is means that cloudflare probably realized themselves that it would be a 100% game over if someone managed to get a foot hold there before the regular security systems are set up. It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up. Also remember, at the beginning of data centre builds, all switches/equipment have default / blank root passwords (admin/admin), and all switch/equipment firmware are old and full of exploits (you either go into each one and update the firmware one by one or hook them up to automation for fleet wide patching) Imagine that this exploit is taking place before automation services had a chance to patch all the firmware ... that's a "return all devices to make sure the manufacturer ships us something new" event.

vasco 3y ago
What I think they meant is customers would keep paying them. And they are right, one just has to look at Okta, Solarwinds and other providers that have been owned, not done half of this and somehow are still in business. Everyone whistles to the side and pretends they shouldn't switch vendors, rotate all creds, cycle hardware, because it saves lots of work and this stuff falls under "reasonable oopsie" to the general public, when in fact there should be rules about what to do in the event of a breach that should be much stricter. So they do some partial actions to "show work" in case of lawsuits and keep going. The old engineers leave, new ones come in, and now you have systems who are potentially owned for years to come. It takes some honesty and good values by someone in the decision-making to go ahead with such a comprehensive plan. This is sad because it should be tablestakes, as you say correctly, but having seen many other cases, I think although they did "the expected", it's definitely above and beyond what peers have done.
swyx 3y ago
do manufacturers share some of the cost of this kind of security related return or is this a straight up "pay twice for the same thing" financial hit?
- eastdakota 3y ago
  We have very good relations with our network vendors (in this case, Cisco, Juniper, and Arista). The CEOs of all of them 1) immediately got on a call with me late on a weekend; 2) happily RMAed the boxes at no cost; and 3) lent us their most senior forensics engineers to help with our investigation. Hat tip to all of them for first class customer service.
  - swyx 3y ago
    shows how much they value you as a partner and i'm sure they appreciate your overall business. thanks Matthew! love the transparency and dedication to security as always. really sucks to have this be continuing fallout from Okta's breach. wish large scale key rotation was more easily automatable (or at least as a fallback, there should be a way to track key age on clientside? so that old keys stick out like a sore thumb). i guess in the absence of industry standard key rotation apis someday you might be able to "throw AI at it".
tgsovlerkhgsel 3y ago
> It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up. It wouldn't. Most people like to assume the impact of breaches to be what it should be, not what it actually is. Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed.
- mschuster91 3y ago
  > Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed. The problem with this is that while security minded people know what Okta is and why to stay the fuck away from handing over your crown jewels to a SaaS company is warranted, C-level execs don't care. They only care about their golf course or backroom deal friends and about releasing PR statements full of buzzwords like "zero trust", "AI based monitoring" and whatever. The stock markets don't care either, they only look at the financial data, and as long as there still are enough gullible fools signing up, they don't care and stonk goes up.
  - cqqxo4zV46cp 3y ago
    Yes, that’s literally the point being made. The point is that it isn’t a company-ending event. You are going on an unrelated rant about how those darn dumb executives aren’t as smart as God’s gift to earth, engineers.
    - mschuster91 3y ago
      The thing is, some events should be company ending. Something like Okta shouldn't even exist in a halfway competent world in the first place - given how many Fortune 500 companies, even governments use it, it's just a too fucking juicy target for nation states both friendly and hostile. Instead, even the "self correcting" mechanisms of the "free market" obviously didn't work out, as the free market doesn't value technical merit, it only values financial bullshittery. And the end result will be that once the war with China or Russia inevitably breaks out, virtually all major Western companies and governments will be out cold for weeks once Okta and Azure's AD go down, because that is where any adversary will hit first to deal immense damage.
nolok 3y ago
> It would be a company ending event Given they got out of cloudbleed without any real damage let alone lasting damage, I disagree. (I don't disagree with your point about how bad of a problem this would be, I'm just insisting that security failure is not taken seriously at all by anyone)
- Hrundi 3y ago
  I don't remember any companies that ended thanks to cloudbleed, but I'd be happy to be proven wrong
- chx 3y ago
  Presuming taviso is not exaggerating and why would he CF's reply to cloudbleed was ... not quite nice. https://twitter.com/taviso/status/1566077115992133634 https://twitter.com/taviso/status/1566077115992133634 > True story: After cloudbleed, cloudflare literally lobbied the FTC to investigate me and question the legality of openly discussing security research. How come they're not lobbying their DC friends to investigate the legality KF? For those not familiar with the history , this tweet started the cloudbleed disclosure to cloudflare: https://twitter.com/taviso/status/832744397800214528 https://twitter.com/taviso/status/832744397800214528 > Could someone from cloudflare security urgently contact me. This followed: https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug https://blog.cloudflare.com/incident-report-on-memory-leak-c...
  - cowsandmilk 3y ago
    I love this quote: > However, Server-Side Excludes are rarely used and only activated for malicious IP addresses. So… you’re celebrating that you only had buffer overruns for malicious IP addresses?
  - hughesjj 3y ago
    Yeah cloudflare is pretty sketchy too imo. They present as transparent but they've had some actions over the years that signal otherwise. Heck, pretty much every performance blog post they hype up buries the caveats, kinda reminiscent of Intel always using their custom cpp compiler for benchmarks. Not technically lying, but definitely omitting some context.
sneak 3y ago
> Imagine getting in at the centre of a new Meet-Me room and having persistent access to key switches there. This wouldn't get you much. We already assume the network is insecure. This is why TLS is a thing (and mTLS for those who are serious).
- noizejoy 3y ago
  > We already assume the network is insecure. Maybe naively, I wish this assumption became universal.
- sophacles 3y ago
  I suspect "we" is a much smaller group than you imagine. I've gotten pcaps from customers as recently as this year that include unencrypted financial transaction data. These were captured on a router, not an end host, so the traffic was going across the client's network raw.
meowface 3y ago
>Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. I can just imagine the attackers licking their lips when they first breached the data center. Good reminder to use "Full (Strict)" SSL in Cloudflare. Then even if they do get compromised, your reverse-proxied traffic still won't be readable. (Of course other things you might use Cloudflare for could be vectors, though.)
- ownagefool 3y ago
  Cloudflare is essentially a massive mitm proxy. If you manage to pwn a key, you have access to traffic. I'm sure they're better than this than me, but ipxe & tftp are plain text, so it wouldn't be shocking if something in the bootstrap process was plaintext. At the very least you need to tell the server what to trust.
  - meowface 3y ago
    >If you manage to pwn a key, you have access to traffic. That's why I mentioned "Full (Strict)" SSL. If you configure this in Cloudflare then the entire user <-> Cloudflare <-> origin path is encrypted and attackers can't snoop on the plaintext even if they have access. They'll get some metadata, but every ISP in the world gets that at all times anyway.
    - is39 3y ago
      While both client and origin network connections are encrypted with "Full (Strict)" SSL mode, Cloudflare proxy in the middle decrypts client traffic and then encrypts it towards the server (and vice versa). It does have access to plaintext, which is how various mitigations work. So it's indeed MITM proxy, by design.
      - meowface 3y ago
        Ah, yeah, you're right.

vasco 3y ago

What I think they meant is customers would keep paying them. And they are right, one just has to look at Okta, Solarwinds and other providers that have been owned, not done half of this and somehow are still in business. Everyone whistles to the side and pretends they shouldn't switch vendors, rotate all creds, cycle hardware, because it saves lots of work and this stuff falls under "reasonable oopsie" to the general public, when in fact there should be rules about what to do in the event of a breach that should be much stricter. So they do some partial actions to "show work" in case of lawsuits and keep going. The old engineers leave, new ones come in, and now you have systems who are potentially owned for years to come. It takes some honesty and good values by someone in the decision-making to go ahead with such a comprehensive plan. This is sad because it should be tablestakes, as you say correctly, but having seen many other cases, I think although they did "the expected", it's definitely above and beyond what peers have done.

swyx 3y ago

do manufacturers share some of the cost of this kind of security related return or is this a straight up "pay twice for the same thing" financial hit?

eastdakota 3y ago
We have very good relations with our network vendors (in this case, Cisco, Juniper, and Arista). The CEOs of all of them 1) immediately got on a call with me late on a weekend; 2) happily RMAed the boxes at no cost; and 3) lent us their most senior forensics engineers to help with our investigation. Hat tip to all of them for first class customer service.
- swyx 3y ago
  shows how much they value you as a partner and i'm sure they appreciate your overall business. thanks Matthew! love the transparency and dedication to security as always. really sucks to have this be continuing fallout from Okta's breach. wish large scale key rotation was more easily automatable (or at least as a fallback, there should be a way to track key age on clientside? so that old keys stick out like a sore thumb). i guess in the absence of industry standard key rotation apis someday you might be able to "throw AI at it".

tgsovlerkhgsel 3y ago

> It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up. It wouldn't. Most people like to assume the impact of breaches to be what it should be, not what it actually is. Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed.

mschuster91 3y ago
> Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed. The problem with this is that while security minded people know what Okta is and why to stay the fuck away from handing over your crown jewels to a SaaS company is warranted, C-level execs don't care. They only care about their golf course or backroom deal friends and about releasing PR statements full of buzzwords like "zero trust", "AI based monitoring" and whatever. The stock markets don't care either, they only look at the financial data, and as long as there still are enough gullible fools signing up, they don't care and stonk goes up.
- cqqxo4zV46cp 3y ago
  Yes, that’s literally the point being made. The point is that it isn’t a company-ending event. You are going on an unrelated rant about how those darn dumb executives aren’t as smart as God’s gift to earth, engineers.
  - mschuster91 3y ago
    The thing is, some events should be company ending. Something like Okta shouldn't even exist in a halfway competent world in the first place - given how many Fortune 500 companies, even governments use it, it's just a too fucking juicy target for nation states both friendly and hostile. Instead, even the "self correcting" mechanisms of the "free market" obviously didn't work out, as the free market doesn't value technical merit, it only values financial bullshittery. And the end result will be that once the war with China or Russia inevitably breaks out, virtually all major Western companies and governments will be out cold for weeks once Okta and Azure's AD go down, because that is where any adversary will hit first to deal immense damage.

nolok 3y ago

> It would be a company ending event Given they got out of cloudbleed without any real damage let alone lasting damage, I disagree. (I don't disagree with your point about how bad of a problem this would be, I'm just insisting that security failure is not taken seriously at all by anyone)

Hrundi 3y ago
I don't remember any companies that ended thanks to cloudbleed, but I'd be happy to be proven wrong
chx 3y ago
Presuming taviso is not exaggerating and why would he CF's reply to cloudbleed was ... not quite nice. https://twitter.com/taviso/status/1566077115992133634 https://twitter.com/taviso/status/1566077115992133634 > True story: After cloudbleed, cloudflare literally lobbied the FTC to investigate me and question the legality of openly discussing security research. How come they're not lobbying their DC friends to investigate the legality KF? For those not familiar with the history , this tweet started the cloudbleed disclosure to cloudflare: https://twitter.com/taviso/status/832744397800214528 https://twitter.com/taviso/status/832744397800214528 > Could someone from cloudflare security urgently contact me. This followed: https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug https://blog.cloudflare.com/incident-report-on-memory-leak-c...
- cowsandmilk 3y ago
  I love this quote: > However, Server-Side Excludes are rarely used and only activated for malicious IP addresses. So… you’re celebrating that you only had buffer overruns for malicious IP addresses?
- hughesjj 3y ago
  Yeah cloudflare is pretty sketchy too imo. They present as transparent but they've had some actions over the years that signal otherwise. Heck, pretty much every performance blog post they hype up buries the caveats, kinda reminiscent of Intel always using their custom cpp compiler for benchmarks. Not technically lying, but definitely omitting some context.

sneak 3y ago

> Imagine getting in at the centre of a new Meet-Me room and having persistent access to key switches there. This wouldn't get you much. We already assume the network is insecure. This is why TLS is a thing (and mTLS for those who are serious).

noizejoy 3y ago
> We already assume the network is insecure. Maybe naively, I wish this assumption became universal.
sophacles 3y ago
I suspect "we" is a much smaller group than you imagine. I've gotten pcaps from customers as recently as this year that include unencrypted financial transaction data. These were captured on a router, not an end host, so the traffic was going across the client's network raw.

meowface 3y ago

>Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. I can just imagine the attackers licking their lips when they first breached the data center. Good reminder to use "Full (Strict)" SSL in Cloudflare. Then even if they do get compromised, your reverse-proxied traffic still won't be readable. (Of course other things you might use Cloudflare for could be vectors, though.)

ownagefool 3y ago
Cloudflare is essentially a massive mitm proxy. If you manage to pwn a key, you have access to traffic. I'm sure they're better than this than me, but ipxe & tftp are plain text, so it wouldn't be shocking if something in the bootstrap process was plaintext. At the very least you need to tell the server what to trust.
- meowface 3y ago
  >If you manage to pwn a key, you have access to traffic. That's why I mentioned "Full (Strict)" SSL. If you configure this in Cloudflare then the entire user <-> Cloudflare <-> origin path is encrypted and attackers can't snoop on the plaintext even if they have access. They'll get some metadata, but every ISP in the world gets that at all times anyway.
  - is39 3y ago
    While both client and origin network connections are encrypted with "Full (Strict)" SSL mode, Cloudflare proxy in the middle decrypts client traffic and then encrypts it towards the server (and vice versa). It does have access to plaintext, which is how various mitigations work. So it's indeed MITM proxy, by design.
    - meowface 3y ago
      Ah, yeah, you're right.