5 ms·
Ask HN: Why is there not more concern about the physical security of Cloudflare?
Using Hetzner and Azure, we trust that our unencrypted in-memory data and business logic are housed in professional data centers with strong physical security measures. However, Cloudflare has built its Workers and serverless offerings on top of its Cache/CDN and anti-DDoS infrastructure, which operates out of questionable ISP and IXP colocation facilities in various jurisdictions with dubious standards.
As an EU-based company, whenever we ask Cloudflare about the physical security of their edge locations, they consistently refer to encryption in transit and at rest—measures that do nothing to address threats like RAM interception or other physical security vulnerabilities in these questionable facilities. Moreover, when we raise these concerns, they attempt to upsell us on their Enterprise EU/FedRAMP offerings. Cloudflare has also deliberately restricted our ability to block non-Enterprise Workers, KV, and R2 from specific regions, leaving us with limited control over where our data is processed.
- pestatije 2y agoultimately you trust the company and the jurisdiction in which they operate...if they give you the wrong answers then you should adjust your level of trust accordingly...thankfully there are other platforms and this is one concern that can certainly be better marketed
- viraptor 2y agoWhile this is mostly true, the "trust the jurisdiction" is hard to apply here. They operate their PoPs in over 120 countries, so it's close to "do you trust all countries?" - it's a completely different model than trusting an AWS region for example.
- lfmunoz4 2y agonever heard of a story where physical security at any cloud provider has been a problem. are you worried about governments, or employees, or someone breaking in?
- altdataseller 2y agoJust because there hasnt been a story doesnt mean its not important or critical
- oceanplexian 2y agoWhen I worked at AWS they were insanely hardcore about mandating physical access controls, that is all I’ll say, even to the point of ridiculousness. For all the things AWS does poorly security is not one of them. If I were to guess CF is locating their PoPs at cheap peering points and the reason they are evading the question is because other customers in the facility have physical access to their equipment, which is both an expensive problem to solve and something that is not even remotely allowed at a real cloud provider.
- Alupis 2y agoThis seems unlikely to me, at least in the US. Even your cheapest of colo's offer locked cage areas. For someone on Cloudflare's scale, the cost is trivial. I've been inside some really "low rent" colo's and even they would provide an escort to unlock your cabinet. Obviously standards/expectations will vary from DC to DC. I'd wager the situation might be different in some of the smaller countries CF operates in around the world though.
- joelhaasnoot 2y agocough https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire-france-data-center/ https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire... (physical security isn't just access control)
- zendist 2y agoThat's definitely a thing. Additionally, humans are surprisingly friendly in all the wrong ways when it comes to physical security (tailgating, "forgotten ID/credentials", etc.).
- simonw 2y agoIt's interesting to explore https://where.durableobjects.live/ https://where.durableobjects.live/ - a tool that maps where Cloudflare's worker scripts actually run. Notably, while Cloudflare has CDN edge locations in countries like China and Russia they don't appear to run workers there. EDIT: I was wrong - I misinterpreted the map. A solid border circle around a location indicates "Worker-only Datacenter" (see the map legend) and there are indeed locations with those solid borders in Russia (including Moscow and Yekaterinburg) and China (Haidong, Lanzhou and more). I doubt we could get them on the record for this, but I suspect this may be very deliberate. Maybe CDN edge locations can be run completely securely with forwarded encrypted traffic, while workers are at a higher risk of physical attack.
- victorbjorklund 2y agoNice map but are you sure it overlaps with Workers in general? Workers can run without DO.
- chaikasumu 2y agoThat tool maps where Durable Objects run, not Workers. Durable Objects run in more limited locations for their "Durableness". For example, writes have to be confirmed by multiple nearby data centers, if a location goes offline they have to be able to go to a near one, etc. So they basically have their own clusters/regions. Workers run everywhere. On every location shown on that map. Granted, you need a special license and Enterprise feature to use the China ones.
- simonw 2y agoYeah I got that wrong - see edit to my comment, there are indeed circles with thick borders indicating "Worker-only Datacenter" in China and Russia on that map.
- deleted 2y ago[deleted]
- dtquad 2y agoRecently, many European countries have started using Cloudflare’s anti-DDoS protection for government services, which is fine since it can be done securely with end-to-end encryption in a zero-trust model. However, these countries are now increasingly using Cloudflare Workers without realizing that this creates total uncertainty about where their data and business logic might end up—including potentially adversarial countries like Russia and China. Data in transit and data at rest can be encrypted. But Clouflare Workers is neither. It's data being processed in-memory where it will be unencrypted.
- arccy 2y agoWhat's the threat model that ram interception is an issue? I think the upsell is entirely reasonable, you get charged more for weird compliance demands.
- candiddevmike 2y agoWhat's a threat model where RAM intercept wouldn't be an issue?
- udev4096 2y agoIsn't it difficult to actually analyze the RAM content for any sensitive data? There is so much noise in it
- foota 2y agoI don't think it'd be too hard to find something like private RSA keys in memory, for example.
- viraptor 2y agoA bit hard, but not crazy. You have to potentially deal with not knowing the mappings, but even random blocks will have obvious marks like Json fragments with "password=", crypto keys with common PEM/DER headers, high entropy 8/16/32-byte data, etc.
- _hyn3 2y agoFor fun, sometime, try this: sudo strings /dev/sda | more # press q to quit You'll immediately see plain text of various things on your hard drive, complete HTML, etc. Eye opening.
- g-b-r 2y agoNo it isn't? I recovered data of mine from ram many times
- foota 2y ago
- kstrauser 2y agoI don't use Cloudflare and I don't have strong opinions for or against them. These questions are the same I'd ask if you were discussing any other company: > which operates out of questionable ISP and IXP colocation facilities in various jurisdictions with dubious standards. Why do you say that? Do you have signals that their colo facilities are less secure than they should be, and/or that Cloudflare hasn't gotten those facilities to beef up their security as part of their contract? Again, not saying this to defend Cloudflare. I just hadn't heard this before. > Moreover, when we raise these concerns, they attempt to upsell us on their Enterprise EU/FedRAMP offerings. That's going to be the case with almost all providers in the space. If you're asking for special treatment, you're going to have to pay for it. I don't mean that to insult you. At a past job, for various reasons we had strict compliance obligations that our data could not be accessed outside of the US. Some of our vendors used offshore tech support who'd have access to our data, and a couple times we faced a decision: pay that vendor $$$ to special-case our support setup to meet our requirements, or choose another vendor. > Cloudflare has also deliberately restricted our ability to block non-Enterprise Workers, KV, and R2 from specific regions, leaving us with limited control over where our data is processed. Same. Those fine-grained controls are often going to be an enterprise feature. Again, I'm not saying this to defend Cloudflare in particular. They have their own paid spokespeople. I'm not one. Nothing you've said sounds particularly egregious though. If your data is sensitive enough that you're legitimately worried about someone sneaking in undetected and intercepting RAM, prepare to pay the enterprise tax with all your cloud vendors.
- iwontberude 2y agoWhy not use your own datacenter if it’s that important to ensure physical security? If the business can’t justify it, then maybe your use case isn’t that sensitive or important. Or maybe the business isn’t viable.
- advisedwang 2y agoBecause edge serving has numerous advantages the OP would presumably like to take advantage of. If you have customers all over the world, "using your own datacenter" either means operating tens of datacenters or giving some of your customers high latency.
- organsnyder 2y agoThen share the nonsensitive assets via a CDN, and host a separate sensitive endpoint (with higher latency) in a more controlled environment.
- iwontberude 2y agoI think anything which can be delivered to the web browser from CloudFlare could be part of the kill chain for a eager attacker.
- organsnyder 2y agoIf that's your threat model, then you either need to build your own CDN (from the data center up) or find a vendor willing to sell you the features you need. It won't be cheap.
- xet7 2y agoSome of my own thoughts about this. I don't know, I don't work at CloudFlare. Anyway: > infrastructure, which operates out of questionable ISP and IXP colocation facilities in various jurisdictions with dubious standards What are these questionable facilities? When CloudFlare has installed those servers and software, and they have also made software that manages those servers, what is the problem? CloudFlare has written some articles about some of those very many security protections they have. But that is a very lot of technical detail to explain, so if that is so important, their Enterprise/FedRAMP offerings fund CloudFlare to make possible to explain that amount of detail. But question is, do you really need that amount of detail? How much you have expertise to build same amount of security protections? Isn't it better to use CloudFlare Workers security features to concentrate on building your app? Alternative is to get your own bare metal servers, and manage them yourself. With CloudFlare Workers, they have security features to keep code and data of each customer separate from each other.
- foota 2y agoThe issue here is there's sort of a missing middle. The main cloud provides offer physical security without you needing fedRAMP etc.,.
- spr-alex 2y agoOn Azure -- they've been playing catch up this year after repeated congressional inquiries from breaches. It's only in 2024 that Azure has started to build a better device inventory on their infrastructure networks and started doing appropriate employee access control mechanisms. Here is Satya's May Post, https://blogs.microsoft.com/blog/2024/05/03/prioritizing-security-above-all-else/ https://blogs.microsoft.com/blog/2024/05/03/prioritizing-sec...
- nijave 2y agoCurious how they protect private keys for transit encryption. I'd imagine all their edge locations need to be able to decrypt traffic and therefor need a way to fetch corresponding private keys for everything they're proxying. If physical theft is a concern, how do they prevent someone from hijacking the key distribution process?
- deleted 2y ago[deleted]
- sirspacey 2y agoSounds like you want an elevated level of security that cloudflare provides but you don’t want to purchase?
- crote 2y agoIsn't this basically why modern server CPUs support Secure Boot and memory encryption? If I understand it correctly, it should be possible to set up your machines in such a way that 1) the server can only boot genuine firmware images, and 2) unencrypted data is only available inside the CPU itself. That is going to rule out most attacks which don't involve dragging the server into a high-end laboratory, is it not?
- rollcat 2y agoTangential: how does memory encryption mesh with features like DMA (which I imagine would be key on high-throughput NICs)?
- bri3d 2y agoFor DMA, DMA endpoint buffers will generally be unencrypted and live in the "host domain," outside of the secure enclave running trusted code. This has been used to construct exploits against AMD's Secure Virtualization (which relies on memory encryption), for example: https://www.usenix.org/system/files/sec19-li-mengyuan_0.pdf https://www.usenix.org/system/files/sec19-li-mengyuan_0.pdf . There are some emerging technologies like SEV-TIO and PCI-IDE which attempt to extend the trust chain down into PCIe devices, allowing trusted devices to form a relationship with trusted code in a secure execution environment: https://www.amd.com/content/dam/amd/en/documents/developer/sev-tio-whitepaper.pdf https://www.amd.com/content/dam/amd/en/documents/developer/s... Anyway, the best protection here is to terminate protocol or application-level encryption (TLS or whatever sits on top of your protocol) inside of the secure environment, so that decryption only occurs through encrypted memory. It's slower, but this way an attacker sitting as a RAM snoop could only see encrypted network traffic hit the DMA buffers, which wouldn't help them much.
- tredre3 2y agoThis answer covers it well: https://security.stackexchange.com/a/189978 https://security.stackexchange.com/a/189978 Basically depending on the scenario, DMA region will either be unencrypted or transparently encrypted/decrypted.
- pornel 2y agoCloudflare uses AMD TSME to encrypt the RAM: https://blog.cloudflare.com/securing-memory-at-epyc-scale/ https://blog.cloudflare.com/securing-memory-at-epyc-scale/
- kevlened 2y agoThis was posted with their Gen 10 servers in 2020. As of Sept 2024, they're on Gen 12 [https://blog.cloudflare.com/gen-12-servers/ https://blog.cloudflare.com/gen-12-servers/]. > Gen 12 Servers are currently deployed and live in multiple Cloudflare data centers worldwide Judging from this, the fleet isn't uniform. How uniform is the fleet? Is it possible to be running on Gen 9 in some locations?
- andrewaylett 2y agoCloudflare have some really interesting tech, like https://blog.cloudflare.com/announcing-keyless-ssl-all-the-benefits-of-cloudflare-without-having-to-turn-over-your-private-ssl-keys/ https://blog.cloudflare.com/announcing-keyless-ssl-all-the-b..., that I really wish we had an open implementation of so I could do similar myself.
- exe34 2y ago> Moreover, when we raise these concerns, they attempt to upsell us on their Enterprise EU/FedRAMP offerings so you want the cheap plan with the features from the premium plan?
- sarlalian 2y agoThey have some docs on the security of their server hardware, which is one of the ways they can assure that their systems haven't been tampered with by local staff. https://blog.cloudflare.com/anchoring-trust-a-hardware-secure-boot-story/ https://blog.cloudflare.com/anchoring-trust-a-hardware-secur...
- amalcon 2y agoThere are, roughly, four kinds of physical security one can have in a datacenter: 1) Nothing 2) Visitor logs 3) Locks and alarms on your racks, and/or (if you have enough) the rooms they are in. Remote monitoring is pretty common. 4) First party human security professionals I don't have any special knowledge of Cloudflare's set-up, but 2 and 3 are by far the most common. Lacking #1 means your hardware just gets stolen. #4 is too expensive. #2 and #3 are where most people are, so probably something around there?
- cqiDFbfRUggaQph 2y agoWhat about other providers like Fastly and Akamai? I've not looked into this but maybe they do a better job?
- red-iron-pine 2y ago> which operates out of questionable ISP and IXP colocation facilities in various jurisdictions with dubious standards. what does this mean?
- dorristim6 2y ago[dead]