6 ms·

An information leak is an information leak : we still fail to realise that it's something that's happening daily. There's no drama in it. Criminals are taking

An information leak is an information leak : we still fail to realise that it's something that's happening daily. There's no drama in it. Criminals are taking advantage of opportunities like this every day, still no one cares too much about it (HN bubble & friends excluded). Things like this may have a strong impact or not in the press/popularity circus, but in this particular case it seems they promptly monitored the situation (thanks to their competent staff). What most surprises me is that their highly competent staff is thoughtlessly violating one of the security principles in sw : SECURITY BY ISOLATION . No one (no matter how able you are) can write absolutely bug-free algorithms : even when dealing with formal verified software you can still attack the assumptions. Security by correctness is a laudable effort, but computing customers data with a single process is not sane. I'm aware they're doing this for performance reasons, but a well implemented isolation layer would have prevented this (even while dealing with a bug like that). Their architecture is vulnerable.

jsmeaton 10y ago
> What most surprises me is that their highly competent staff is thoughtlessly violating one of the security principles in sw : SECURITY BY ISOLATION . I don't think this is really true, but I'm open to hearing your thoughts on this. There was a bug in their HTML parser which caused unrelated memory to be dumped to the process. Their SSL termination servers were isolated elsewhere which is why SSL keys weren't dumped into public caches. Where would you like them to draw the isolation boundary? Per function? Per rule? Per service? From what I understand, these processes were a part of a single service, but not every request was using each type of rule. Even if they'd only had customers using their html parser isolated on separate servers, other customers would have been affected even if their HTML was perfectly valid according to the parser.
- icebraining 10y ago
  I think the implication is that the isolation should be per customer, each being allocated their own parsing process, isolated from the other customers. That's roughly what we do, though we run an hosted version of an open source webapp, not a CDN. It's more expensive resource-wise (particularly RAM), but it has meant that we were immune to 90%+ of the security bugs discovered in the platform.
  - jsmeaton 10y ago
    Sure, that's a valid question to ask. But imagine you have 1,000,000 customers. Now you have to calculate and manage scaling groups for 1,000,000 customers * number of services. The resourcing costs alone would be outlandish, not to mention trying to independently scale each customer. Perhaps container systems would make this easier, but do they have better memory isolation? Is it possible for a container process to overrun into another containers memory without an exploit in the container system?
    - icebraining 10y ago
      A container is a process with some extra isolation (namespaces), they certainly can't overrun into each other without an exploit. Why would the costs be outlandish? We offer that and we're fairly cheap. Since the cost is mostly fixed per customer, it should scale linearly. As for scaling, they already have to do that, by pointing different requests at different servers depending on their load, etc.
      - jsmeaton 10y ago
        Assuming for a minute that containers aren't in play, then the isolation model becomes that of a server/vm with the associated overhead of each. To make this easier we'll assume there's only a single service, even though we know this to be untrue. If there are 1M customers that's a minimum of 1M servers. Some customers are obviously larger and would need more. There's also HA. Let's conservatively call it 2.5M servers. At an absolute bare minimum we'd need to allocate 2.5M GB of ram and 2.5M vCPU. That's a huge amount of resources. If you could reliably fit 10000 Small customers on a single server at 32gb ram and 8 cpus you can already start to see how many resources can be saved. Without customer isolation you've got the entire cluster to handle load spikes and HA. With isolation you have to have scheduling monitoring each of the 1M clusters and scaling appropriately by anticipating demand. Scaling a service is way easier than scaling customers within a service or many services.
        tgragnato 10y ago
- tgragnato 10y ago
  I read their post "incident report" too . Isolation is claimed, but if I understand correctly, HTTP handling is shared between customers. Am I wrong ? Suppose you're a "bad actor", knowing this is a shared service, wouldn't you look for 0 days in it ? A carefully crafted exploit has the potential to leak specific content from unaware customers again. The attack surface is nginx (http://nginx.org/en/security_advisories.html http://nginx.org/en/security_advisories.html) plus each component of each loaded module ... It would be saner to apply isolation to each element of the cartesian product between customers and services. The performance (and cost) impact can be mitigated by scheduling resources over a pool of disposable virtual machines (obviously in xen and with iommu protection), but I bet they can develop even better solutions.
  - jsmeaton 10y ago
    Even if there were a set of VMs per customer (and all the scaling per customer overhead that goes along with that), a carefully crafted exploit would still reveal details for that customer. Then it'd be a matter of enumerating all of the customers you were interested in exploiting, which would make it easier to get data for a specific target. The operational overhead of VM/Container isolation for the cartesian product of customer + service sounds like it'd be extremely prohibitive. It's certainly a tradeoff, but to claim it's saner is missing all of the other costs associated with such a system.
    - tgragnato 10y ago
      Yeah it makes a targeted attack easier. But it prevents attacks across different customers. Tradeoffs ... Maintaining a running pool of VMs per service in a sufficient number to serve the load of requests grouped by customer, and assigning the VM to a specific customer only at needs is different than running permanently a pool of (n).customers x (m).services VMs. This is why an efficient scheduler and the usage of disposable VMs is a need. Still depending on the load and the variety of the traffic it may not be feasible, you are absolutely right ! Another approach to ensure isolation is the usage of a MAC framework. As I wrote "I bet they can develop even better solutions" ;)
    - jacquesm 10y ago
predakanga 10y ago
It's not much to go on, but it seems they are looking into better isolation going forward: https://disqus.com/home/discussion/cloudflare/incident_report_on_memory_leak_caused_by_cloudflare_parser_bug_11/#comment-3172534987 https://disqus.com/home/discussion/cloudflare/incident_repor...
- jgrahamc 10y ago
  We'll be sure to write up details of things we have changed. It's ongoing.

jsmeaton 10y ago

> What most surprises me is that their highly competent staff is thoughtlessly violating one of the security principles in sw : SECURITY BY ISOLATION . I don't think this is really true, but I'm open to hearing your thoughts on this. There was a bug in their HTML parser which caused unrelated memory to be dumped to the process. Their SSL termination servers were isolated elsewhere which is why SSL keys weren't dumped into public caches. Where would you like them to draw the isolation boundary? Per function? Per rule? Per service? From what I understand, these processes were a part of a single service, but not every request was using each type of rule. Even if they'd only had customers using their html parser isolated on separate servers, other customers would have been affected even if their HTML was perfectly valid according to the parser.

icebraining 10y ago
I think the implication is that the isolation should be per customer, each being allocated their own parsing process, isolated from the other customers. That's roughly what we do, though we run an hosted version of an open source webapp, not a CDN. It's more expensive resource-wise (particularly RAM), but it has meant that we were immune to 90%+ of the security bugs discovered in the platform.
- jsmeaton 10y ago
  Sure, that's a valid question to ask. But imagine you have 1,000,000 customers. Now you have to calculate and manage scaling groups for 1,000,000 customers * number of services. The resourcing costs alone would be outlandish, not to mention trying to independently scale each customer. Perhaps container systems would make this easier, but do they have better memory isolation? Is it possible for a container process to overrun into another containers memory without an exploit in the container system?
  - icebraining 10y ago
    A container is a process with some extra isolation (namespaces), they certainly can't overrun into each other without an exploit. Why would the costs be outlandish? We offer that and we're fairly cheap. Since the cost is mostly fixed per customer, it should scale linearly. As for scaling, they already have to do that, by pointing different requests at different servers depending on their load, etc.
    - jsmeaton 10y ago
      Assuming for a minute that containers aren't in play, then the isolation model becomes that of a server/vm with the associated overhead of each. To make this easier we'll assume there's only a single service, even though we know this to be untrue. If there are 1M customers that's a minimum of 1M servers. Some customers are obviously larger and would need more. There's also HA. Let's conservatively call it 2.5M servers. At an absolute bare minimum we'd need to allocate 2.5M GB of ram and 2.5M vCPU. That's a huge amount of resources. If you could reliably fit 10000 Small customers on a single server at 32gb ram and 8 cpus you can already start to see how many resources can be saved. Without customer isolation you've got the entire cluster to handle load spikes and HA. With isolation you have to have scheduling monitoring each of the 1M clusters and scaling appropriately by anticipating demand. Scaling a service is way easier than scaling customers within a service or many services.
      - tgragnato 10y ago
tgragnato 10y ago
I read their post "incident report" too . Isolation is claimed, but if I understand correctly, HTTP handling is shared between customers. Am I wrong ? Suppose you're a "bad actor", knowing this is a shared service, wouldn't you look for 0 days in it ? A carefully crafted exploit has the potential to leak specific content from unaware customers again. The attack surface is nginx (http://nginx.org/en/security_advisories.html http://nginx.org/en/security_advisories.html) plus each component of each loaded module ... It would be saner to apply isolation to each element of the cartesian product between customers and services. The performance (and cost) impact can be mitigated by scheduling resources over a pool of disposable virtual machines (obviously in xen and with iommu protection), but I bet they can develop even better solutions.
- jsmeaton 10y ago
  Even if there were a set of VMs per customer (and all the scaling per customer overhead that goes along with that), a carefully crafted exploit would still reveal details for that customer. Then it'd be a matter of enumerating all of the customers you were interested in exploiting, which would make it easier to get data for a specific target. The operational overhead of VM/Container isolation for the cartesian product of customer + service sounds like it'd be extremely prohibitive. It's certainly a tradeoff, but to claim it's saner is missing all of the other costs associated with such a system.
  - tgragnato 10y ago
    Yeah it makes a targeted attack easier. But it prevents attacks across different customers. Tradeoffs ... Maintaining a running pool of VMs per service in a sufficient number to serve the load of requests grouped by customer, and assigning the VM to a specific customer only at needs is different than running permanently a pool of (n).customers x (m).services VMs. This is why an efficient scheduler and the usage of disposable VMs is a need. Still depending on the load and the variety of the traffic it may not be feasible, you are absolutely right ! Another approach to ensure isolation is the usage of a MAC framework. As I wrote "I bet they can develop even better solutions" ;)
  - jacquesm 10y ago

predakanga 10y ago

It's not much to go on, but it seems they are looking into better isolation going forward: https://disqus.com/home/discussion/cloudflare/incident_report_on_memory_leak_caused_by_cloudflare_parser_bug_11/#comment-3172534987 https://disqus.com/home/discussion/cloudflare/incident_repor...

jgrahamc 10y ago
We'll be sure to write up details of things we have changed. It's ongoing.