13 ms·

To be fair to Clouldflare, they do a lot of very thoughtful work on sandboxing at all levels, including at the process level as well as above and below that. A

by azakai 4y ago

To be fair to Clouldflare, they do a lot of very thoughtful work on sandboxing at all levels, including at the process level as well as above and below that. A lot of interesting details here: https://blog.cloudflare.com/mitigating-spectre-and-other-security-threats-the-cloudflare-workers-security-model/ https://blog.cloudflare.com/mitigating-spectre-and-other-sec... The article focuses on Spectre, but it also mentions general stuff. edit: It is true they are using V8 for a novel purpose, but a lot of their techniques mirror how Chrome sandboxes V8 (process sandboxing, an outer OS sandbox, etc.).

jeremyjh 4y ago
I have a lot of respect for Kenton, but the facts are: v8 security team recommends running untrusted code in separate processes [1], and as described in your link, Cloudflare doesn't do that. From what Kenton said earlier on this[2], it sounded like they had already committed to their architecture before that v8 recommendation was made. They thought through a lot of possible attacks and created some clever mitigations, and ultimately decided "its probably ok". [1] https://v8.dev/docs/untrusted-code-mitigations#sandbox-untrusted-execution-in-a-separate-process https://v8.dev/docs/untrusted-code-mitigations#sandbox-untru... [2] https://news.ycombinator.com/item?id=18280061 https://news.ycombinator.com/item?id=18280061 eta: Also, Chrome does use process isolation - every browser tab runs in a separate process. The whole point of Cloudflare isolates is to avoid the overhead of a fork() for each worker request.
- hinkley 4y ago
  Also Cloudflare has already broken the Internet once by doing unorthodox things. Sometimes the guy missing a finger is exactly the one you want to take safety advice from. Sometimes it's just a matter of time before he's missing three fingers.
- skybrian 4y ago
  I don't think what Chrome is doing is quite that simple? When you click on a link in a tab, it might take you to a different site. Also, there can be iframes from different websites in the same browser tab. Also see "limitations" on this page: https://www.chromium.org/Home/chromium-security/site-isolation/ https://www.chromium.org/Home/chromium-security/site-isolati...
  - jeremyjh 4y ago
    That link tells you this: > Cross-site documents are always put into a different process, whether the navigation is in the current tab, a new tab, or an iframe (i.e., one web page embedded inside another). Note that only a subset of sites are isolated on Android, to reduce overhead.
- kentonv 4y ago
  Chrome's use of strict process isolation is a fairly new thing. For most of its history, it was trivially easy to open any other site in the same process as your own site (via window.open(), iframe, etc.), and V8's sandboxing was the thing protecting those sites from each other. So V8 was, in fact, designed for this. When Spectre hit, Chrome concluded that defending the web platform from Spectre in userspace was too hard, so they decided to go all-in on process isolation so that Spectre was the kernel's problem instead. This is a great defense-in-depth strategy if you can spare the resources. On most platforms where Chrome runs, the additional overhead was deemed worth it. (Last I heard there were still some lower-powered Android phones where the overhead was too high but that was a while ago, maybe they've moved past that.) That doesn't mean V8 has decided their own security doesn't matter anymore. Securely sandboxing the renderer process as a whole isn't trivial, especially as it interacts with the GPU and whatnot. So it's still very much important to Chrome that V8's sandbox remains tight, and the renderer sandbox provides secondary defense-in-depth. When it comes to Cloudflare Workers, the performance penalty for strict process isolation is much higher than it is for a browser, due to the finer-grained nature of our compute. (Imagine a browser that has 10,000 tabs open and all of them are regularly receiving events, rather than just one being in the foreground...) But we have some advantages: we were able to rethink the platform from the ground up with side channel defense in mind, and we are free to reset the state of any isolate's state any time we need to. That lets us implement a different set of defense-in-depth measures, including dynamic process isolation. More details in the blog post linked earlier [0]. We also did research and co-authored a paper with the TU Graz team (co-discoverers of Spectre) on this [1]. I may be biased, but to be perfectly honest, the security model I find most terrifying is the public clouds that run arbitrary native code in hardware VMs. The VM software may be a narrower attack surface than V8, but the hardware attack surface is gigantic. One misimplemented CPU instruction allowing, say, reading physical addresses without access control could blow up the whole industry overnight and take years to fix. Spectre was a near miss / grazing hit. M1 had a near miss with M1racles [2]. I think it's more likely than not that really disastrous bugs exist in every CPU, and so I feel much, much safer accepting customer code as JavaScript / Wasm rather than native code. [0] https://blog.cloudflare.com/mitigating-spectre-and-other-security-threats-the-cloudflare-workers-security-model/ https://blog.cloudflare.com/mitigating-spectre-and-other-sec... [1] https://blog.cloudflare.com/spectre-research-with-tu-graz/ https://blog.cloudflare.com/spectre-research-with-tu-graz/ [2] https://m1racles.com/ https://m1racles.com/
- azakai 4y ago
  Browser requirements may differ from Cloudflare's, though. In particular, browsers run arbitrary code from anywhere - just clicking a link downloads and runs code off of anywhere on the planet, theoretically - while Cloudflare runs code from their customers. They do have a free tier, so yes, there are risks here. But one mechanism they do according to that post, IIUC, is to keep paying customers separate from the free tier. So if you are a paying customer, any attacker against you is at least someone that Cloudflare has the credit card information of. Still dangerous, but very different than the browser risk model.
  - comex 4y ago
    It’s worse than the browser threat model. In a browser, security-maximalist users can choose to only visit trusted sites, which reduces the attack surface somewhat, though far from completely (there’s still exploits hidden in ads and watering hole attacks to worry about). With CloudFlare, you’re keeping out… people without credit cards? What kind of even remotely serious attacker can’t obtain a stolen credit card, or else sign up for a credit card with false identification? At most the credit card restriction prevents attackers from creating large numbers of accounts, reducing the chance they’ll be co-located with you, but even that can probably be circumvented by sufficiently sophisticated attackers.
    - azakai 4y ago
      You're right that credit cards are far from guarantees. Still, I'd imagine Cloudflare do something like cluster users by risk level at some granularity. Brand-new paying accounts might be put together in the same process, while paying customers for over a year might be in another. You might be attacked even in the latter group, but the attacker can do that at a much lower frequency (the card isn't recently stolen; and they've been paying for this attack for a year - only to be kicked out now if found). Also, the browser risk model is much worse because of the scale. If you manage to get malicious code onto a popular website then you get many, many opportunities to attack people. With Cloudflare you can only attack the other tenants of your process, and you're paying every time you do it (in a browser, it runs on the target's machine, free for you). And even then as the blogpost mentions they randomize pairings, limit the amount of time they spend together, etc. Overall I think this is far less risky than the browser situation. And Cloudflare isn't the only one that thinks so - their competitors also put multiple tenants in the same process. Done carefully, it's a reasonable model.

by azakai 4y ago

jeremyjh 4y ago

I have a lot of respect for Kenton, but the facts are: v8 security team recommends running untrusted code in separate processes [1], and as described in your link, Cloudflare doesn't do that. From what Kenton said earlier on this[2], it sounded like they had already committed to their architecture before that v8 recommendation was made. They thought through a lot of possible attacks and created some clever mitigations, and ultimately decided "its probably ok". [1] https://v8.dev/docs/untrusted-code-mitigations#sandbox-untrusted-execution-in-a-separate-process https://v8.dev/docs/untrusted-code-mitigations#sandbox-untru... [2] https://news.ycombinator.com/item?id=18280061 https://news.ycombinator.com/item?id=18280061 eta: Also, Chrome does use process isolation - every browser tab runs in a separate process. The whole point of Cloudflare isolates is to avoid the overhead of a fork() for each worker request.

hinkley 4y ago
Also Cloudflare has already broken the Internet once by doing unorthodox things. Sometimes the guy missing a finger is exactly the one you want to take safety advice from. Sometimes it's just a matter of time before he's missing three fingers.
skybrian 4y ago
I don't think what Chrome is doing is quite that simple? When you click on a link in a tab, it might take you to a different site. Also, there can be iframes from different websites in the same browser tab. Also see "limitations" on this page: https://www.chromium.org/Home/chromium-security/site-isolation/ https://www.chromium.org/Home/chromium-security/site-isolati...
- jeremyjh 4y ago
  That link tells you this: > Cross-site documents are always put into a different process, whether the navigation is in the current tab, a new tab, or an iframe (i.e., one web page embedded inside another). Note that only a subset of sites are isolated on Android, to reduce overhead.
kentonv 4y ago
Chrome's use of strict process isolation is a fairly new thing. For most of its history, it was trivially easy to open any other site in the same process as your own site (via window.open(), iframe, etc.), and V8's sandboxing was the thing protecting those sites from each other. So V8 was, in fact, designed for this. When Spectre hit, Chrome concluded that defending the web platform from Spectre in userspace was too hard, so they decided to go all-in on process isolation so that Spectre was the kernel's problem instead. This is a great defense-in-depth strategy if you can spare the resources. On most platforms where Chrome runs, the additional overhead was deemed worth it. (Last I heard there were still some lower-powered Android phones where the overhead was too high but that was a while ago, maybe they've moved past that.) That doesn't mean V8 has decided their own security doesn't matter anymore. Securely sandboxing the renderer process as a whole isn't trivial, especially as it interacts with the GPU and whatnot. So it's still very much important to Chrome that V8's sandbox remains tight, and the renderer sandbox provides secondary defense-in-depth. When it comes to Cloudflare Workers, the performance penalty for strict process isolation is much higher than it is for a browser, due to the finer-grained nature of our compute. (Imagine a browser that has 10,000 tabs open and all of them are regularly receiving events, rather than just one being in the foreground...) But we have some advantages: we were able to rethink the platform from the ground up with side channel defense in mind, and we are free to reset the state of any isolate's state any time we need to. That lets us implement a different set of defense-in-depth measures, including dynamic process isolation. More details in the blog post linked earlier [0]. We also did research and co-authored a paper with the TU Graz team (co-discoverers of Spectre) on this [1]. I may be biased, but to be perfectly honest, the security model I find most terrifying is the public clouds that run arbitrary native code in hardware VMs. The VM software may be a narrower attack surface than V8, but the hardware attack surface is gigantic. One misimplemented CPU instruction allowing, say, reading physical addresses without access control could blow up the whole industry overnight and take years to fix. Spectre was a near miss / grazing hit. M1 had a near miss with M1racles [2]. I think it's more likely than not that really disastrous bugs exist in every CPU, and so I feel much, much safer accepting customer code as JavaScript / Wasm rather than native code. [0] https://blog.cloudflare.com/mitigating-spectre-and-other-security-threats-the-cloudflare-workers-security-model/ https://blog.cloudflare.com/mitigating-spectre-and-other-sec... [1] https://blog.cloudflare.com/spectre-research-with-tu-graz/ https://blog.cloudflare.com/spectre-research-with-tu-graz/ [2] https://m1racles.com/ https://m1racles.com/
azakai 4y ago
Browser requirements may differ from Cloudflare's, though. In particular, browsers run arbitrary code from anywhere - just clicking a link downloads and runs code off of anywhere on the planet, theoretically - while Cloudflare runs code from their customers. They do have a free tier, so yes, there are risks here. But one mechanism they do according to that post, IIUC, is to keep paying customers separate from the free tier. So if you are a paying customer, any attacker against you is at least someone that Cloudflare has the credit card information of. Still dangerous, but very different than the browser risk model.
- comex 4y ago
  It’s worse than the browser threat model. In a browser, security-maximalist users can choose to only visit trusted sites, which reduces the attack surface somewhat, though far from completely (there’s still exploits hidden in ads and watering hole attacks to worry about). With CloudFlare, you’re keeping out… people without credit cards? What kind of even remotely serious attacker can’t obtain a stolen credit card, or else sign up for a credit card with false identification? At most the credit card restriction prevents attackers from creating large numbers of accounts, reducing the chance they’ll be co-located with you, but even that can probably be circumvented by sufficiently sophisticated attackers.
  - azakai 4y ago
    You're right that credit cards are far from guarantees. Still, I'd imagine Cloudflare do something like cluster users by risk level at some granularity. Brand-new paying accounts might be put together in the same process, while paying customers for over a year might be in another. You might be attacked even in the latter group, but the attacker can do that at a much lower frequency (the card isn't recently stolen; and they've been paying for this attack for a year - only to be kicked out now if found). Also, the browser risk model is much worse because of the scale. If you manage to get malicious code onto a popular website then you get many, many opportunities to attack people. With Cloudflare you can only attack the other tenants of your process, and you're paying every time you do it (in a browser, it runs on the target's machine, free for you). And even then as the blogpost mentions they randomize pairings, limit the amount of time they spend together, etc. Overall I think this is far less risky than the browser situation. And Cloudflare isn't the only one that thinks so - their competitors also put multiple tenants in the same process. Done carefully, it's a reasonable model.