4 ms·

I agree that WASM is only really interested for embedding untrusted code, but am not sure that the browser is the only place where you need that. A good example

I agree that WASM is only really interested for embedding untrusted code, but am not sure that the browser is the only place where you need that. A good example is Cloudflare's workers, which they talked about at the last SF Rust Meetup. Once you get a _good_ solution to embedding untrusted code, you might find it turns out to be pretty useful.

kllrnohj 8y ago
Do you really need embedded code for things like Cloudflare's workers, though? It doesn't seem like it does, seems like each worker can just be its own process using normal process isolation mechanisms. Switching to embedded untrusted code for things like that would actually make a lot of stuff a lot harder - scheduling, resource monitoring, etc... would all need to be re-invented from scratch. Is it really useful to re-invent the kernel in userspace to use a bunch of embedded webasm contexts instead of just a bunch of processes?
- kentonv 8y ago
  Hi. Cloudflare Workers tech lead here. No, process isolation does not work for us, because it does not scale well enough. The point of Cloudflare Workers is to distribute your code across Cloudflare's entire edge network -- 154 locations and growing -- so that you can respond to requests at the closest location to your end user. We want to put every customer's code in every location, rather than forcing you to choose a handful. Meanwhile, obviously, not every one of our locations is a mega-datacenter. This means we need a way to support extremely large numbers of tenants per machine, with relatively low traffic per tenant (because each tenant's traffic is spread out over the world). We need this to be efficient. This means, we need low memory overhead (to support many tenants per machine), very fast cold-start time (since cold starts are a lot more common in a decentralized scenario), and low overhead to switch between workers (because we're mixing traffic from all our customers everywhere). Using embedded V8 isolates within a single process gives us 10x-1000x better performance on all these metrics than if we gave a whole process to each customer. Relatedly, here's a talk I gave at Heavybit about the neat things you can do when you have fine-grained server compute: https://www.youtube.com/watch?v=YZSvJNBZsxg https://www.youtube.com/watch?v=YZSvJNBZsxg
  - jeremyjh 8y ago
    V8's developer guide doesn't recommend running untrusted code in a shared process with "sensitive" data. How do you know if your different customers data is sensitive? https://v8.dev/docs/untrusted-code-mitigations#sandbox-untrusted-execution-in-a-separate-process https://v8.dev/docs/untrusted-code-mitigations#sandbox-untru...
    - kentonv 8y ago
      This is a relatively new recommendation on V8's part specifically in response to Spectre-type vulnerabilities. We've spent a lot of time thinking about and building mitigations for speculative side channel attacks. For example, early on in the project -- before anyone even knew about Spectre -- we made the decision that `Date.now()` would not advance during code execution, only when waiting for I/O. So, a tight loop that calls `Date.now()` repeatedly will keep getting the same value returned. We did this to mitigate timing side channels -- again, even though we didn't know about Spectre yet at the time. Chrome has indeed stated that they believe process isolation is the only mitigation that will work for them. However, this statement is rather specific to the browser environment. The DOM API is gigantic, and it contains many different sources of non-determinism, including several explicit timers as well as concurrent operations (e.g. layout, rendering, etc.). Side channel attacks are necessarily dependent on non-determinism; a fully-deterministic environment essentially by definition has no covert side channels. But, there's no way Chrome can get there. The Cloudflare Workers environment is very different. The only kinds of I/O available to a worker are HTTP in, HTTP out, `Date.now()`, and `crypto.getRandomValues()`. Everything else is perfectly deterministic. So, for us, the problem is much narrower. We need to make sure those four inputs cannot effectively be leveraged into a side channel attack. This is still by no means trivial, but unlike in the browser, it's feasible. `getRandomValues()` is not useful to an attacker because it is completely non-deterministic. `Date.now()` we've already locked down as mentioned above. HTTP in/out can potentially be leveraged to provide external timers -- but the network is extremely noisy. A practical attack would require a lot of time in order to average out the noise -- enough time that we can do a bunch of higher-level things to detect and disrupt possible attacks. It helps that Workers are stateless, so we can reset a worker at any time and move it around, which makes attacks harder. NetSpectre demonstrated that even physical network separation does not necessarily protect you against Spectre attacks. There's simply no such thing as a system that's perfectly secure against Spectre, process isolation or not. All we can do -- aside from going full BSG and giving up on networks altogether -- is make attacks harder to the point of infeasibility. Luckily, we have lots of tools in our toolbox for making Spectre attacks infeasible in the case of Cloudflare Workers.

kllrnohj 8y ago

Do you really need embedded code for things like Cloudflare's workers, though? It doesn't seem like it does, seems like each worker can just be its own process using normal process isolation mechanisms. Switching to embedded untrusted code for things like that would actually make a lot of stuff a lot harder - scheduling, resource monitoring, etc... would all need to be re-invented from scratch. Is it really useful to re-invent the kernel in userspace to use a bunch of embedded webasm contexts instead of just a bunch of processes?

kentonv 8y ago
Hi. Cloudflare Workers tech lead here. No, process isolation does not work for us, because it does not scale well enough. The point of Cloudflare Workers is to distribute your code across Cloudflare's entire edge network -- 154 locations and growing -- so that you can respond to requests at the closest location to your end user. We want to put every customer's code in every location, rather than forcing you to choose a handful. Meanwhile, obviously, not every one of our locations is a mega-datacenter. This means we need a way to support extremely large numbers of tenants per machine, with relatively low traffic per tenant (because each tenant's traffic is spread out over the world). We need this to be efficient. This means, we need low memory overhead (to support many tenants per machine), very fast cold-start time (since cold starts are a lot more common in a decentralized scenario), and low overhead to switch between workers (because we're mixing traffic from all our customers everywhere). Using embedded V8 isolates within a single process gives us 10x-1000x better performance on all these metrics than if we gave a whole process to each customer. Relatedly, here's a talk I gave at Heavybit about the neat things you can do when you have fine-grained server compute: https://www.youtube.com/watch?v=YZSvJNBZsxg https://www.youtube.com/watch?v=YZSvJNBZsxg
- jeremyjh 8y ago
  V8's developer guide doesn't recommend running untrusted code in a shared process with "sensitive" data. How do you know if your different customers data is sensitive? https://v8.dev/docs/untrusted-code-mitigations#sandbox-untrusted-execution-in-a-separate-process https://v8.dev/docs/untrusted-code-mitigations#sandbox-untru...
  - kentonv 8y ago
    This is a relatively new recommendation on V8's part specifically in response to Spectre-type vulnerabilities. We've spent a lot of time thinking about and building mitigations for speculative side channel attacks. For example, early on in the project -- before anyone even knew about Spectre -- we made the decision that `Date.now()` would not advance during code execution, only when waiting for I/O. So, a tight loop that calls `Date.now()` repeatedly will keep getting the same value returned. We did this to mitigate timing side channels -- again, even though we didn't know about Spectre yet at the time. Chrome has indeed stated that they believe process isolation is the only mitigation that will work for them. However, this statement is rather specific to the browser environment. The DOM API is gigantic, and it contains many different sources of non-determinism, including several explicit timers as well as concurrent operations (e.g. layout, rendering, etc.). Side channel attacks are necessarily dependent on non-determinism; a fully-deterministic environment essentially by definition has no covert side channels. But, there's no way Chrome can get there. The Cloudflare Workers environment is very different. The only kinds of I/O available to a worker are HTTP in, HTTP out, `Date.now()`, and `crypto.getRandomValues()`. Everything else is perfectly deterministic. So, for us, the problem is much narrower. We need to make sure those four inputs cannot effectively be leveraged into a side channel attack. This is still by no means trivial, but unlike in the browser, it's feasible. `getRandomValues()` is not useful to an attacker because it is completely non-deterministic. `Date.now()` we've already locked down as mentioned above. HTTP in/out can potentially be leveraged to provide external timers -- but the network is extremely noisy. A practical attack would require a lot of time in order to average out the noise -- enough time that we can do a bunch of higher-level things to detect and disrupt possible attacks. It helps that Workers are stateless, so we can reset a worker at any time and move it around, which makes attacks harder. NetSpectre demonstrated that even physical network separation does not necessarily protect you against Spectre attacks. There's simply no such thing as a system that's perfectly secure against Spectre, process isolation or not. All we can do -- aside from going full BSG and giving up on networks altogether -- is make attacks harder to the point of infeasibility. Luckily, we have lots of tools in our toolbox for making Spectre attacks infeasible in the case of Cloudflare Workers.