7 ms·

Using Firecracker and Go to run short, untrusted code execution jobs (2021)

171 points by hasheddan 4y ago

okasaki 4y ago
How do web sites like http://cpp.sh/ http://cpp.sh/ run code? Wouldn't it be enough to forbid/intercept system calls somehow and set limits with systemd-run or do you really need a VM?
- Bayart 4y ago
  There's a talk by Matt Godbolt about the way https://godbolt.org https://godbolt.org works : https://www.youtube.com/watch?v=kIoZDUd5DKw https://www.youtube.com/watch?v=kIoZDUd5DKw
- petercooper 4y ago
  There's the source code for such a site, if that would help: https://github.com/radian-software/riju https://github.com/radian-software/riju Docker + heavily restricted user + firewalls.. seems to get you much of the way there. I am aware that some work was done back in the pre-Docker day with Ruby's online sandbox to neuter Ruby's ability to make certain syscalls, but I imagine Docker, eBPF, or even using WebAssembly makes it a lot easier now.
- jrockway 4y ago
  Some options already listed in replies to this comment, but you might consider gVisor as well: https://gvisor.dev/ https://gvisor.dev/ gVisor is kind of Linux running in Linux. More insolation than containers; less overhead than VMs (but less isolation, of course).
  - jmillikin 4y ago
    > less overhead than VMs Make sure to benchmark your workload first -- gVisor's I/O subsystem is a lot slower than the Linux kernel's, so a VM can be materially faster if you're doing a lot of filesystem operations or file I/O. One of the systems I built at a former employer supported both gVisor and Firecracker for isolation, and the gVisor version was 10-50x slower for a specific class of workload that did ~millions of stat() calls at startup.
    - jrockway 4y ago
      Yup, very good point. I think that something like gVisor should probably be your second choice after you've eliminated VMs for whatever reason.
    - deleted 4y ago
      [deleted]
- jerf 4y ago
  "It's complicated." One thing to bear in mind is that these sites use super-paranoid security because it has been proved time and time again that it is necessary. I wouldn't look at any particular solution for running arbitrary code from a user and assume that it's actually 100% correct. I think this can help remove some of the mystery of how they do it, which is that there is very likely some way in which they actually aren't doing it. Once you remove that idea from the possibility space, the ways it is done start making much more sense. (And the idea becomes much more scary.)
- coffeeblack 4y ago
  Would it be possible to compile the compiler to wasm and then use it in the browser to compile the user code to wasm in the browser and then run that?
kodka 4y ago
I TIL - firecracker, thank you for that! :) Great article and project, i didn't understand the idea of using RabbitMQ and fetching for new events in a loop. I am not an architect, but i was thinking that real-time databases, are used for those purposes? About the containers - if you are not running them in privileged mode, you should be pretty secured, especially by limiting what kind of binaries the containers have.
- eatonphil 4y ago
  What is a "real-time database"?
  - davidjfelix 4y ago
    I think they're referring to firebase which sells itself as a "realtime database" or rephrased, a database designed with "realtime" (automatically updating) websites/apps in mind. The use of the word "realtime" for web tends to trigger a lot of systems developers who use the term to indicate that you can predict the actual real world time that something will take to compute, typically used in automotive and robotic settings. That being said, I didn't invent the word and words have multiple meanings and contexts. In this context it simply means push-delivered data that is pushed when updated. /disclaimer
  - MattyMc 4y ago
    It's a term that Firebase uses. From [their documentation](https://firebase.google.com/docs/database https://firebase.google.com/docs/database): "The Firebase Realtime Database is a cloud-hosted database. Data is stored as JSON and synchronized in realtime to every connected client." I believe "realtime" in this case pertains to the synchronization of data amongst clients through websockets. This is how I've seen the term "realtime database" most commonly used.
  - seabrookmx 4y ago
    People mention firebase but another one was (is? haven't followed it in recent years) RethinkDB.
- paulgb 4y ago
lovingCranberry 4y ago
I was recently looking into an equivalent for V8 isolates. I'd like something like this for python, but it looks like micro VMs is my best bet here. For anyone working in this field or having hands-on experience: is weave's ignite a choice if I'd want to execute long-running python scripts of users? I remember that there was a lot of overhead for I/O. Or should I just go with raw Firecracker like Stan does in this article? The scripts would be long-running, but don't require much computational power. Just a few arithmetic operations on an array every second. The array is being fed in via WebSocket.
- paulgb 4y ago
  I'm a fan of Firecracker, but your use case might be a better fit for plain old containers because the tooling is currently more mature. If it's the isolation that attracts you to Firecracker, gVisor is an option: https://gvisor.dev/ https://gvisor.dev/.
  - lovingCranberry 4y ago
    The isolation is the main pin point why I was looking at firecracker. Thank you for linking gVisor, went through the docs real quick and it looks very promising for my use case.
- babelfish 4y ago
  Cloudflare will be opening their V8 isolate runtime soon (for Cloudflare Workers)
  - asadlionpk 4y ago
    Can you expand on this (a relevant link will suffice)?
    - babelfish 4y ago
      https://blog.cloudflare.com/workers-open-source-announcement/ https://blog.cloudflare.com/workers-open-source-announcement...
johnbellone 4y ago
I've been meaning to take a look at Bottlerocket[^1] as an alternative to a custom spin of Kubernetes but we haven't really had a chance to dig into it. The folks over at Fly[^2] have built an awesome edge platform out of Firecracker, and ultimately, where I want to take the next generation of our internal compute offering. I am eagerly looking forward to any and all presentations they do on their work. [^1]: https://github.com/bottlerocket-os/bottlerocket https://github.com/bottlerocket-os/bottlerocket [^2]: https://fly.io/docs/reference/architecture/ https://fly.io/docs/reference/architecture/
AccountAccount1 4y ago
Isn't web assembly in the browser suited for these kinds of problems? You could run the code, spit the benchmarks and save that (maybe saving the code and the time of the submit for validation.)? Does anyone know about this?
- paulgb 4y ago
  Even for languages that can compile to wasm (Go, Rust), the compilation toolchain doesn't necessarily run in the browser. AFAIK there's no good way to run LLVM in wasm yet. You could do the compilation on the server and then send it back down to the browser to run, though. Theoretically, it should be possible to do it all in the browser, just a lot more work given the state of things now.
  - steveklabnik 4y ago
    Someone did port clang to the browser, though I don’t have the moment to Google it right now. So LLVM has been done at least once.
    - paulgb 4y ago
      Oh cool, so they did! https://binji.github.io/wasm-clang/ https://binji.github.io/wasm-clang/
      - azakai 4y ago
        Here are some more ports of LLVM and clang to wasm/the browser: * https://github.com/kripken/llvm-js https://github.com/kripken/llvm-js * https://github.com/tbfleming/cib https://github.com/tbfleming/cib * https://github.com/jprendes/emception https://github.com/jprendes/emception LLVM just does pure computation, really, so it's not hard to port to wasm - much simpler than say Python (which has also been ported several times). The only challenges with LLVM are the build system (which has self-execution), working around some issues like clang wanting to open a subprocess, and adding some ifdefs.
solarengineer 4y ago
Illumos Zones are another worthwhile technology to explore.
- tptacek 4y ago
  Illumos Zones are shared-kernel isolation, which isn't safe for multitenant untrusted workloads. They're a great way to segregate components of microservice ensembles from a single tenant, to reduce blast radius.
  - bcantrill 4y ago
    Citation needed. I/we ran illumos zones for multitenant untrusted workloads for over a decade -- and at Oxide, we use it as a containment mechanism on top of that provided by the virtual machine. Yes, there have existed vulns (just as in any software), but (in my experience) not at a higher rate than seen in hypervisors or in the CPU itself. (Specifically, the biggest single vuln we had in a decade+ of running zones in a public cloud was -- by far -- Meltdown.)
    - tptacek 4y ago
      I didn't give a statistic in this quote, so I won't cite one. Illumos Zones are shared-kernel isolation: all the tenants share a kernel attack surface. That's not sufficient for untrusted cotenants; any kernel LPE compromises the whole scheme. The kernel's attack surface is drastically larger than (say) the KVM attack surface.
      - bcantrill 4y ago
        Well, it's not just the KVM attack surface -- it's KVM + QEMU, and there have emphatically been escapes. Yes, the shared kernel is an issue -- but so is a shared hypervisor or a shared CPU. It's a risk to be mitigated, and it's a gross exaggeration to say that it "isn't safe for multitenant untrusted workloads."
        4y ago

171 points by hasheddan 4y ago

How do web sites like http://cpp.sh/ http://cpp.sh/ run code? Wouldn't it be enough to forbid/intercept system calls somehow and set limits with systemd-run or do you really need a VM?

Bayart 4y ago
There's a talk by Matt Godbolt about the way https://godbolt.org https://godbolt.org works : https://www.youtube.com/watch?v=kIoZDUd5DKw https://www.youtube.com/watch?v=kIoZDUd5DKw
petercooper 4y ago
There's the source code for such a site, if that would help: https://github.com/radian-software/riju https://github.com/radian-software/riju Docker + heavily restricted user + firewalls.. seems to get you much of the way there. I am aware that some work was done back in the pre-Docker day with Ruby's online sandbox to neuter Ruby's ability to make certain syscalls, but I imagine Docker, eBPF, or even using WebAssembly makes it a lot easier now.
jrockway 4y ago
Some options already listed in replies to this comment, but you might consider gVisor as well: https://gvisor.dev/ https://gvisor.dev/ gVisor is kind of Linux running in Linux. More insolation than containers; less overhead than VMs (but less isolation, of course).
- jmillikin 4y ago
  > less overhead than VMs Make sure to benchmark your workload first -- gVisor's I/O subsystem is a lot slower than the Linux kernel's, so a VM can be materially faster if you're doing a lot of filesystem operations or file I/O. One of the systems I built at a former employer supported both gVisor and Firecracker for isolation, and the gVisor version was 10-50x slower for a specific class of workload that did ~millions of stat() calls at startup.
  - jrockway 4y ago
    Yup, very good point. I think that something like gVisor should probably be your second choice after you've eliminated VMs for whatever reason.
  - deleted 4y ago
    [deleted]
jerf 4y ago
"It's complicated." One thing to bear in mind is that these sites use super-paranoid security because it has been proved time and time again that it is necessary. I wouldn't look at any particular solution for running arbitrary code from a user and assume that it's actually 100% correct. I think this can help remove some of the mystery of how they do it, which is that there is very likely some way in which they actually aren't doing it. Once you remove that idea from the possibility space, the ways it is done start making much more sense. (And the idea becomes much more scary.)
coffeeblack 4y ago
Would it be possible to compile the compiler to wasm and then use it in the browser to compile the user code to wasm in the browser and then run that?

kodka 4y ago

I TIL - firecracker, thank you for that! :) Great article and project, i didn't understand the idea of using RabbitMQ and fetching for new events in a loop. I am not an architect, but i was thinking that real-time databases, are used for those purposes? About the containers - if you are not running them in privileged mode, you should be pretty secured, especially by limiting what kind of binaries the containers have.

eatonphil 4y ago
What is a "real-time database"?
- davidjfelix 4y ago
  I think they're referring to firebase which sells itself as a "realtime database" or rephrased, a database designed with "realtime" (automatically updating) websites/apps in mind. The use of the word "realtime" for web tends to trigger a lot of systems developers who use the term to indicate that you can predict the actual real world time that something will take to compute, typically used in automotive and robotic settings. That being said, I didn't invent the word and words have multiple meanings and contexts. In this context it simply means push-delivered data that is pushed when updated. /disclaimer
- MattyMc 4y ago
  It's a term that Firebase uses. From [their documentation](https://firebase.google.com/docs/database https://firebase.google.com/docs/database): "The Firebase Realtime Database is a cloud-hosted database. Data is stored as JSON and synchronized in realtime to every connected client." I believe "realtime" in this case pertains to the synchronization of data amongst clients through websockets. This is how I've seen the term "realtime database" most commonly used.
- seabrookmx 4y ago
  People mention firebase but another one was (is? haven't followed it in recent years) RethinkDB.
paulgb 4y ago

lovingCranberry 4y ago

I was recently looking into an equivalent for V8 isolates. I'd like something like this for python, but it looks like micro VMs is my best bet here. For anyone working in this field or having hands-on experience: is weave's ignite a choice if I'd want to execute long-running python scripts of users? I remember that there was a lot of overhead for I/O. Or should I just go with raw Firecracker like Stan does in this article? The scripts would be long-running, but don't require much computational power. Just a few arithmetic operations on an array every second. The array is being fed in via WebSocket.

paulgb 4y ago
I'm a fan of Firecracker, but your use case might be a better fit for plain old containers because the tooling is currently more mature. If it's the isolation that attracts you to Firecracker, gVisor is an option: https://gvisor.dev/ https://gvisor.dev/.
- lovingCranberry 4y ago
  The isolation is the main pin point why I was looking at firecracker. Thank you for linking gVisor, went through the docs real quick and it looks very promising for my use case.
babelfish 4y ago
Cloudflare will be opening their V8 isolate runtime soon (for Cloudflare Workers)
- asadlionpk 4y ago
  Can you expand on this (a relevant link will suffice)?
  - babelfish 4y ago
    https://blog.cloudflare.com/workers-open-source-announcement/ https://blog.cloudflare.com/workers-open-source-announcement...

johnbellone 4y ago

I've been meaning to take a look at Bottlerocket[^1] as an alternative to a custom spin of Kubernetes but we haven't really had a chance to dig into it. The folks over at Fly[^2] have built an awesome edge platform out of Firecracker, and ultimately, where I want to take the next generation of our internal compute offering. I am eagerly looking forward to any and all presentations they do on their work. [^1]: https://github.com/bottlerocket-os/bottlerocket https://github.com/bottlerocket-os/bottlerocket [^2]: https://fly.io/docs/reference/architecture/ https://fly.io/docs/reference/architecture/

AccountAccount1 4y ago

Isn't web assembly in the browser suited for these kinds of problems? You could run the code, spit the benchmarks and save that (maybe saving the code and the time of the submit for validation.)? Does anyone know about this?

paulgb 4y ago
Even for languages that can compile to wasm (Go, Rust), the compilation toolchain doesn't necessarily run in the browser. AFAIK there's no good way to run LLVM in wasm yet. You could do the compilation on the server and then send it back down to the browser to run, though. Theoretically, it should be possible to do it all in the browser, just a lot more work given the state of things now.
- steveklabnik 4y ago
  Someone did port clang to the browser, though I don’t have the moment to Google it right now. So LLVM has been done at least once.
  - paulgb 4y ago
    Oh cool, so they did! https://binji.github.io/wasm-clang/ https://binji.github.io/wasm-clang/
    - azakai 4y ago
      Here are some more ports of LLVM and clang to wasm/the browser: * https://github.com/kripken/llvm-js https://github.com/kripken/llvm-js * https://github.com/tbfleming/cib https://github.com/tbfleming/cib * https://github.com/jprendes/emception https://github.com/jprendes/emception LLVM just does pure computation, really, so it's not hard to port to wasm - much simpler than say Python (which has also been ported several times). The only challenges with LLVM are the build system (which has self-execution), working around some issues like clang wanting to open a subprocess, and adding some ifdefs.

solarengineer 4y ago

Illumos Zones are another worthwhile technology to explore.

tptacek 4y ago
Illumos Zones are shared-kernel isolation, which isn't safe for multitenant untrusted workloads. They're a great way to segregate components of microservice ensembles from a single tenant, to reduce blast radius.
- bcantrill 4y ago
  Citation needed. I/we ran illumos zones for multitenant untrusted workloads for over a decade -- and at Oxide, we use it as a containment mechanism on top of that provided by the virtual machine. Yes, there have existed vulns (just as in any software), but (in my experience) not at a higher rate than seen in hypervisors or in the CPU itself. (Specifically, the biggest single vuln we had in a decade+ of running zones in a public cloud was -- by far -- Meltdown.)
  - tptacek 4y ago
    I didn't give a statistic in this quote, so I won't cite one. Illumos Zones are shared-kernel isolation: all the tenants share a kernel attack surface. That's not sufficient for untrusted cotenants; any kernel LPE compromises the whole scheme. The kernel's attack surface is drastically larger than (say) the KVM attack surface.
    - bcantrill 4y ago
      Well, it's not just the KVM attack surface -- it's KVM + QEMU, and there have emphatically been escapes. Yes, the shared kernel is an issue -- but so is a shared hypervisor or a shared CPU. It's a risk to be mitigated, and it's a gross exaggeration to say that it "isn't safe for multitenant untrusted workloads."
      - 4y ago