9 ms·

Fast CI with MicroVMs

132 points by alexellisuk 4y ago

kernelbugs 4y ago
Would have loved to see more of the technical details involved in spinning up Firecracker VMs on demand for Github Actions.
- alexellisuk 4y ago
  Hey thanks for the feedback. We may do some more around this. What kinds of things do you want to know? To get hands on, you can run my Firecracker lab that I shared in the blog post, then add a runner can be done with "arkade system install actions-runner" We also explain how it works here: https://docs.actuated.dev/faq/ https://docs.actuated.dev/faq/
  - thehabbos007 4y ago
    Not the poster you were replying to, but I've looked at your firecracker init lab (cool stuff!) and just wondering how that fits together with a control plane. Would be cool to see how the orchestration happens in terms of messaging between host/guest and how I/O is provisioned on the host dynamically.
fideloper 4y ago
This project looks really neat! Firecracker is very cool, I wish/hope tooling around it matures enough to be super easy. I'd love to see the technical details on how this is run. It looks like it's closed source? The need for baremetal for Firecracker is a bit of a shame, but it's still wicked cool. (You can run it on a DO droplet but nested virtualization feels a bit icky?) I run a CI app myself, and have looked at firecracker. Right now I'm working on moving some compute to Fly.io and it's Machines API, which is well suited for on-demand compute.
- alexellisuk 4y ago
  Hey thanks for the interest, this is probably the best resource I have on Firecracker, hope you enjoy it: https://www.youtube.com/watch?v=CYCsa5e2vqg https://www.youtube.com/watch?v=CYCsa5e2vqg For info on actuated, check out the FAQ or the docs: https://docs.actuated.dev https://docs.actuated.dev We're running a pilot and looking for customers who want to make CI faster for public or self-hosted runners, want to avoid side-effects and security compromise of DIND / sharing a Docker socket or need to build on ARM64 for speed. Feel free to reach out
bkq 4y ago
Good article. Firecracker is something that has definitely piqued my interest when it comes to quickly spinning up a throwaway environment to use for either development or CI. I run a CI platform [1], which currently uses QEMU for the build environments (Docker is also supported but currently disabled on the hosted offering), startup times are ok, but having a boot time of 1-2s is definitely highly appealing. I will have to investigate Firecracker further to see if I could incorporate this into what I'm doing. Julia Evans has also written about Firecracker in the past too [2][3]. [1] - https://about.djinn-ci.com https://about.djinn-ci.com [2] - https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-less-than-a-second/ https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-l... [3] - https://news.ycombinator.com/item?id=25883253 https://news.ycombinator.com/item?id=25883253
- alexellisuk 4y ago
  Thanks for commenting, and your product looks cool btw. Yeah a lot of people have talked about Firecracker in the past, that's why I focus on the pain and the problem being solved. The tech is cool, but it's not the only thing that matters. People need to know that there are better alternatives to sharing a docker socket or using DIND with K8s runners.
ridiculous_fish 4y ago
The article does not say what a MicroVM is. From what I can gather, it's using KVM to virtualize specifically a Linux kernel. In this way, Firecracker is somewhat intermediate between Docker (which shares the host kernel) and Vagrant (which is not limited to running Linux). Is that accurate? Is it possible to use a MicroVM to virtualize a non-Linux OS?
- alexellisuk 4y ago
  Thanks for the feedback. That video covers this is great detail. Click on the the video under 1) and have a watch, it should answer all your questions. I didn't want to repeat the content there
  - ridiculous_fish 4y ago
    Will do, thanks!
Sytten 4y ago
Wondering if it would be possible to run macos. The hosted runner of Github Actions for macos are really really horrible, our builds take easily 2x to 3x more time than hosted Windows and Linux machines.
f0e4c2f7 4y ago
This seems pretty interesting to me. I haven't messed with firecracker yet but it seems like a possible alternative to docker in the future.
- alexellisuk 4y ago
  It is, but is also a very low-level tool, and there is very little support around it. We've been building this platform since the summer and there are many nuances and edge cases to cater for. But if you just want to try out Firecracker, I've got a free lab listed in the blog post. I hear Podman desktop is also getting some traction, if you have particular issues with Docker Desktop.
lxe 4y ago
Firecracker is nice but still very limited to what it can do. My gripe with all CI systems is that an an industry standard we've universally sacrificed performance for hermeticity and re-entrancy, even when it doesn't really gives us a practical advantage. Downloading and re-running containers and vms, endlessly checking out code, installing deps over and over is just a waste of time, even with caching, COW, and other optimizations.
- jxf 4y ago
  > My gripe with all CI systems is that an an industry standard we've universally sacrificed performance for hermeticity and re-entrancy, even when it doesn't really gives us a practical advantage. The perceived practical advantage is the incremental confidence that the thing you built won't blow up in production. > even with caching, COW, and other optimizations Many CI systems do employ caching. For example, Circle.
- IshKebab 4y ago
  Hermeticity is precisely what allows you to avoid endlessly downloading and building the same dependencies. Without hermeticity you can't rely on caching. I feel like 90% of the computer industry is ignoring the lessons of Bazel and is probably going to wake up in 10 years and go "ooooooh, that's how we should have been doing it".
  - no_wizard 4y ago
    for the frontend space, NX gives you bazel caching / features. It just doesn't cache dependencies, but in my experience with GitHub Actions running `pnpm install` or `yarn install` is not the slowest operation, its running the tools after.
  - throwaway894345 4y ago
    I think everyone agrees that the Bazel/Nix approach is correct, the problem is that Bazel/Nix/etc are insanely hard to use. For example, I spent a good chunk of last weekend trying to get Bazel to build a multiarch Go image, and I couldn't figure it out. Someone needs to figure out how to polish Bazel/Nix so they're viable for organizations that can't invest in a team to operate and provide guidance on Bazel/Nix/etc.
a-dub 4y ago
this is cool. throwing firecracker at CI is something i've been thinking about since i first read about firecracker. i was thinking more along the lines of, can you checkpoint a bunch of common initialization and startup and then massively parallelize?
- alexellisuk 4y ago
  You can checkpoint and restore, but only once for security reasons, so it doesn't help much. https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md#snapshot-security-and-uniqueness https://github.com/firecracker-microvm/firecracker/blob/main... The VMs launch super quick in < 1s they are actually running a job.
  - a-dub 4y ago
    hm. well assuming they have no network and don't otherwise encrypt bits that an attacker could get ahold of, it's probably fine. the bigger issue would be something like spawning a bunch of servers that share the same rng state that can then be manipulated by an attacker (and therefore encrypt different data with the same key+nonce and such).
avita1 4y ago
Something I've increasingly wondered is if the model of CI where a totally pristine container (or VM) gets spun on each change for each test set imposes an floor on how fast CI can run. Each job will always have to run a clone, always pay the cost of either bootstrapping a toolchain or download a giant container with the toolchain, and always have to download a big remote cache. If I had infinity time, I'd build a CI system that found a runner that maintained some state (gasp!) about the build and went to a test runner that had most of its local build cache downloaded, source code cloned, and toolchain bootstrapped.
- lytedev 4y ago
  I have zero experience with bazel, but I believe it offers the possibility of mechanisms similar to this? Or a mechanism that makes this "somewhat safe"?
  - hobofan 4y ago
    Yes it does, but one should be warned that adopting Bazel isn't the lightest decision to make. But yeah, the CI experience is one of its best attributes. We are using Bazel with Github self-hosted runners, and have consistent low build times with a growing codebase and test suite, as Bazel will only re-build and re-test what needs to be changed. The CI experience compared to e.g. doing naive caching of some directories with Github managed runners is amazing, and it's probably the most reliable build/test setup I've had. The most common failure we have of the build system itself (which is still rare with ~once a week) is network issues with one of the package managers, rather than quirks introduced by one of the engineers (and there would be a straightforward path towards preventing those failures, we just haven't bothered to set that up yet).
- capableweb 4y ago
  You'd love a service like that, until you have some weird stuff working in CI but not in local (or vice-versa), that's why things are built from scratch all the time, to prevent any such issues from happening. Npm was (still is?) famously bad at installing dependencies, where sometimes the fix is to remove node_modules and simply reinstalling. Back when npm was more brittle (yes, possible) it was nearly impossible to maintain caches of node_modules directories, as they ended up being different than if you reinstalled with no existing node_modules directory.
brightball 4y ago
I’m curious to see how k8s isn’t a good fit for this? I’m not a k8s advocate for production code but at the CI level it seems ideal.
- alexellisuk 4y ago
  Great questions, we answer those here in the FAQ: https://docs.actuated.dev/faq/ https://docs.actuated.dev/faq/
no_wizard 4y ago
Firecracker is pretty great, good to see it can be used in a CI environment like this, definitely peaking my interest. I know its the backbone of what runs fly.io[0] as well [0]: https://fly.io/docs/reference/architecture/#microvms https://fly.io/docs/reference/architecture/#microvms
- emmelaich 4y ago
  *piquing
- waz0wski 4y ago
  Firecracker has been running AWS Lambda & Fargate for a few years now https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/ https://aws.amazon.com/blogs/aws/firecracker-lightweight-vir... There's also similar microVM project with a bit more container-focused support called Kata https://katacontainers.io/ https://katacontainers.io/
  - digianarchist 4y ago
    Not a hypervisor expert by any means but what's stopping projects backporting the super fast startup time of Firecracker into regular VM hypervisors? I'm assuming that Firecracker is somewhat constrained in some way.
    - antonvs 4y ago
      It’s written specifically to host the Linux kernel, and doesn’t use a bios or a boot loader. If you backported that into another hypervisor, it would probably have to be something like “are we loading a compatible Linux? If so switch to Firecracker mode”. But of course you can do that yourself, with a small shell script that either starts the traditional VM or Firecracker. Or they could do what QEMU has done and put out a separate product/mode: https://github.com/qemu/qemu/blob/a082fab9d25/docs/system/i386/microvm.rst https://github.com/qemu/qemu/blob/a082fab9d25/docs/system/i3...
      - 4y ago
deltaci 4y ago
congratulations on the launch. it looks pretty much like a self-hosted version of https://buildjet.com/for-github-actions https://buildjet.com/for-github-actions
- alexellisuk 4y ago
  Thanks for commenting. It seems like buildjet is competing directly with GitHub on price (GitHub has bigger runners available now, pay per minute), and GitHub will always win because they own Azure, so I'm not sure what their USP is and worry they will get commoditised and then lose their market share. Actuated is hybrid, not self-hosted. We run actuated as a managed service and scheduler, you provide your own compute and run our agent, then it's a very hands-off experience. This comes with support from our team, and extensive documentation. Agents can even be cheap-ish VMs using nested virtualisation, you can learn a bit more here: https://docs.actuated.dev/add-agent/ https://docs.actuated.dev/add-agent/
rad_gruchalski 4y ago
Congratulations on the launch. The interesting part of this is that the client supplies the most difficult resource to get for this setup. As in, a machine on which Firecracker can run.
- alexellisuk 4y ago
  Users provide a number of hosts and run a simple agent. We maintain the OS image, Kernel configuration and control plane service, with support for ARM64 too.
  - rad_gruchalski 4y ago
    Great stuff, undeniably. There’s not much going on in the open source space around multi-host schedules for Firecracker. So that’s a mountain of work. With regards to the host, I made remark because of Firecracker requirements regarding virtualisation. Running Firecracker is no brainer when an org maintains a fleet of their own hardware.
imachine1980_ 4y ago
Realy cool what is the license? , there any way I can contribute code/test/documentation to this project ?
throwawaaarrgh 4y ago
> I spoke to the GitHub Actions engineering team, who told me that using an ephemeral VM and an immutable OS image would solve the concerns. that doesn't solve them all. the main problem is secrets. if a job has access to an api token that can be used to modify your code or access a cloud service, a PR can abuse that to modify things it shouldn't. a second problem is even if you don't have secrets exposed, a PR can run a crypto miner, wasting your money. finally, a self-hosted runner is a step into your private network and can be used for attacks, which firecracker can help mitigate but never eliminate. the best solution to these problems is 1) don't allow repos to trigger your CI unless the user is trusted or the change has been reviewed, 2) always use least privilege and zero-trust for all access (yes even for dev services), 3) add basic constraints by default on all jobs running to prevent misuse, and then finally 4) provide strong isolation in addition to ephemeral environments.
- alexellisuk 4y ago
  You still have those same problems with hosted runners, don't you? We're trying to re-create the hosted experience, but with self-hosted, faster, infrastructure, without needing to account for metered billing.
  - Twirrim 4y ago
    https://marcyoung.us/post/zuckerpunch/ https://marcyoung.us/post/zuckerpunch/ Facebook had this exact problem recently with the pytorch repo. Their self hosted CI runners would run on all PRs, and it could leak all sorts of stuff.
- deleted 4y ago
  [deleted]
ignoramous 4y ago
Sounds similar to webapp.io (layerci) that has been discussed quite a few times here: https://news.ycombinator.com/item?id=31062301 https://news.ycombinator.com/item?id=31062301 > Friction starts when the 7GB of RAM and 2 cores allocated causes issues for us Well, I just create a 20GB swap. There's ample disk space but swap is slow for sure. > MicroVM Coincidentally, QEMU now sports a firecracker-inspired microvm: https://github.com/qemu/qemu/blob/a082fab9d25/docs/system/i386/microvm.rst https://github.com/qemu/qemu/blob/a082fab9d25/docs/system/i3... / https://mergeboard.com/blog/2-qemu-microvm-docker/ https://mergeboard.com/blog/2-qemu-microvm-docker/
- alexellisuk 4y ago
  Hi, I'd not heard of webapp.io before so thanks for mentioning it. Actuated is not a preview branch product, that's an interesting area but not the problem we're trying to solve. actuated is not trying to be a CI system or a replacement for one like webapp. It's a direct integration with GitHub Actions, and as we get interest from pilot customer for GitLab etc, we'll consider adding support for those platforms too. Unopinionated, without lock-in. We want to create the hosted experience, with safety and speed built in.
- synergy20 4y ago
  did not know qemu has its own firecraker machine now, thanks for the info! going to test how fast it boots.
- colinchartier 4y ago
  Hey, yeah this looks somewhat similar to what we're building at https://webapp.io https://webapp.io (nee LayerCI, YC S20) We migrated to a fork of firecracker, but we're a fully hosted product that doesn't directly interact with GHA at all (similar to how CircleCI works), so there's some positioning difference between us and OP at the very least. Always happy to see innovation in the space :)
  - dblooman 4y ago
    What do you use to orchestrate firecracker?

132 points by alexellisuk 4y ago

Would have loved to see more of the technical details involved in spinning up Firecracker VMs on demand for Github Actions.

alexellisuk 4y ago
Hey thanks for the feedback. We may do some more around this. What kinds of things do you want to know? To get hands on, you can run my Firecracker lab that I shared in the blog post, then add a runner can be done with "arkade system install actions-runner" We also explain how it works here: https://docs.actuated.dev/faq/ https://docs.actuated.dev/faq/
- thehabbos007 4y ago
  Not the poster you were replying to, but I've looked at your firecracker init lab (cool stuff!) and just wondering how that fits together with a control plane. Would be cool to see how the orchestration happens in terms of messaging between host/guest and how I/O is provisioned on the host dynamically.

fideloper 4y ago

This project looks really neat! Firecracker is very cool, I wish/hope tooling around it matures enough to be super easy. I'd love to see the technical details on how this is run. It looks like it's closed source? The need for baremetal for Firecracker is a bit of a shame, but it's still wicked cool. (You can run it on a DO droplet but nested virtualization feels a bit icky?) I run a CI app myself, and have looked at firecracker. Right now I'm working on moving some compute to Fly.io and it's Machines API, which is well suited for on-demand compute.

alexellisuk 4y ago
Hey thanks for the interest, this is probably the best resource I have on Firecracker, hope you enjoy it: https://www.youtube.com/watch?v=CYCsa5e2vqg https://www.youtube.com/watch?v=CYCsa5e2vqg For info on actuated, check out the FAQ or the docs: https://docs.actuated.dev https://docs.actuated.dev We're running a pilot and looking for customers who want to make CI faster for public or self-hosted runners, want to avoid side-effects and security compromise of DIND / sharing a Docker socket or need to build on ARM64 for speed. Feel free to reach out

bkq 4y ago

Good article. Firecracker is something that has definitely piqued my interest when it comes to quickly spinning up a throwaway environment to use for either development or CI. I run a CI platform [1], which currently uses QEMU for the build environments (Docker is also supported but currently disabled on the hosted offering), startup times are ok, but having a boot time of 1-2s is definitely highly appealing. I will have to investigate Firecracker further to see if I could incorporate this into what I'm doing. Julia Evans has also written about Firecracker in the past too [2][3]. [1] - https://about.djinn-ci.com https://about.djinn-ci.com [2] - https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-less-than-a-second/ https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-l... [3] - https://news.ycombinator.com/item?id=25883253 https://news.ycombinator.com/item?id=25883253

alexellisuk 4y ago
Thanks for commenting, and your product looks cool btw. Yeah a lot of people have talked about Firecracker in the past, that's why I focus on the pain and the problem being solved. The tech is cool, but it's not the only thing that matters. People need to know that there are better alternatives to sharing a docker socket or using DIND with K8s runners.

ridiculous_fish 4y ago

The article does not say what a MicroVM is. From what I can gather, it's using KVM to virtualize specifically a Linux kernel. In this way, Firecracker is somewhat intermediate between Docker (which shares the host kernel) and Vagrant (which is not limited to running Linux). Is that accurate? Is it possible to use a MicroVM to virtualize a non-Linux OS?

alexellisuk 4y ago
Thanks for the feedback. That video covers this is great detail. Click on the the video under 1) and have a watch, it should answer all your questions. I didn't want to repeat the content there
- ridiculous_fish 4y ago
  Will do, thanks!

Sytten 4y ago

Wondering if it would be possible to run macos. The hosted runner of Github Actions for macos are really really horrible, our builds take easily 2x to 3x more time than hosted Windows and Linux machines.

f0e4c2f7 4y ago

This seems pretty interesting to me. I haven't messed with firecracker yet but it seems like a possible alternative to docker in the future.

alexellisuk 4y ago
It is, but is also a very low-level tool, and there is very little support around it. We've been building this platform since the summer and there are many nuances and edge cases to cater for. But if you just want to try out Firecracker, I've got a free lab listed in the blog post. I hear Podman desktop is also getting some traction, if you have particular issues with Docker Desktop.

lxe 4y ago

Firecracker is nice but still very limited to what it can do. My gripe with all CI systems is that an an industry standard we've universally sacrificed performance for hermeticity and re-entrancy, even when it doesn't really gives us a practical advantage. Downloading and re-running containers and vms, endlessly checking out code, installing deps over and over is just a waste of time, even with caching, COW, and other optimizations.

jxf 4y ago
> My gripe with all CI systems is that an an industry standard we've universally sacrificed performance for hermeticity and re-entrancy, even when it doesn't really gives us a practical advantage. The perceived practical advantage is the incremental confidence that the thing you built won't blow up in production. > even with caching, COW, and other optimizations Many CI systems do employ caching. For example, Circle.
IshKebab 4y ago
Hermeticity is precisely what allows you to avoid endlessly downloading and building the same dependencies. Without hermeticity you can't rely on caching. I feel like 90% of the computer industry is ignoring the lessons of Bazel and is probably going to wake up in 10 years and go "ooooooh, that's how we should have been doing it".
- no_wizard 4y ago
  for the frontend space, NX gives you bazel caching / features. It just doesn't cache dependencies, but in my experience with GitHub Actions running `pnpm install` or `yarn install` is not the slowest operation, its running the tools after.
- throwaway894345 4y ago
  I think everyone agrees that the Bazel/Nix approach is correct, the problem is that Bazel/Nix/etc are insanely hard to use. For example, I spent a good chunk of last weekend trying to get Bazel to build a multiarch Go image, and I couldn't figure it out. Someone needs to figure out how to polish Bazel/Nix so they're viable for organizations that can't invest in a team to operate and provide guidance on Bazel/Nix/etc.

a-dub 4y ago

this is cool. throwing firecracker at CI is something i've been thinking about since i first read about firecracker. i was thinking more along the lines of, can you checkpoint a bunch of common initialization and startup and then massively parallelize?

alexellisuk 4y ago
You can checkpoint and restore, but only once for security reasons, so it doesn't help much. https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md#snapshot-security-and-uniqueness https://github.com/firecracker-microvm/firecracker/blob/main... The VMs launch super quick in < 1s they are actually running a job.
- a-dub 4y ago
  hm. well assuming they have no network and don't otherwise encrypt bits that an attacker could get ahold of, it's probably fine. the bigger issue would be something like spawning a bunch of servers that share the same rng state that can then be manipulated by an attacker (and therefore encrypt different data with the same key+nonce and such).

avita1 4y ago

Something I've increasingly wondered is if the model of CI where a totally pristine container (or VM) gets spun on each change for each test set imposes an floor on how fast CI can run. Each job will always have to run a clone, always pay the cost of either bootstrapping a toolchain or download a giant container with the toolchain, and always have to download a big remote cache. If I had infinity time, I'd build a CI system that found a runner that maintained some state (gasp!) about the build and went to a test runner that had most of its local build cache downloaded, source code cloned, and toolchain bootstrapped.

lytedev 4y ago
I have zero experience with bazel, but I believe it offers the possibility of mechanisms similar to this? Or a mechanism that makes this "somewhat safe"?
- hobofan 4y ago
  Yes it does, but one should be warned that adopting Bazel isn't the lightest decision to make. But yeah, the CI experience is one of its best attributes. We are using Bazel with Github self-hosted runners, and have consistent low build times with a growing codebase and test suite, as Bazel will only re-build and re-test what needs to be changed. The CI experience compared to e.g. doing naive caching of some directories with Github managed runners is amazing, and it's probably the most reliable build/test setup I've had. The most common failure we have of the build system itself (which is still rare with ~once a week) is network issues with one of the package managers, rather than quirks introduced by one of the engineers (and there would be a straightforward path towards preventing those failures, we just haven't bothered to set that up yet).
capableweb 4y ago
You'd love a service like that, until you have some weird stuff working in CI but not in local (or vice-versa), that's why things are built from scratch all the time, to prevent any such issues from happening. Npm was (still is?) famously bad at installing dependencies, where sometimes the fix is to remove node_modules and simply reinstalling. Back when npm was more brittle (yes, possible) it was nearly impossible to maintain caches of node_modules directories, as they ended up being different than if you reinstalled with no existing node_modules directory.

brightball 4y ago

I’m curious to see how k8s isn’t a good fit for this? I’m not a k8s advocate for production code but at the CI level it seems ideal.

alexellisuk 4y ago
Great questions, we answer those here in the FAQ: https://docs.actuated.dev/faq/ https://docs.actuated.dev/faq/

no_wizard 4y ago

Firecracker is pretty great, good to see it can be used in a CI environment like this, definitely peaking my interest. I know its the backbone of what runs fly.io[0] as well [0]: https://fly.io/docs/reference/architecture/#microvms https://fly.io/docs/reference/architecture/#microvms

emmelaich 4y ago
*piquing
waz0wski 4y ago
Firecracker has been running AWS Lambda & Fargate for a few years now https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/ https://aws.amazon.com/blogs/aws/firecracker-lightweight-vir... There's also similar microVM project with a bit more container-focused support called Kata https://katacontainers.io/ https://katacontainers.io/
- digianarchist 4y ago
  Not a hypervisor expert by any means but what's stopping projects backporting the super fast startup time of Firecracker into regular VM hypervisors? I'm assuming that Firecracker is somewhat constrained in some way.
  - antonvs 4y ago
    It’s written specifically to host the Linux kernel, and doesn’t use a bios or a boot loader. If you backported that into another hypervisor, it would probably have to be something like “are we loading a compatible Linux? If so switch to Firecracker mode”. But of course you can do that yourself, with a small shell script that either starts the traditional VM or Firecracker. Or they could do what QEMU has done and put out a separate product/mode: https://github.com/qemu/qemu/blob/a082fab9d25/docs/system/i386/microvm.rst https://github.com/qemu/qemu/blob/a082fab9d25/docs/system/i3...
    - 4y ago

deltaci 4y ago

congratulations on the launch. it looks pretty much like a self-hosted version of https://buildjet.com/for-github-actions https://buildjet.com/for-github-actions

alexellisuk 4y ago
Thanks for commenting. It seems like buildjet is competing directly with GitHub on price (GitHub has bigger runners available now, pay per minute), and GitHub will always win because they own Azure, so I'm not sure what their USP is and worry they will get commoditised and then lose their market share. Actuated is hybrid, not self-hosted. We run actuated as a managed service and scheduler, you provide your own compute and run our agent, then it's a very hands-off experience. This comes with support from our team, and extensive documentation. Agents can even be cheap-ish VMs using nested virtualisation, you can learn a bit more here: https://docs.actuated.dev/add-agent/ https://docs.actuated.dev/add-agent/

rad_gruchalski 4y ago

Congratulations on the launch. The interesting part of this is that the client supplies the most difficult resource to get for this setup. As in, a machine on which Firecracker can run.

alexellisuk 4y ago
Users provide a number of hosts and run a simple agent. We maintain the OS image, Kernel configuration and control plane service, with support for ARM64 too.
- rad_gruchalski 4y ago
  Great stuff, undeniably. There’s not much going on in the open source space around multi-host schedules for Firecracker. So that’s a mountain of work. With regards to the host, I made remark because of Firecracker requirements regarding virtualisation. Running Firecracker is no brainer when an org maintains a fleet of their own hardware.

imachine1980_ 4y ago

Realy cool what is the license? , there any way I can contribute code/test/documentation to this project ?

throwawaaarrgh 4y ago

> I spoke to the GitHub Actions engineering team, who told me that using an ephemeral VM and an immutable OS image would solve the concerns. that doesn't solve them all. the main problem is secrets. if a job has access to an api token that can be used to modify your code or access a cloud service, a PR can abuse that to modify things it shouldn't. a second problem is even if you don't have secrets exposed, a PR can run a crypto miner, wasting your money. finally, a self-hosted runner is a step into your private network and can be used for attacks, which firecracker can help mitigate but never eliminate. the best solution to these problems is 1) don't allow repos to trigger your CI unless the user is trusted or the change has been reviewed, 2) always use least privilege and zero-trust for all access (yes even for dev services), 3) add basic constraints by default on all jobs running to prevent misuse, and then finally 4) provide strong isolation in addition to ephemeral environments.

alexellisuk 4y ago
You still have those same problems with hosted runners, don't you? We're trying to re-create the hosted experience, but with self-hosted, faster, infrastructure, without needing to account for metered billing.
- Twirrim 4y ago
  https://marcyoung.us/post/zuckerpunch/ https://marcyoung.us/post/zuckerpunch/ Facebook had this exact problem recently with the pytorch repo. Their self hosted CI runners would run on all PRs, and it could leak all sorts of stuff.
deleted 4y ago
[deleted]

ignoramous 4y ago

Sounds similar to webapp.io (layerci) that has been discussed quite a few times here: https://news.ycombinator.com/item?id=31062301 https://news.ycombinator.com/item?id=31062301 > Friction starts when the 7GB of RAM and 2 cores allocated causes issues for us Well, I just create a 20GB swap. There's ample disk space but swap is slow for sure. > MicroVM Coincidentally, QEMU now sports a firecracker-inspired microvm: https://github.com/qemu/qemu/blob/a082fab9d25/docs/system/i386/microvm.rst https://github.com/qemu/qemu/blob/a082fab9d25/docs/system/i3... / https://mergeboard.com/blog/2-qemu-microvm-docker/ https://mergeboard.com/blog/2-qemu-microvm-docker/

alexellisuk 4y ago
Hi, I'd not heard of webapp.io before so thanks for mentioning it. Actuated is not a preview branch product, that's an interesting area but not the problem we're trying to solve. actuated is not trying to be a CI system or a replacement for one like webapp. It's a direct integration with GitHub Actions, and as we get interest from pilot customer for GitLab etc, we'll consider adding support for those platforms too. Unopinionated, without lock-in. We want to create the hosted experience, with safety and speed built in.
synergy20 4y ago
did not know qemu has its own firecraker machine now, thanks for the info! going to test how fast it boots.
colinchartier 4y ago
Hey, yeah this looks somewhat similar to what we're building at https://webapp.io https://webapp.io (nee LayerCI, YC S20) We migrated to a fork of firecracker, but we're a fully hosted product that doesn't directly interact with GHA at all (similar to how CircleCI works), so there's some positioning difference between us and OP at the very least. Always happy to see innovation in the space :)
- dblooman 4y ago
  What do you use to orchestrate firecracker?