6 ms·

I’m kind of curious if AWS is ever going to launch a firecracker as a service thing independent from lambda. It would be wonderful for CI or other tasks where y

by easton 4y ago

I’m kind of curious if AWS is ever going to launch a firecracker as a service thing independent from lambda. It would be wonderful for CI or other tasks where you want to rapidly spin up a box and you don’t know how long it needs to be up. EC2 and Fargate take enormous amounts of time to provision compared to firecracker.

1337shadow 4y ago
Well that's going to be a very exensive CI, when virt-lightning spawns a VM in less than 10 seconds with virtio, and you can have plenty on a dedicated server, which you probably have for CI because CI runs faster on dedicated hardware.
rfoo 4y ago
I'd also like to see a Firecracker powered EC2 (with some constraints, of course), but ~6s provision time of current EC2 is already pretty awesome and TBH I don't care about 6s for CI things much.
- easton 4y ago
  We use Azure DevOps at work for our CI/CD, and although they provide an ephemeral runner setup (where you can run the agent with a --once flag, and it will exit after a single job runs so you know to destroy the container/VM), jobs will fail if there are no runners in the pool when the build starts. If we could get VM starts down to milliseconds or a second at most in AWS, we could scale our CI runners down to zero and use a webhook (for PR/commit) from ADO to trigger a VM launch on AWS, and by time the pipeline actually started, there would be an agent ready to take the job. A very specific use case, I know, but if I could have the CI runners run as needed, we could get instances that are way bigger so our builds run faster, and pay around the same amount since they don't have to sit around when they aren't being used.
Dunedan 4y ago
AWS Fargate uses Firecracker as well.
- capableweb 4y ago
  Strange, Fargate is anything but fast.
  - sexy_panda 4y ago
    From my experience the allocation of resources and other tasks preparing the run of a container are consuming quite a lot of time. Pulling the image and building the container is actually just a matter of a few seconds. I have no data about it though.
  - nijave 4y ago
    From testing a couple years ago (things are likely different now), image pull/setup made a pretty noticeable difference. A 1GB container was about 20 seconds slower than a 500MB one--I assume I/O since Fargate instance size didn't make a difference On the other hand, ECS still seems slow compared to k8s where things are nearly instance unless you're measuring so ECS control plane speed might be part of the issue, too
    - easton 4y ago
      This is still a thing, Fargate pull times are super slow: https://github.com/aws/containers-roadmap/issues/696 https://github.com/aws/containers-roadmap/issues/696. We run all of our workloads on fargate, and it's really annoying when you're trying to iterate on something and you have to sit there waiting on "Provisioning..." for 1-2 minutes every time you launch a task. I don't think the control plane is that slow, as EC2 based ECS launches tasks really fast if the images are already cached on the machine.
  - acdha 4y ago
    People have mentioned image loading but one other shockingly slow thing is allocating ENIs (this also affects Lambda, VPC endpoints, etc.). I've had a few times where I've looked at the logs and it's basically been like 5 minutes to launch something where 4 of those were waiting for the ENI.
JoshTriplett 4y ago
I would love to see this as well. I currently can launch a Linux VM in milliseconds, but EC2 takes ~6s before the first user-provided instruction gets to run.
staticassertion 4y ago
How fast do you want? My bet is that you can get EC2 to boot up very quickly, ie: ~1 minute or less with a bit of effort.

by easton 4y ago

1337shadow 4y ago

Well that's going to be a very exensive CI, when virt-lightning spawns a VM in less than 10 seconds with virtio, and you can have plenty on a dedicated server, which you probably have for CI because CI runs faster on dedicated hardware.

rfoo 4y ago

I'd also like to see a Firecracker powered EC2 (with some constraints, of course), but ~6s provision time of current EC2 is already pretty awesome and TBH I don't care about 6s for CI things much.

easton 4y ago
We use Azure DevOps at work for our CI/CD, and although they provide an ephemeral runner setup (where you can run the agent with a --once flag, and it will exit after a single job runs so you know to destroy the container/VM), jobs will fail if there are no runners in the pool when the build starts. If we could get VM starts down to milliseconds or a second at most in AWS, we could scale our CI runners down to zero and use a webhook (for PR/commit) from ADO to trigger a VM launch on AWS, and by time the pipeline actually started, there would be an agent ready to take the job. A very specific use case, I know, but if I could have the CI runners run as needed, we could get instances that are way bigger so our builds run faster, and pay around the same amount since they don't have to sit around when they aren't being used.

Dunedan 4y ago

AWS Fargate uses Firecracker as well.

capableweb 4y ago
Strange, Fargate is anything but fast.
- sexy_panda 4y ago
  From my experience the allocation of resources and other tasks preparing the run of a container are consuming quite a lot of time. Pulling the image and building the container is actually just a matter of a few seconds. I have no data about it though.
- nijave 4y ago
  From testing a couple years ago (things are likely different now), image pull/setup made a pretty noticeable difference. A 1GB container was about 20 seconds slower than a 500MB one--I assume I/O since Fargate instance size didn't make a difference On the other hand, ECS still seems slow compared to k8s where things are nearly instance unless you're measuring so ECS control plane speed might be part of the issue, too
  - easton 4y ago
    This is still a thing, Fargate pull times are super slow: https://github.com/aws/containers-roadmap/issues/696 https://github.com/aws/containers-roadmap/issues/696. We run all of our workloads on fargate, and it's really annoying when you're trying to iterate on something and you have to sit there waiting on "Provisioning..." for 1-2 minutes every time you launch a task. I don't think the control plane is that slow, as EC2 based ECS launches tasks really fast if the images are already cached on the machine.
- acdha 4y ago
  People have mentioned image loading but one other shockingly slow thing is allocating ENIs (this also affects Lambda, VPC endpoints, etc.). I've had a few times where I've looked at the logs and it's basically been like 5 minutes to launch something where 4 of those were waiting for the ENI.

JoshTriplett 4y ago

I would love to see this as well. I currently can launch a Linux VM in milliseconds, but EC2 takes ~6s before the first user-provided instruction gets to run.

staticassertion 4y ago

How fast do you want? My bet is that you can get EC2 to boot up very quickly, ie: ~1 minute or less with a bit of effort.