4 ms·

When I first starting using AWS a few years ago, having known generally what it was for far longer, I was flabbergasted it was at how slow it was to get an inst

When I first starting using AWS a few years ago, having known generally what it was for far longer, I was flabbergasted it was at how slow it was to get an instance booted. I expected much less, thinking about things from first principals, even if you're literally talking about cold booting a physical machine via IPMI. But it seemed like everyone accepted that as the way it was and now I do too. So I'm glad people are still interested in making things fast. Right now I'm doing Postgres stuff (RDS) and dealing with taking 10+ minutes to boot a fresh instance. I'm tempted to try out fly.io and their Postgres clusters but I'm afraid I'd be spoiled and hate my life after (my job has me stuck in AWS for the interminable future). I would be interested to know where all that time is being spent in on the AWS side. To be a fly on the wall seeing their full, unfiltered logging and metrics.

boulos 4y ago
Disclosure: I used to work on GCE. EC2 has historically not focused much on instance boot time. We did for GCE and drove it down pretty heavily. The post here from fly has a good set of sequence diagrams for "what are the various phases of creating an instance from scratch" that are generally applicable. I'll note though that different users have different targets. Some people care about "time from request to first instruction ticks over" while others only care about "time from request to ssh'able from the public internet". There's an interesting middle ground of "time from request to being able to talk to other services like GCS or S3". It's not clear to me what the networking / discovery story is for a Fly Machine that is stopped and then starts. That is, how long does fly-proxy take to update (globally? within a metro?) to add and remove the new Fly Machine? I vaguely recall that only external endpoints support IPv4, so I assume Fly is reserving and registering the internal IPv6 endpoints in the more expensive "create" step and then "start" is just about propagating liveness.
- ZeroCool2u 4y ago
  My main gig runs workloads primarily on AWS, but I work with a small company as well that is completely on GCP and I gotta say the difference is night and day in instance allocation and startup time. GCE is so much faster it's infuriating when I've gotta go back to work and sometimes have to wait more than 10 minutes in the worst case for an EC2 instance to finish booting in an EKS cluster that's already running!
- derefr 4y ago
  > We did for GCE and drove it down pretty heavily. As a heavy GCE user, something weird about GCE to me is that instance boot times can be extremely variable — predictable within a given instance group, but unpredictable with even very small changes to e.g. instance sizes within the same instance family. (And this isn’t the instance blocking on getting scheduled onto a hypervisor; I can recognize that point, because it’s when any quota limits hit to potentially kill the instance provision. That phase of the delay is very stable.) The variable delays also seem to apply to “reset” of the instance (which won’t involve an even-temporary deschedule, as reset keeps NVMe state) — but not to kexec-reboot, if one opts for that. Is GCE built on a hybrid of two different hypervisor systems with wildly differing boot-time performance characteristics, where subtle tweaks of your instance config determine which hypervisor you get? Maybe one that relies on a hardware offload for something (PXE kernel signature verification?) that the other does purely in software? One thing that’s clear to me is that instance types introduced after a certain point (e.g. all n2d instances) are always of the fast-booting type. So I’m guessing this is just old hypervisors being stuck on some legacy config because of long-term customers partially pinning those host machines with workloads that somehow prevent live migration (workloads with NVMe or GPU), and so make those hypervisors unable to be drained for a hard-upgrade.
- dmitriid 4y ago
  > I'll note though that different users have different targets. Some people care about "time from request to first instruction ticks over" while others only care about "time from request to ssh'able from the public internet". This is the same target: a machine (that usually only has single app on it) shouldn't take more time to boot than a general-purpose consumer PC/laptop. The reason it takes so darn long to start in so many cases is just how horrendously overcomplicated the whole cloud setup is internally and externally (sometimes for good reasons, sometimes because we don't know better, sometimes just because it really is just overly complicated and overengineered)
  - nousermane 4y ago
    > shouldn't take more time to boot than a general-purpose consumer PC/laptop. That's an incredibly easy target. VMs can and should boot much faster than that - just look at firecracker hypervisor. Even with KVM, if you replace systemd with something small and simple [0] (which you totally should, for single-app VMs), boot times of couple of seconds are within reach. [0] https://github.com/Yelp/dumb-init https://github.com/Yelp/dumb-init
    - dmitriid 4y ago
      I'm still sad nanokernels or nanokernel-like never took off. I also remember clicking around Ling (Erlang on Xen, sadly no longer active [1]) where the whole VM could boot up, service the request, and shut down in less time than it takes a cloud to start spinning up an instance :) [1] https://github.com/cloudozer/ling https://github.com/cloudozer/ling
- tptacek 4y ago
  There's not much about how our system operates that we're unwilling to talk about. But as you're probably aware, the broad questions of how networking and discovery work are big, so I wouldn't know where to start. Feel free to shoot questions at us, though!
jamesfinlayson 4y ago
I'm not intimately familiar with all of the AWS infrastructure around a running EC2 but I imagine a lot of time is spent on creating the associated Elastic Block Storage, allocating an Elastic IP Address, creating an Elastic Network Interface, creating a default security group etc, then attaching all of those things to the EC2, then attaching the required resources to an Internet Gateway etc.
thdxr 4y ago
what's funny is firecracker, the underlying tech fly is using, was created by AWS only used by Lambda at the moment though, not in ec2
- seabrookmx 4y ago
  And firecracker is a fork of crosvm, created by Google to run the Android app and Debian dev layers in ChromeOS, of all things. Open source is so cool when it works.
pugz 4y ago
I think that's an artifact of RDS specifically. It's dreadfully slow. An EC2 instance will launch and have SSH connectivity in 17-22 seconds in my testing. (I was testing this a fair bit a while back for a silly idea I had)
deleted 4y ago
[deleted]

boulos 4y ago

Disclosure: I used to work on GCE. EC2 has historically not focused much on instance boot time. We did for GCE and drove it down pretty heavily. The post here from fly has a good set of sequence diagrams for "what are the various phases of creating an instance from scratch" that are generally applicable. I'll note though that different users have different targets. Some people care about "time from request to first instruction ticks over" while others only care about "time from request to ssh'able from the public internet". There's an interesting middle ground of "time from request to being able to talk to other services like GCS or S3". It's not clear to me what the networking / discovery story is for a Fly Machine that is stopped and then starts. That is, how long does fly-proxy take to update (globally? within a metro?) to add and remove the new Fly Machine? I vaguely recall that only external endpoints support IPv4, so I assume Fly is reserving and registering the internal IPv6 endpoints in the more expensive "create" step and then "start" is just about propagating liveness.

ZeroCool2u 4y ago
My main gig runs workloads primarily on AWS, but I work with a small company as well that is completely on GCP and I gotta say the difference is night and day in instance allocation and startup time. GCE is so much faster it's infuriating when I've gotta go back to work and sometimes have to wait more than 10 minutes in the worst case for an EC2 instance to finish booting in an EKS cluster that's already running!
derefr 4y ago
> We did for GCE and drove it down pretty heavily. As a heavy GCE user, something weird about GCE to me is that instance boot times can be extremely variable — predictable within a given instance group, but unpredictable with even very small changes to e.g. instance sizes within the same instance family. (And this isn’t the instance blocking on getting scheduled onto a hypervisor; I can recognize that point, because it’s when any quota limits hit to potentially kill the instance provision. That phase of the delay is very stable.) The variable delays also seem to apply to “reset” of the instance (which won’t involve an even-temporary deschedule, as reset keeps NVMe state) — but not to kexec-reboot, if one opts for that. Is GCE built on a hybrid of two different hypervisor systems with wildly differing boot-time performance characteristics, where subtle tweaks of your instance config determine which hypervisor you get? Maybe one that relies on a hardware offload for something (PXE kernel signature verification?) that the other does purely in software? One thing that’s clear to me is that instance types introduced after a certain point (e.g. all n2d instances) are always of the fast-booting type. So I’m guessing this is just old hypervisors being stuck on some legacy config because of long-term customers partially pinning those host machines with workloads that somehow prevent live migration (workloads with NVMe or GPU), and so make those hypervisors unable to be drained for a hard-upgrade.
dmitriid 4y ago
> I'll note though that different users have different targets. Some people care about "time from request to first instruction ticks over" while others only care about "time from request to ssh'able from the public internet". This is the same target: a machine (that usually only has single app on it) shouldn't take more time to boot than a general-purpose consumer PC/laptop. The reason it takes so darn long to start in so many cases is just how horrendously overcomplicated the whole cloud setup is internally and externally (sometimes for good reasons, sometimes because we don't know better, sometimes just because it really is just overly complicated and overengineered)
- nousermane 4y ago
  > shouldn't take more time to boot than a general-purpose consumer PC/laptop. That's an incredibly easy target. VMs can and should boot much faster than that - just look at firecracker hypervisor. Even with KVM, if you replace systemd with something small and simple [0] (which you totally should, for single-app VMs), boot times of couple of seconds are within reach. [0] https://github.com/Yelp/dumb-init https://github.com/Yelp/dumb-init
  - dmitriid 4y ago
    I'm still sad nanokernels or nanokernel-like never took off. I also remember clicking around Ling (Erlang on Xen, sadly no longer active [1]) where the whole VM could boot up, service the request, and shut down in less time than it takes a cloud to start spinning up an instance :) [1] https://github.com/cloudozer/ling https://github.com/cloudozer/ling
tptacek 4y ago
There's not much about how our system operates that we're unwilling to talk about. But as you're probably aware, the broad questions of how networking and discovery work are big, so I wouldn't know where to start. Feel free to shoot questions at us, though!

jamesfinlayson 4y ago

I'm not intimately familiar with all of the AWS infrastructure around a running EC2 but I imagine a lot of time is spent on creating the associated Elastic Block Storage, allocating an Elastic IP Address, creating an Elastic Network Interface, creating a default security group etc, then attaching all of those things to the EC2, then attaching the required resources to an Internet Gateway etc.

thdxr 4y ago

what's funny is firecracker, the underlying tech fly is using, was created by AWS only used by Lambda at the moment though, not in ec2

seabrookmx 4y ago
And firecracker is a fork of crosvm, created by Google to run the Android app and Debian dev layers in ChromeOS, of all things. Open source is so cool when it works.

pugz 4y ago

I think that's an artifact of RDS specifically. It's dreadfully slow. An EC2 instance will launch and have SSH connectivity in 17-22 seconds in my testing. (I was testing this a fair bit a while back for a silly idea I had)

deleted 4y ago

[deleted]