7 ms·

Maybe AWS should actually take a look into this. I know comparing AWS to other (smaller) cloud providers is not totally fair given the size of AWS, but for exam

Maybe AWS should actually take a look into this. I know comparing AWS to other (smaller) cloud providers is not totally fair given the size of AWS, but for example creating / booting an instance in Hetzner takes a few seconds.

matt-p 2y ago
What's size got to do with boot time? Serious question.
- RationPhantoms 2y ago
  More employed eyes on an issue or ability to compensate the best-in-class engineers to take a look.
  - matt-p 2y ago
    Exactly, so a large provider should theoretically have faster boot times if anything, no? More chance of having a customer who cares deeply, more wasted CPU cycles (and therefore incentive to optimise) as it's multiplied by 10 million rather than 10 thousand instance starts. More likely to have larger quantities of engineers available, more monitoring/data/telematics.
- CaptainOfCoit 2y ago
  Smaller companies are faster and more nimble than larger corporations.
  - matt-p 2y ago
    Yes, but why specifically does that mean boot times of a VM are faster? One has to be 'nimble' to improve that?
- develatio 2y ago
  By "the size" I meant to say "the size of the infrastructure", meaning that AWS has to manage orders of magnitude more instances than Hetzner. This might as well contribute to "things" being slower.
  - londons_explore 2y ago
    Arguably it can also make things faster. A small provider might need to migrate other instances around to make space for your new instance, whereas a big provider almost certainly can satisfy your request from existing free capacity, and it should therefore be a matter of milliseconds to identify the physical machine your new VM will run on.
  - matt-p 2y ago
    I understand that, apologies for not making that clear. I guess I'm just struggling to see how having more VMs means it takes longer to provision one? A database query to find the space and get an allocation should be milliseconds in either case.
- playingalong 2y ago
  Likely they mean that following Conway's law in AWS there are more abstraction layers involved.
  - matt-p 2y ago
    Most likely true, but things are going deeply wrong if fresh Debian VM boot times are a function of the organisational structure of your hosting provider.
- ldoughty 2y ago
  The article mentions hydration (a process to reduce the penalty of first access of a data block). This is done by essentially reading the entire EBS volume with the tool like fio or dd. The time it takes to complete this process is relative to the amount of dirty blocks. Therefore, it will take twice as long to hydrate 20 GB of data compared to 10 GB.
tekla 2y ago
They have and I know this because I've hammered them on this because we demand thousands of instances to autoscale very aggressively in 1-3 minutes. Very few people give a shit about initialization times. They care more about instance ready times which is constrained by the OS that is running.
everfrustrated 2y ago
Hetzner does not offer a network block storage comparable to EBS that can be used as a root (bootable) file system. AWS local-attached ephemeral disk are also immediately available but cannot be seeded with data (same as Hetzner they are wiped clean ahead of boot).
- andersa 2y ago
  This is an advantage. EBS is terrible! Literally orders of magnitude slower than modern SSDs.
  - tekla 2y ago
    EBS is great for workloads that dont require SSDs, which most don't. If it does, you can do provisioned which will get you alot more or go NVME.
    - Nextgrid 2y ago
      Even provisioned won't get you the access times of a direct-attached SSD. Speed of light and all that - EBS is using the network under the hood, it's not a direct connection to the host.
      - tekla 2y ago
        Yes I know, and? Thats why I mentioned NVME
  - stingraycharles 2y ago
    Depends on your definition of slow. Throughput-wise, I think it’s fairly decent — we typically set up 4 EBS volumes in raid0 and get 4GB/sec for a really decent price.
    - Nextgrid 2y ago
      Sequential throughput can be fine. Random access is always going to be orders of magnitude slower than a direct-attach disk. Remember why we switched from spinning hard drives to SSDs? Well EBS is like going back to a spinning drive.
torginus 2y ago
It also takes a few seconds on AWS. The guy is comparing setting up a whole new machine from an image, with network and all, to turning on a stopped EC2 instance. The latter takes a few seconds, the former is presumably longer. This is the great relevation of this blog post.
- dylan604 2y ago
  wait, restarting a stopped machine is faster than launching an AMI from scracth is a great revelation? That's like saying waking your MacbookPro is faster than booting from powered off state. Of course it is, and that's precisely why the option exists.
  - mdeeks 2y ago
    If you aren't familiar with how EBS works and how volumes are warmed, then yes, this is an interesting blog post. Not everyone is an expert. They become experts by reading things like this and learning. If you didn't know about this EBS behavior it would be logical to assume that booting from scratch is roughly equivalent to starting/stopping/starting again.
  - jpambrun 2y ago
    I think this is unexpected. I expected that once created, my boot volume would have the same performance on the first boot than on the second. It's really not obvious that the volume is really empty and lazily loaded from S3. The proposed work around is also a bit silly: read all blocks one by one even tho maybe 1% of the block have something in them on a new VM. This is actually a revelation.
  - pluies 2y ago
    It's definitely news to me! Intuitively, I would have expected AWS to send back the EBS volume backing a powered-off machine into the Whatever Long-Term Storage Is Behind EBS, and therefore start-up time to be ~identical to a fresh start as the steps would be the same: retrieve data from long-term storage into a readable EBS volume, start VM etc. It's very interesting that it is not the case, and that keeping the EBS volume around after a first boot makes a second boot so much faster.
attentive 2y ago
It depends on instance type and OS and can be real short on ec2.
flaminHotSpeedo 2y ago
At least if you're using the Ec2 optimized amis, Ec2 instances frequently boot fast enough that they'll be executing your username initialization before you get the run instances response. Though there's a long tail, so sometimes there can be a gap on the order of a couple second between the sync response and when the bootloader transfers over to the kernel.

matt-p 2y ago

What's size got to do with boot time? Serious question.

RationPhantoms 2y ago
More employed eyes on an issue or ability to compensate the best-in-class engineers to take a look.
- matt-p 2y ago
  Exactly, so a large provider should theoretically have faster boot times if anything, no? More chance of having a customer who cares deeply, more wasted CPU cycles (and therefore incentive to optimise) as it's multiplied by 10 million rather than 10 thousand instance starts. More likely to have larger quantities of engineers available, more monitoring/data/telematics.
CaptainOfCoit 2y ago
Smaller companies are faster and more nimble than larger corporations.
- matt-p 2y ago
  Yes, but why specifically does that mean boot times of a VM are faster? One has to be 'nimble' to improve that?
develatio 2y ago
By "the size" I meant to say "the size of the infrastructure", meaning that AWS has to manage orders of magnitude more instances than Hetzner. This might as well contribute to "things" being slower.
- londons_explore 2y ago
  Arguably it can also make things faster. A small provider might need to migrate other instances around to make space for your new instance, whereas a big provider almost certainly can satisfy your request from existing free capacity, and it should therefore be a matter of milliseconds to identify the physical machine your new VM will run on.
- matt-p 2y ago
  I understand that, apologies for not making that clear. I guess I'm just struggling to see how having more VMs means it takes longer to provision one? A database query to find the space and get an allocation should be milliseconds in either case.
playingalong 2y ago
Likely they mean that following Conway's law in AWS there are more abstraction layers involved.
- matt-p 2y ago
  Most likely true, but things are going deeply wrong if fresh Debian VM boot times are a function of the organisational structure of your hosting provider.
ldoughty 2y ago
The article mentions hydration (a process to reduce the penalty of first access of a data block). This is done by essentially reading the entire EBS volume with the tool like fio or dd. The time it takes to complete this process is relative to the amount of dirty blocks. Therefore, it will take twice as long to hydrate 20 GB of data compared to 10 GB.

tekla 2y ago

They have and I know this because I've hammered them on this because we demand thousands of instances to autoscale very aggressively in 1-3 minutes. Very few people give a shit about initialization times. They care more about instance ready times which is constrained by the OS that is running.

everfrustrated 2y ago

Hetzner does not offer a network block storage comparable to EBS that can be used as a root (bootable) file system. AWS local-attached ephemeral disk are also immediately available but cannot be seeded with data (same as Hetzner they are wiped clean ahead of boot).

andersa 2y ago
This is an advantage. EBS is terrible! Literally orders of magnitude slower than modern SSDs.
- tekla 2y ago
  EBS is great for workloads that dont require SSDs, which most don't. If it does, you can do provisioned which will get you alot more or go NVME.
  - Nextgrid 2y ago
    Even provisioned won't get you the access times of a direct-attached SSD. Speed of light and all that - EBS is using the network under the hood, it's not a direct connection to the host.
    - tekla 2y ago
      Yes I know, and? Thats why I mentioned NVME
- stingraycharles 2y ago
  Depends on your definition of slow. Throughput-wise, I think it’s fairly decent — we typically set up 4 EBS volumes in raid0 and get 4GB/sec for a really decent price.
  - Nextgrid 2y ago
    Sequential throughput can be fine. Random access is always going to be orders of magnitude slower than a direct-attach disk. Remember why we switched from spinning hard drives to SSDs? Well EBS is like going back to a spinning drive.

torginus 2y ago

It also takes a few seconds on AWS. The guy is comparing setting up a whole new machine from an image, with network and all, to turning on a stopped EC2 instance. The latter takes a few seconds, the former is presumably longer. This is the great relevation of this blog post.

dylan604 2y ago
wait, restarting a stopped machine is faster than launching an AMI from scracth is a great revelation? That's like saying waking your MacbookPro is faster than booting from powered off state. Of course it is, and that's precisely why the option exists.
- mdeeks 2y ago
  If you aren't familiar with how EBS works and how volumes are warmed, then yes, this is an interesting blog post. Not everyone is an expert. They become experts by reading things like this and learning. If you didn't know about this EBS behavior it would be logical to assume that booting from scratch is roughly equivalent to starting/stopping/starting again.
- jpambrun 2y ago
  I think this is unexpected. I expected that once created, my boot volume would have the same performance on the first boot than on the second. It's really not obvious that the volume is really empty and lazily loaded from S3. The proposed work around is also a bit silly: read all blocks one by one even tho maybe 1% of the block have something in them on a new VM. This is actually a revelation.
- pluies 2y ago
  It's definitely news to me! Intuitively, I would have expected AWS to send back the EBS volume backing a powered-off machine into the Whatever Long-Term Storage Is Behind EBS, and therefore start-up time to be ~identical to a fresh start as the steps would be the same: retrieve data from long-term storage into a readable EBS volume, start VM etc. It's very interesting that it is not the case, and that keeping the EBS volume around after a first boot makes a second boot so much faster.

attentive 2y ago

It depends on instance type and OS and can be real short on ec2.

flaminHotSpeedo 2y ago

At least if you're using the Ec2 optimized amis, Ec2 instances frequently boot fast enough that they'll be executing your username initialization before you get the run instances response. Though there's a long tail, so sometimes there can be a gap on the order of a couple second between the sync response and when the bootloader transfers over to the kernel.