9 ms·

So basically, the latency from "CreateInstance" to "I can SSH in" for the fastest AMI is roughly 3 seconds, with max being 8 seconds. That's actually pretty so

So basically, the latency from "CreateInstance" to "I can SSH in" for the fastest AMI is roughly 3 seconds, with max being 8 seconds. That's actually pretty solid. If it were 5-10x faster than that you could probably fit that into a lot of interesting use-cases for on-demand workloads. The bottleneck is the EC2 allocation itself, and I'd be interested in seeing what warm EC2 instances can do for you there. That said, I think for the majority of use cases boot performance is not particularly important. If you want really fast 'boot' you might just want VMs and containers - that would cut out the 3-8 seconds of instance allocation time, as well as most of the rest of the work. Curious to see a follow up on what's going on with FreeBSD - seems like it takes ages to get the network up.

deleted 5y ago
[deleted]
sjnu 5y ago
Where did you get 3? The fastest numbers I could see in the post add up to 7-8s
- masklinn 5y ago
  Per the post it’s 8.4 and independent of the OS: > The first two values — the time taken for a RunInstances API call to successfully return, and the time taken after RunInstances returns before a DescribeInstances call says that the instance is "running" — are consistent across all the AMIs I tested, at roughly 1.5 and 6.9 seconds respectively “Running to available” is what’s in the table, ranging from 1.23s to 70 or so.
nijave 5y ago
>If it were 5-10x faster than that you could probably fit that into a lot of interesting use-cases for on-demand workloads That's basically lambda although you lose control of the kernel and some of the userspace (although you can use Docker containers and the HTTP interface on Lambda to get some flexibility back) Under the hood, Lambda uses optimized Firecracker VMs for < 1 sec boot >I think for the majority of use cases boot performance is not particularly important Anything with auto scaling. I think CI is probably a big use case (those get torn up and down pretty quickly) and you introduce hairy things like Docker in Docker unprivileged builds trying to run nested in a container
- JoshTriplett 5y ago
  There's no fundamental reason Lambda needs to make you lose control of the kernel; I'd love to have Lambda with a custom kernel, and that doesn't seem like it'd be too difficult to support.
  - nuclearnice1 5y ago
    You can do lambda with containers which should get you close, I think.
    - JoshTriplett 5y ago
      A container does not include the kernel, so it doesn't get any closer. I just want a single static binary and a kernel, not a full container.
      - deleted 5y ago
        [deleted]
      - deleted 5y ago
        [deleted]
      - staticassertion 5y ago
        I wonder why they don't expose a kernel instead of just the rootfs. It's hard to imagine a great reason. Maybe they harden their guest kernel?
        JoshTriplett 5y ago
        At one point, Lambda didn't expose the ability to write custom runtimes, and only provided runtimes for specific languages. People reverse-engineered those runtimes and figured out how to build custom ones. Eventually, Amazon provided a documented, well-supported way to build and run custom runtimes, but that required documenting the interfaces provided to those runtimes (e.g. environment variables and HTTP APIs instead of language-runtime-specific APIs). I'd love to see Lambda support custom kernels. That would require a similar approach: document the actual VM interface provided to the kernel, including the virtual hardware and the mechanism used to expose the Lambda API. I'd guess that they haven't yet due to a combination of 1) not enough demand for Lambda with custom kernels and 2) the freedom to modify the kernel/hardware interface arbitrarily because they control both sides of it.
citrin_ru 5y ago
> Curious to see a follow up on what's going on with FreeBSD - seems like it takes ages to get the network up. FreeBSD rc executes all rc.d stripts sequentially, in one thread. OpenRC AFAIK can start daemons in parallel, but unfortunately switch to OpenRC was abandoned: https://reviews.freebsd.org/D18578 https://reviews.freebsd.org/D18578
cperciva 5y ago
Curious to see a follow up on what's going on with FreeBSD - seems like it takes ages to get the network up. Our BIOS boot loader is very slow. I'll be writing about FreeBSD boot performance in a later post.
- alberth 5y ago
  Hi Colin Do you know if there is any plans on FreeBSD creating a super minimal server version that can be used as a docker host ... similar in size to Alpine Linux.
  - cperciva 5y ago
    I know lots of people have made stripped down versions of FreeBSD, e.g. nanobsd. Beyond that, I'm not aware of anything specific... but I probably wouldn't be since I don't pay much attention to the container space. Try asking on one of the mailing lists maybe?
rawoke083600 5y ago
>use-cases for on-demand workloads. Yea true ! Maybe depending on the availability maybe they can "spawn up" some x-amount of "spare servers" to get the median time even lower..

deleted 5y ago

[deleted]

sjnu 5y ago

Where did you get 3? The fastest numbers I could see in the post add up to 7-8s

masklinn 5y ago
Per the post it’s 8.4 and independent of the OS: > The first two values — the time taken for a RunInstances API call to successfully return, and the time taken after RunInstances returns before a DescribeInstances call says that the instance is "running" — are consistent across all the AMIs I tested, at roughly 1.5 and 6.9 seconds respectively “Running to available” is what’s in the table, ranging from 1.23s to 70 or so.

nijave 5y ago

>If it were 5-10x faster than that you could probably fit that into a lot of interesting use-cases for on-demand workloads That's basically lambda although you lose control of the kernel and some of the userspace (although you can use Docker containers and the HTTP interface on Lambda to get some flexibility back) Under the hood, Lambda uses optimized Firecracker VMs for < 1 sec boot >I think for the majority of use cases boot performance is not particularly important Anything with auto scaling. I think CI is probably a big use case (those get torn up and down pretty quickly) and you introduce hairy things like Docker in Docker unprivileged builds trying to run nested in a container

JoshTriplett 5y ago
There's no fundamental reason Lambda needs to make you lose control of the kernel; I'd love to have Lambda with a custom kernel, and that doesn't seem like it'd be too difficult to support.
- nuclearnice1 5y ago
  You can do lambda with containers which should get you close, I think.
  - JoshTriplett 5y ago
    A container does not include the kernel, so it doesn't get any closer. I just want a single static binary and a kernel, not a full container.
    - deleted 5y ago
      [deleted]
    - deleted 5y ago
      [deleted]
    - staticassertion 5y ago
      I wonder why they don't expose a kernel instead of just the rootfs. It's hard to imagine a great reason. Maybe they harden their guest kernel?
      - JoshTriplett 5y ago
        At one point, Lambda didn't expose the ability to write custom runtimes, and only provided runtimes for specific languages. People reverse-engineered those runtimes and figured out how to build custom ones. Eventually, Amazon provided a documented, well-supported way to build and run custom runtimes, but that required documenting the interfaces provided to those runtimes (e.g. environment variables and HTTP APIs instead of language-runtime-specific APIs). I'd love to see Lambda support custom kernels. That would require a similar approach: document the actual VM interface provided to the kernel, including the virtual hardware and the mechanism used to expose the Lambda API. I'd guess that they haven't yet due to a combination of 1) not enough demand for Lambda with custom kernels and 2) the freedom to modify the kernel/hardware interface arbitrarily because they control both sides of it.

citrin_ru 5y ago

> Curious to see a follow up on what's going on with FreeBSD - seems like it takes ages to get the network up. FreeBSD rc executes all rc.d stripts sequentially, in one thread. OpenRC AFAIK can start daemons in parallel, but unfortunately switch to OpenRC was abandoned: https://reviews.freebsd.org/D18578 https://reviews.freebsd.org/D18578

cperciva 5y ago

Curious to see a follow up on what's going on with FreeBSD - seems like it takes ages to get the network up. Our BIOS boot loader is very slow. I'll be writing about FreeBSD boot performance in a later post.

alberth 5y ago
Hi Colin Do you know if there is any plans on FreeBSD creating a super minimal server version that can be used as a docker host ... similar in size to Alpine Linux.
- cperciva 5y ago
  I know lots of people have made stripped down versions of FreeBSD, e.g. nanobsd. Beyond that, I'm not aware of anything specific... but I probably wouldn't be since I don't pay much attention to the container space. Try asking on one of the mailing lists maybe?

rawoke083600 5y ago

>use-cases for on-demand workloads. Yea true ! Maybe depending on the availability maybe they can "spawn up" some x-amount of "spare servers" to get the median time even lower..