12 ms·

Making EC2 boot time faster

275 points by jacobwg 2y ago

amluto 2y ago
I don’t use EC2 enough to have played with this, but a big part here is the population of the AMI into the per-instance EBS volume. ISTM one could do much better with an immutable/atomic setup: set up an immutable read-only EBS volume, and have each instance share that volume and have a per-instance volume that starts out blank. Actually pulling this off looks like it would be limited by the rules of EBS Multi-Attach. One could have fun experimenting with an extremely minimal boot AMI that streams a squashfs or similar file from S3 and unpacks it. edit: contemplating a bit, unless you are willing to babysit your deployment and operate under serious constraints, EBS multi-attach looks like the wrong solution. I think the right approach would be build a very very small AMI that sets up a rootfs using s3fs or a similar technology and optionally puts an overlayfs on top. Alternatively, it could set up a block device backed by an S3 file and optionally use it as a base layer of a device-mapper stack. There’s plenty of room to optimize this.
- mdaniel 2y ago
  I believe they addressed this in their post because one cannot (currently?) `aws ec2 run-instances --volume-id vol-cafebabe`, rather one can only tell AWS what volume parameters to use when they create the root device. Your theory may still be sound about using some kind of super bare bones AMI but there will be no such outcome of "hey, friend, use this existing EBS as your root volume, don't create a new one"
- stingraycharles 2y ago
  Isn’t EBS multi-attach only available for the (very expensive) io1 / io2 volume types?
  - amluto 2y ago
    Hmm, it does look like it, although one could carefully use large IO. But the bigger issue might be durability. Most EBS types have rather low quoted durability, and, for a shared volume like this, that’s a problem. Using S3 instead would be better all around except for the smallish engineering effort and deployment effort needed. Getting a tool like mkosi to generate a boot-from-S3 setup should be straightforward. Converting most any bootable container should also be doable, even automatically. Converting an AMI would involve more heuristics and be more fragile, but it ought to work reliably with most modern Linux distros.
- Szpadel 2y ago
  we used s3fs in production. please don't use it, it's unreliable, unpredictable failure modes, can bring whole instance down. if you really need something like that use rclone mount
- attentive 2y ago
  That's reinventing ebs/ami/snapshots. They are already doing it i.e. data goes lazily from s3 to ebs/ec2.
  - amluto 2y ago
    It’s not, though. The way to boot a transient OS (just like a transient instance of a container on a machine/instance with a container runtime) is to give userspace read-only access to the image. It can be outright read-only or it can be an actual efficient overlay mechanism (qcow, overlayfs, device-mapper snapshot, etc). EBS, as the article notes, can’t actually do a read-only mount of a snapshot at all, and it’s very inefficient at instantiating a volume from a snapshot.
- antihero 2y ago
  Could you make a snapshot of the booted instance then and boot other instances from that?
  - amluto 2y ago
    That seems like it would have exactly the same problem. The problem is that EBS volumes load very inefficiently from snapshots. (They’re also unnecessarily expensive: you pay for (number of instances times size) despite the fact that what you actually want is for each instance to read exactly the same data.) Fundamentally, EBS volumes are designed to work more or less like actual disks, whereas modern scaled workloads want something closer to net-booted diskless machines.
maccard 2y ago
I don't use GHA as some of our code is stored in Perforce, but we've faced the same challenges with EC2 instance startup times on our self managed runners on a different provider. We would happily pay someone like depot for "here's the AMI I want to run & autoscale, can you please do it faster than AWS?" We hit this problem with containers too - we'd _love_ to just run all our CI on something like fargate and have it automatically scale and respond to our demand, but the response times and rate limting are just _so slow_ that it means instead we just end up starting/stopping instances with a lambda which feels so 2014.
- CaptainOfCoit 2y ago
  > We would happily pay someone like depot for "here's the AMI I want to run & autoscale, can you please do it faster than AWS?" Change that to "here's the ISO/IMG I want to run & autoscale, can you please do it faster than AWS?" and you'll have tons of options. Most platforms using Firecracker would most likely be faster, maybe try to use that as a search vector.
  - maccard 2y ago
    Can you maybe share some examples? We're fine to use other image formats, but a lot of the value of AWS is that the services interact, IAM works nicely together, etc. Fly.io comes up often [0] on HN, but there's an overwhelming amount of "it's a nice idea, but it just doesn't work" feedback on it. [0] https://news.ycombinator.com/item?id=39363499 https://news.ycombinator.com/item?id=39363499
- everfrustrated 2y ago
  Out of curiosity what CI system are you using with Perforce?
  - maccard 2y ago
    We use buildkite with a customised verison of https://github.com/improbable-eng/perforce-buildkite-plugin/ https://github.com/improbable-eng/perforce-buildkite-plugin/ Our game code is in P4, but our backend services are on GH. Having a single CI system means we get easy interop e.g. game updates can trigger backend pipelines and vice versa. In the past I've used TeamCity, Jenkins, and ElectricCommander(!)
immibis 2y ago
There's something to say about building a tower of abstractions and then trying to tear it back down. We used to just run a compiler on a machine. Startup time: 0.001 seconds. Then we'd run a Docker container on a machine. Startup time: 0.01 sections. Fine, if you need that abstraction. Now apparently we're booting full VMs to run compilers - startup time: 5 seconds. But that's not enough, because we're also allocating a bunch of resources in a distributed network - startup time: 40 seconds. Do we actually need all this stuff, or does it suffice to get one really powerful server (price less than $40k) and run Docker on it?
- cjk2 2y ago
  I'm mostly just running the (Go) compiler on my laptop which is considerably faster than on docker and considerably cheaper than the server... I mean an ass end M3 macbook has the same compile time as an i9-14900k. God knows what an equivalent Xeon/Epyc costs...
  - immibis 2y ago
    Maybe your container isn't set up right - Docker contains run directly on the host, just partitioned off from accessing stuff outside of themselves with the equivalent of chroot. Or it could be a Mac-specific thing. Docker only works that way on Linux, and has to emulate Linux on other platforms.
    - yjftsjthsd-h 2y ago
      Right, they said they're on a macbook so unless they're going out of their way to run Linux bare-metal it has to use a VM. And AIUI there are extra footguns in that situation, especially that mapping volumes from the host is slower because instead of just telling the kernel to make the directory visible you have to actually share from the host to the VM. See also: https://reece.tech/posts/osx-docker-performance/ https://reece.tech/posts/osx-docker-performance/ See also: https://docs.docker.com/desktop/settings/mac/ https://docs.docker.com/desktop/settings/mac/ > Shared folders are designed to allow application code to be edited on the host while being executed in containers. For non-code items such as cache directories or databases, the performance will be much better if they are stored in the Linux VM, using a data volume (named volume) or data container.
develatio 2y ago
Maybe AWS should actually take a look into this. I know comparing AWS to other (smaller) cloud providers is not totally fair given the size of AWS, but for example creating / booting an instance in Hetzner takes a few seconds.
- matt-p 2y ago
  What's size got to do with boot time? Serious question.
  - RationPhantoms 2y ago
    More employed eyes on an issue or ability to compensate the best-in-class engineers to take a look.
    - matt-p 2y ago
      Exactly, so a large provider should theoretically have faster boot times if anything, no? More chance of having a customer who cares deeply, more wasted CPU cycles (and therefore incentive to optimise) as it's multiplied by 10 million rather than 10 thousand instance starts. More likely to have larger quantities of engineers available, more monitoring/data/telematics.
  - CaptainOfCoit 2y ago
    Smaller companies are faster and more nimble than larger corporations.
    - matt-p 2y ago
      Yes, but why specifically does that mean boot times of a VM are faster? One has to be 'nimble' to improve that?
  - develatio 2y ago
    By "the size" I meant to say "the size of the infrastructure", meaning that AWS has to manage orders of magnitude more instances than Hetzner. This might as well contribute to "things" being slower.
everfrustrated 2y ago
It's too bad that EBS doesn't natively support Copy-On-Write. Snapshots are persisted into S3 (transparently to the user) but it means each new EBS volume spawned doesn't start at full IOPS allocation. I presume this is due to EBS volumes being specific-AZ so to be able to launch an AMI-seeded EBS volume in any AZ it needs to go via S3 (multi-AZ)
- Twirrim 2y ago
  EBS volumes are "expensive" compared to S3, due to the limitations of what you can do with live block volumes + replicas, vs S3. It takes more disk space to have an image be a provisioned volume ready to be used for copy-on-write, vs having it as something backed up in S3. So the incentives aren't there vs just trying to make the volume creation process as smooth and fast as possible. I'd guess it's likely that EBS is using a tiered caching system, where they'll keep live volumes around for Copy-on-write cloning for the more popular images/snapshots, with slightly less popular images maybe stored in an EBS cache of some form, before it goes all the way back to S3. You're just not likely to end up getting a live volume level of caching until you hit a certain threshold of launches.
cmckn 2y ago
You can enable fast restore on the EBS snapshot that backs your AMI: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-fast-snapshot-restore.html https://docs.aws.amazon.com/ebs/latest/userguide/ebs-fast-sn... It’s not cheap, but it speeds things up.
- stingraycharles 2y ago
  $540/month per EBS volume per AZ. And it’s still fairly limited, at a maximum of 8 credits, it wouldn’t nearly cover the use case described in the article (launching 50 instances quickly).
bingemaker 2y ago
Curious, how do you measure the time taken for those 4 steps listed in "What takes so long?" section?
waiwai933 2y ago
I believe this is similar to EC2 Fast Launch which is available for Windows AMIs, but I don't know exactly how that works under the hood. https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/win-ami-config-fast-launch.html https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/win-a...
- mcbain 2y ago
  It does launch an instance and take a snapshot but what's happening is the sysprep and OOBE stuff that can take 10 mins or so (you can find it in the console and startup logs). That's a lot more overheard than just hydrating an EBS volume. https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/win-ami-config-fast-launch.html#win-fast-launch-key-terms https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/win-a...
necovek 2y ago
From a technical perspective, Amazon has actually optimized this but turned that into "serverless functions": their ultra-optimized image paired with Firecracker achieve ultra-fast boot-up of virtual Linux machines. IIRC from when Firecracker was being introduced, they are booting up in sub-second times. I wonder if Amazon would ever decide to offer booting the same image with the same hypervisor in EC2 as they do for lambdas?
- arianvanp 2y ago
  And AWS now has a product to spin up Lambdas for GitHub Actions CI runners https://docs.aws.amazon.com/codebuild/latest/userguide/action-runner.html https://docs.aws.amazon.com/codebuild/latest/userguide/actio...
- cr125rider 2y ago
  Fargate is an alternative that runs on Firecracker as well. It's hidden behind ECS and EKS, however.
  - scarface_74 2y ago
    And CodeBuild…
  - mochomocha 2y ago
    According to [1] Fargate is actually not using Firecracker, but probably something closer to a single container running in a single-tenant ec2 VM. If true, this makes VM boot-time optimizations and warm pooling even more important for such product. [1]: https://justingarrison.com/blog/2024-02-08-fargate-is-not-firecracker/ https://justingarrison.com/blog/2024-02-08-fargate-is-not-fi...
  - plopz 2y ago
    Fargate is too slow without the container cache you can get with ec2
- 20thr 2y ago
  100% -- EC2's general purpose nature is not in my opinion the best fit for ephemeral use-cases. You'll be constantly fighting the infrastructure as the set of trade-offs and design goals are widely different. This is why CodeSandbox, Namespace, and even fly.io built special-purpose architectures to guarantee extremely start-up time. In the case of Namespace it's ~2sec on cold boots with a set of user-supplied containers, with storage allocations. (Disclaimer, I'm with Namespace -- https://namespace.so https://namespace.so)
crohr 2y ago
> while we can boot the Actions runner within 5 seconds of a job starting, it can take GitHub 10+ seconds to actually deliver that job to the runner This. I went the same route with regards to boot time optimisations for [1] (cleaning up the AMI, cloud-init, etc.), and can boot a VM from cold in 15s (I can't rely on prewarming pools of machines -- even stopped -- since RunsOn doesn't share machines with multiple clients and this would not make sense economically). But the time taken by the official runner binary to load and then get assigned a job by GitHub always takes around 8s, which is more than half of the VM boot time :( At some point it would be great if GitHub could give us a leaner runner binary with less legacy stuff, and tailored for ephemeral runners (that, or reverse-engineer the protocol). [1] https://runs-on.com https://runs-on.com
suryao 2y ago
This is very cool optimization. I make a similar product offering fast Github actions runners[1] and we've been down this rabbit hole of boot time optimization. Eventually, we realized that the best solution is to actually build scale. There are two factors in your favor then: 1) Spikes are less pronounced and the workloads are a lot more predictable. 2) The predictability means that you have a decent estimate of the workload to expect at any given time, within reason for maintaining an efficient warm pool. This enables us to simplify the stack and not have high-maintenance optimizations while delivering great user experience. We have some pretty heavy use customers that enable us to do this. [1] https://www.warpbuild.com https://www.warpbuild.com
- matt-p 2y ago
  This is almost always the answer, adding an instance should be a fairly rare event.
Nextgrid 2y ago
I don't get why they're using EBS here to begin with. EBS trades off cost and performance for durability. It's slow because it's a network-attached volume that's most likely also replicated under the hood. You use this for data that you need high durability for. It looks like their use-case fetches all the data it needs from the network (in the form of the GH Actions runner getting the job from GitHub, and then pulling down Docker containers, etc). What they need is a minimal Linux install (Arch Linux would be good for this) in a squashfs/etc and the only thing in EBS should be an HTTP-aware boot loader like IPXE or a kernel+initrd capable of pulling down the squashfs from S3 and run it from memory. Local "scratchspace" storage for the build jobs can be provided by the ephemeral NVME drives which are also direct-attach and much faster than EBS.
- jedberg 2y ago
  By using EBS they don't have to wait for disk to fill from network on second+ boot.
  - Nextgrid 2y ago
    Ah so they are keeping the machines around? Do they need to do that - does the GH runner actually persist anything worth keeping in between runs?
    - jedberg 2y ago
      They keep the instances in a "stopped" state, which means keeping the EBS volume around (and paying for it) but not paying for the instance (which could be another machine when turn it back on, which is why you can't load it into scratch space and then stop it). What's on the EBS is their docker image, so they don't have to load it back up again.
      - Nextgrid 2y ago
        Makes sense. I still think it would be cheaper to just reload it from S3 (straight into memory, not using EBS at all) on every boot. The entire OS shouldn't be more than a gigabyte which is quite fast to download as a bulk transfer straight into RAM.
jedberg 2y ago
Boot time is the number one factor in your success with auto-scaling. The smaller your boot time, the smaller your prediction window needs to be. Ex. If your boot time is five minutes, you need to predict what your traffic will be in five minutes, but if you can boot in 20 seconds, you only need to predict 20 seconds ahead. By definition your predictions will be more accurate the smaller the window is. But! Autoscaling serves two purposes. One is to address load spikes. The other is to reduce costs with scaling down. What this solution does is trade off some of the cost savings by prewarming the EBS volumes and then paying for them. This feels like a reasonable tradeoff if you can justify the cost with better auto-scaling. And if you're not autoscaling, it's still worth the cost if the trade off is having your engineers wait around for instance boots.
- sfilmeyer 2y ago
  >By definition your predictions will be more accurate the smaller the window is. Small nit, and this doesn't detract from your points. I don't think this is universally true by definition, even if it is almost always true. You could come up with some rare conditions where your traffic at t+5 minutes is actually easier to predict than at t+20 seconds. Of course, even in that case you're better off (ceteris paribus) being able to spin things up in 20 seconds.
  - jedberg 2y ago
    I can come up with a lot of examples where it is easier to predict further out[0], but that also means I can predict them 20 seconds out. :) [0] For example I can tell you exactly when spikes will happen to Netflix's servers on Saturday morning (because the kids all get up at the same time). And I can tell you there will be spikes on the hour during prime time as people shift from linear TV to streaming (or at least they did a lot more 10 years ago!). I can also tell you when spikes to Alexa will be because I already know what times peoples alarms are set for.
    - foota 2y ago
      Hm... I think there are things with one sided uncertainty, that make the ops example possible. For example, things rarely start before they're scheduled, so if you know something is supposed to happen at 5 and last an hour, it's reasonable to assume it will be happening at 5:10, so if it's 4:59 right now, you're more certain that it will happen in 11 minutes than in one minute. Of course, as time progresses your uncertainty curve flattens out (at 5:09 you're pretty certain it'll happen any moment now), but that's only in the future. Even then, it's quite reasonable to say that there's no time leading up to the event where you're more confident it will start imminently than you're confident it will be occurring further out.
paulddraper 2y ago
> From a billing perspective, AWS does not charge for the EC2 instance itself when stopped, as there's no physical hardware being reserved; a stopped instance is just the configuration that will be used when the instance is started next. Note that you do pay for the root EBS volume though, as it's still consuming storage. Shutdown standbys absolutely the way to do it. Does AWS offer anything for this, because it's very tedious to set this up.
- tekla 2y ago
  Warm pools
  - paulddraper 2y ago
    yep, that's it, thank you kind person
mnutt 2y ago
They talk about the limitations of the EC2 autoscaler and mention calling LaunchInstances themselves, but are there any autoscaler service projects for EC2 ASGs out there? The AWS-provided one is slow (as they mention), annoyingly opaque, and has all kinds of limitations like not being able to use Warm Pools with multiple instance types etc.
- mdaniel 2y ago
  I am a little confused by your mention of "EC2 autoscaler" and then "EC2 ASG" autoscaler, but if I'm hearing you correctly and you'd want "self managed ASGs," then you may have some success adapting Keda <https://github.com/kedacore/keda#readme https://github.com/kedacore/keda#readme> (or your-favorite-event-driven-gizmo) to monitor the metrics that interest you and driving ec2.LaunchInstances on the other side, since as very best I can tell that's what ASGs are doing just using their serverless-event-something-or-other versus your serverless-event-something-or-other. I would suspect you could even continue to use the existing ec2.LaunchTemplate as the "stamp out copies of these" system, since there doesn't appear to be anything especially ASG-y about them, just that is the only(?) consumer thus far After having typed all that out, I recalled that Open Stack exists and thus they may get you ever further toward your goal since they are trying to be on-prem AWS: https://docs.openstack.org/auto-scaling-sig/latest/theory-of-auto-scaling.html#components-of-auto-scaling https://docs.openstack.org/auto-scaling-sig/latest/theory-of...
  - flaminHotSpeedo 2y ago
    Yeah that's basically what asg does, you can see the createFleet requests in cloudtrail
deleted 2y ago
[deleted]
fduran 2y ago
So I've created ~300k ec2 instances with SadServers and my experience was that starting an ec2 VM from stopped took ~30 seconds and creating one from AMI took ~50 seconds. Recently I decided to actually look at boot times since I store in the db when the servers are requested and when they become ready and it turns out for me it's really bi-modal; some take about 15-20s and many take about 80s, see graph https://x.com/sadservers_com/status/1782081065672118367 https://x.com/sadservers_com/status/1782081065672118367 Pretty baffled by this (same region, same pretty much everything), any idea why?. Definitively going to try this trick in the article.
- fletchowns 2y ago
  Perhaps in one case you are getting a slice of a machine that is already running, versus AWS powering up a machine that was offline and getting a slice of that one?
  - fduran 2y ago
    Yes, some internal (AWS operation) explanation like the one you suggest makes sense.
- paranoidrobot 2y ago
  My guess is probably related to AWS Spot capacity. The second and third spikes at 80 and 140 seconds lines up nicely with this kind of behavior. The second spike would be optimised workloads that can respond to spot interruption in under 60 seconds. The third spike would be Spot workloads that are being force-terminated. The reason it's falling on those bounds is because of whatever is trying to schedule your workload only re-checks for free capacity once a minute. I used to be able to spin up spot instances and basically never get interruptions. They'd stay on for weeks/months. In my experience, it used to be fairly safe to have Spot instances for most workloads. You'd almost never get Spot interruptions. Now, some regions and instance types are difficult to run Spot instances at all.
  - fduran 2y ago
    Thanks, pot capacity being scheduled differently would explain the behavior. Almost all my ec2 instances are spot, and actually I can compare the distribution with the on-demand ones. My spot instances are very short lived (15-30 mins max) and AFAIK I've never seen a spot instance force-terminated (this would be hard to find I think).
uavoperator 2y ago
This is really only tangentially related to the article, but >If AWS responds that there is no current capacity for m7a instances, the instance is updated to a backup type (like m7i) and started again Any ideas why m7i would be chosen as the backup type rather than the other way around? m7a seems to be more expensive than m7i, so maybe there's some performance advantage or something else I'm missing that makes AMD CPU containing instances preferable to Intel ones?
- crohr 2y ago
  m7i CPU is in the same ballpark figure than m7a (https://runs-on.com/benchmarks/aws-ec2-instances/ https://runs-on.com/benchmarks/aws-ec2-instances/). When you look at the interruption percentage for m7a I think m7i (not m7i-flex if you don't want burstable instances) is probably the better choice. But I suppose it depends on availability in their specific zones.
- tenplusfive 2y ago
  At least with other instance types (m5,m6,t3) it was the case that the AMD processors were cheaper. As it turns out, this does not seem to be a general rule. It seems like performance wise the AMD processors are (in certain workloads) quite a bit faster than their Intel equivalent: https://www.phoronix.com/review/aws-m7a-ec2-benchmarks/2 https://www.phoronix.com/review/aws-m7a-ec2-benchmarks/2 (in later pages it seems to be a little bit more mixed)
broknbottle 2y ago
minimizing the image definitely helps. https://aws.amazon.com/blogs/apn/how-to-build-sparse-ebs-volumes-for-fun-and-easy-snapshotting/ https://aws.amazon.com/blogs/apn/how-to-build-sparse-ebs-vol... https://netflixtechblog.medium.com/datastore-flash-upgrades-187f1e4ef859 https://netflixtechblog.medium.com/datastore-flash-upgrades-... https://github.com/Netflix-Skunkworks/s3-flash-bootloader https://github.com/Netflix-Skunkworks/s3-flash-bootloader
orf 2y ago
It seems that you want to make your root volume as small as possible, and use it to only attach a pre-warmed pool of EBS volumes at launch time that contain the actual config/data you need? You can launch a stripped down distribution with what, a 200mb disk? Then attach the “useful” EBS volume, and “do stuff” with that - launch a container, or whatever.
elchief 2y ago
I've noticed that Amazon linux 2023 boots faster than Ubuntu too
albert_e 2y ago
AWS will (should) make this an optional feature. Often the technology is the easier part. The difficult part is how to name the feature intuitively, adding to an ocean of jargon and documentation, and making the configuration knobs intuitive both in UI and CLI/SDK. Amazon Simple Compute Service :) ?
- kylegalbraith 2y ago
  Other founder of Depot here. AWS is pretty close to this idea with their Warm Pools [0]. But for our use case, they're just too slow to react to changes. We observed 60s+ to notice a change and actually start the machine. That doesn't work when we need to launch the machine as quickly as possible in reaction to a pending GHA job. That said, I think this is a problem they could likely solve with that functionality, and we'd love to use it. [0] https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-warm-pools.html https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-au...
- immibis 2y ago
  AWS Lambda.
- ldoughty 2y ago
  The article talks about fio to warm the drive... That's basically fast snapshot restore[0]. This reduces the "first access" penalty for "dirty" blocks. This is probably the slowest part of the entire article (it's about 10 seconds per dirty GB to fio the disk). [0]: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-fast-snapshot-restore.html https://docs.aws.amazon.com/ebs/latest/userguide/ebs-fast-sn...
nathants 2y ago
in the us-west-2-lax-1a local zone, i just booted 100 r5.xlarge spot instances as fortnite like game servers[1]. 1 to be a central server, 99 to be fake players. the server broadcasts x100 write amplified data from every player to every player. the 101st serve is my local pc. the server broadcasts at 200 MB/s[2]. the whole setup costs me $3-4 usd/hour and by far the slowest part of boot is my game compiling on the central server, whether i store ccache data in s3 or not. i've booted this every day for the last 6 months, to test the game. if your system can't handle 30s vm boots, your system should improve. 1. https://r2.nathants.workers.dev/ec2_snitch.png https://r2.nathants.workers.dev/ec2_snitch.png 2. https://r2.nathants.workers.dev/ec2_boot.mp4 https://r2.nathants.workers.dev/ec2_boot.mp4
solatic 2y ago
Makes me wonder why Depot isn't moving to on-prem hardware. When you're reselling compute with a better API, you give up a substantial proportion of your profits to the hyperscaler while offering worse performance (due to being held hostage to the hyperscaler's design decisions, like lazy loading root EBS from S3). Surely an optimized approach here looks something like booting customer CI workloads directly from the hypervisor, using an ISO/squashfs/etc. stored directly on the hypervisor, where the only networked disks are the ones with the customers' BuildKit caches?
gtirloni 2y ago
> tl;dr — boot the instance once, shut the instance down, then boot it again when needed their own tldr should be at the top not middle of the article :)
deleted 2y ago
[deleted]
pid-1 2y ago
Can you have warm pools of spot instances?
thefaux 2y ago
Whenever I see the flaws in aws ux, I remember that they bill by the hour.
mediumsmart 2y ago
Very cool. How many seconds of the faster boot time fit into one regular second?

275 points by jacobwg 2y ago

I don’t use EC2 enough to have played with this, but a big part here is the population of the AMI into the per-instance EBS volume. ISTM one could do much better with an immutable/atomic setup: set up an immutable read-only EBS volume, and have each instance share that volume and have a per-instance volume that starts out blank. Actually pulling this off looks like it would be limited by the rules of EBS Multi-Attach. One could have fun experimenting with an extremely minimal boot AMI that streams a squashfs or similar file from S3 and unpacks it. edit: contemplating a bit, unless you are willing to babysit your deployment and operate under serious constraints, EBS multi-attach looks like the wrong solution. I think the right approach would be build a very very small AMI that sets up a rootfs using s3fs or a similar technology and optionally puts an overlayfs on top. Alternatively, it could set up a block device backed by an S3 file and optionally use it as a base layer of a device-mapper stack. There’s plenty of room to optimize this.

mdaniel 2y ago
I believe they addressed this in their post because one cannot (currently?) `aws ec2 run-instances --volume-id vol-cafebabe`, rather one can only tell AWS what volume parameters to use when they create the root device. Your theory may still be sound about using some kind of super bare bones AMI but there will be no such outcome of "hey, friend, use this existing EBS as your root volume, don't create a new one"
stingraycharles 2y ago
Isn’t EBS multi-attach only available for the (very expensive) io1 / io2 volume types?
- amluto 2y ago
  Hmm, it does look like it, although one could carefully use large IO. But the bigger issue might be durability. Most EBS types have rather low quoted durability, and, for a shared volume like this, that’s a problem. Using S3 instead would be better all around except for the smallish engineering effort and deployment effort needed. Getting a tool like mkosi to generate a boot-from-S3 setup should be straightforward. Converting most any bootable container should also be doable, even automatically. Converting an AMI would involve more heuristics and be more fragile, but it ought to work reliably with most modern Linux distros.
Szpadel 2y ago
we used s3fs in production. please don't use it, it's unreliable, unpredictable failure modes, can bring whole instance down. if you really need something like that use rclone mount
attentive 2y ago
That's reinventing ebs/ami/snapshots. They are already doing it i.e. data goes lazily from s3 to ebs/ec2.
- amluto 2y ago
  It’s not, though. The way to boot a transient OS (just like a transient instance of a container on a machine/instance with a container runtime) is to give userspace read-only access to the image. It can be outright read-only or it can be an actual efficient overlay mechanism (qcow, overlayfs, device-mapper snapshot, etc). EBS, as the article notes, can’t actually do a read-only mount of a snapshot at all, and it’s very inefficient at instantiating a volume from a snapshot.
antihero 2y ago
Could you make a snapshot of the booted instance then and boot other instances from that?
- amluto 2y ago
  That seems like it would have exactly the same problem. The problem is that EBS volumes load very inefficiently from snapshots. (They’re also unnecessarily expensive: you pay for (number of instances times size) despite the fact that what you actually want is for each instance to read exactly the same data.) Fundamentally, EBS volumes are designed to work more or less like actual disks, whereas modern scaled workloads want something closer to net-booted diskless machines.

maccard 2y ago

I don't use GHA as some of our code is stored in Perforce, but we've faced the same challenges with EC2 instance startup times on our self managed runners on a different provider. We would happily pay someone like depot for "here's the AMI I want to run & autoscale, can you please do it faster than AWS?" We hit this problem with containers too - we'd _love_ to just run all our CI on something like fargate and have it automatically scale and respond to our demand, but the response times and rate limting are just _so slow_ that it means instead we just end up starting/stopping instances with a lambda which feels so 2014.

CaptainOfCoit 2y ago
> We would happily pay someone like depot for "here's the AMI I want to run & autoscale, can you please do it faster than AWS?" Change that to "here's the ISO/IMG I want to run & autoscale, can you please do it faster than AWS?" and you'll have tons of options. Most platforms using Firecracker would most likely be faster, maybe try to use that as a search vector.
- maccard 2y ago
  Can you maybe share some examples? We're fine to use other image formats, but a lot of the value of AWS is that the services interact, IAM works nicely together, etc. Fly.io comes up often [0] on HN, but there's an overwhelming amount of "it's a nice idea, but it just doesn't work" feedback on it. [0] https://news.ycombinator.com/item?id=39363499 https://news.ycombinator.com/item?id=39363499
everfrustrated 2y ago
Out of curiosity what CI system are you using with Perforce?
- maccard 2y ago
  We use buildkite with a customised verison of https://github.com/improbable-eng/perforce-buildkite-plugin/ https://github.com/improbable-eng/perforce-buildkite-plugin/ Our game code is in P4, but our backend services are on GH. Having a single CI system means we get easy interop e.g. game updates can trigger backend pipelines and vice versa. In the past I've used TeamCity, Jenkins, and ElectricCommander(!)

immibis 2y ago

There's something to say about building a tower of abstractions and then trying to tear it back down. We used to just run a compiler on a machine. Startup time: 0.001 seconds. Then we'd run a Docker container on a machine. Startup time: 0.01 sections. Fine, if you need that abstraction. Now apparently we're booting full VMs to run compilers - startup time: 5 seconds. But that's not enough, because we're also allocating a bunch of resources in a distributed network - startup time: 40 seconds. Do we actually need all this stuff, or does it suffice to get one really powerful server (price less than $40k) and run Docker on it?

cjk2 2y ago
I'm mostly just running the (Go) compiler on my laptop which is considerably faster than on docker and considerably cheaper than the server... I mean an ass end M3 macbook has the same compile time as an i9-14900k. God knows what an equivalent Xeon/Epyc costs...
- immibis 2y ago
  Maybe your container isn't set up right - Docker contains run directly on the host, just partitioned off from accessing stuff outside of themselves with the equivalent of chroot. Or it could be a Mac-specific thing. Docker only works that way on Linux, and has to emulate Linux on other platforms.
  - yjftsjthsd-h 2y ago
    Right, they said they're on a macbook so unless they're going out of their way to run Linux bare-metal it has to use a VM. And AIUI there are extra footguns in that situation, especially that mapping volumes from the host is slower because instead of just telling the kernel to make the directory visible you have to actually share from the host to the VM. See also: https://reece.tech/posts/osx-docker-performance/ https://reece.tech/posts/osx-docker-performance/ See also: https://docs.docker.com/desktop/settings/mac/ https://docs.docker.com/desktop/settings/mac/ > Shared folders are designed to allow application code to be edited on the host while being executed in containers. For non-code items such as cache directories or databases, the performance will be much better if they are stored in the Linux VM, using a data volume (named volume) or data container.

develatio 2y ago

Maybe AWS should actually take a look into this. I know comparing AWS to other (smaller) cloud providers is not totally fair given the size of AWS, but for example creating / booting an instance in Hetzner takes a few seconds.

matt-p 2y ago
What's size got to do with boot time? Serious question.
- RationPhantoms 2y ago
  More employed eyes on an issue or ability to compensate the best-in-class engineers to take a look.
  - matt-p 2y ago
    Exactly, so a large provider should theoretically have faster boot times if anything, no? More chance of having a customer who cares deeply, more wasted CPU cycles (and therefore incentive to optimise) as it's multiplied by 10 million rather than 10 thousand instance starts. More likely to have larger quantities of engineers available, more monitoring/data/telematics.
- CaptainOfCoit 2y ago
  Smaller companies are faster and more nimble than larger corporations.
  - matt-p 2y ago
    Yes, but why specifically does that mean boot times of a VM are faster? One has to be 'nimble' to improve that?
- develatio 2y ago
  By "the size" I meant to say "the size of the infrastructure", meaning that AWS has to manage orders of magnitude more instances than Hetzner. This might as well contribute to "things" being slower.

everfrustrated 2y ago

It's too bad that EBS doesn't natively support Copy-On-Write. Snapshots are persisted into S3 (transparently to the user) but it means each new EBS volume spawned doesn't start at full IOPS allocation. I presume this is due to EBS volumes being specific-AZ so to be able to launch an AMI-seeded EBS volume in any AZ it needs to go via S3 (multi-AZ)

Twirrim 2y ago
EBS volumes are "expensive" compared to S3, due to the limitations of what you can do with live block volumes + replicas, vs S3. It takes more disk space to have an image be a provisioned volume ready to be used for copy-on-write, vs having it as something backed up in S3. So the incentives aren't there vs just trying to make the volume creation process as smooth and fast as possible. I'd guess it's likely that EBS is using a tiered caching system, where they'll keep live volumes around for Copy-on-write cloning for the more popular images/snapshots, with slightly less popular images maybe stored in an EBS cache of some form, before it goes all the way back to S3. You're just not likely to end up getting a live volume level of caching until you hit a certain threshold of launches.

cmckn 2y ago

You can enable fast restore on the EBS snapshot that backs your AMI: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-fast-snapshot-restore.html https://docs.aws.amazon.com/ebs/latest/userguide/ebs-fast-sn... It’s not cheap, but it speeds things up.

stingraycharles 2y ago
$540/month per EBS volume per AZ. And it’s still fairly limited, at a maximum of 8 credits, it wouldn’t nearly cover the use case described in the article (launching 50 instances quickly).

bingemaker 2y ago

Curious, how do you measure the time taken for those 4 steps listed in "What takes so long?" section?

waiwai933 2y ago

I believe this is similar to EC2 Fast Launch which is available for Windows AMIs, but I don't know exactly how that works under the hood. https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/win-ami-config-fast-launch.html https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/win-a...

mcbain 2y ago
It does launch an instance and take a snapshot but what's happening is the sysprep and OOBE stuff that can take 10 mins or so (you can find it in the console and startup logs). That's a lot more overheard than just hydrating an EBS volume. https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/win-ami-config-fast-launch.html#win-fast-launch-key-terms https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/win-a...

necovek 2y ago

From a technical perspective, Amazon has actually optimized this but turned that into "serverless functions": their ultra-optimized image paired with Firecracker achieve ultra-fast boot-up of virtual Linux machines. IIRC from when Firecracker was being introduced, they are booting up in sub-second times. I wonder if Amazon would ever decide to offer booting the same image with the same hypervisor in EC2 as they do for lambdas?

arianvanp 2y ago
And AWS now has a product to spin up Lambdas for GitHub Actions CI runners https://docs.aws.amazon.com/codebuild/latest/userguide/action-runner.html https://docs.aws.amazon.com/codebuild/latest/userguide/actio...
cr125rider 2y ago
Fargate is an alternative that runs on Firecracker as well. It's hidden behind ECS and EKS, however.
- scarface_74 2y ago
  And CodeBuild…
- mochomocha 2y ago
  According to [1] Fargate is actually not using Firecracker, but probably something closer to a single container running in a single-tenant ec2 VM. If true, this makes VM boot-time optimizations and warm pooling even more important for such product. [1]: https://justingarrison.com/blog/2024-02-08-fargate-is-not-firecracker/ https://justingarrison.com/blog/2024-02-08-fargate-is-not-fi...
- plopz 2y ago
  Fargate is too slow without the container cache you can get with ec2
20thr 2y ago
100% -- EC2's general purpose nature is not in my opinion the best fit for ephemeral use-cases. You'll be constantly fighting the infrastructure as the set of trade-offs and design goals are widely different. This is why CodeSandbox, Namespace, and even fly.io built special-purpose architectures to guarantee extremely start-up time. In the case of Namespace it's ~2sec on cold boots with a set of user-supplied containers, with storage allocations. (Disclaimer, I'm with Namespace -- https://namespace.so https://namespace.so)

crohr 2y ago

> while we can boot the Actions runner within 5 seconds of a job starting, it can take GitHub 10+ seconds to actually deliver that job to the runner This. I went the same route with regards to boot time optimisations for [1] (cleaning up the AMI, cloud-init, etc.), and can boot a VM from cold in 15s (I can't rely on prewarming pools of machines -- even stopped -- since RunsOn doesn't share machines with multiple clients and this would not make sense economically). But the time taken by the official runner binary to load and then get assigned a job by GitHub always takes around 8s, which is more than half of the VM boot time :( At some point it would be great if GitHub could give us a leaner runner binary with less legacy stuff, and tailored for ephemeral runners (that, or reverse-engineer the protocol). [1] https://runs-on.com https://runs-on.com

suryao 2y ago

This is very cool optimization. I make a similar product offering fast Github actions runners[1] and we've been down this rabbit hole of boot time optimization. Eventually, we realized that the best solution is to actually build scale. There are two factors in your favor then: 1) Spikes are less pronounced and the workloads are a lot more predictable. 2) The predictability means that you have a decent estimate of the workload to expect at any given time, within reason for maintaining an efficient warm pool. This enables us to simplify the stack and not have high-maintenance optimizations while delivering great user experience. We have some pretty heavy use customers that enable us to do this. [1] https://www.warpbuild.com https://www.warpbuild.com

matt-p 2y ago
This is almost always the answer, adding an instance should be a fairly rare event.

Nextgrid 2y ago

I don't get why they're using EBS here to begin with. EBS trades off cost and performance for durability. It's slow because it's a network-attached volume that's most likely also replicated under the hood. You use this for data that you need high durability for. It looks like their use-case fetches all the data it needs from the network (in the form of the GH Actions runner getting the job from GitHub, and then pulling down Docker containers, etc). What they need is a minimal Linux install (Arch Linux would be good for this) in a squashfs/etc and the only thing in EBS should be an HTTP-aware boot loader like IPXE or a kernel+initrd capable of pulling down the squashfs from S3 and run it from memory. Local "scratchspace" storage for the build jobs can be provided by the ephemeral NVME drives which are also direct-attach and much faster than EBS.

jedberg 2y ago
By using EBS they don't have to wait for disk to fill from network on second+ boot.
- Nextgrid 2y ago
  Ah so they are keeping the machines around? Do they need to do that - does the GH runner actually persist anything worth keeping in between runs?
  - jedberg 2y ago
    They keep the instances in a "stopped" state, which means keeping the EBS volume around (and paying for it) but not paying for the instance (which could be another machine when turn it back on, which is why you can't load it into scratch space and then stop it). What's on the EBS is their docker image, so they don't have to load it back up again.
    - Nextgrid 2y ago
      Makes sense. I still think it would be cheaper to just reload it from S3 (straight into memory, not using EBS at all) on every boot. The entire OS shouldn't be more than a gigabyte which is quite fast to download as a bulk transfer straight into RAM.

jedberg 2y ago

Boot time is the number one factor in your success with auto-scaling. The smaller your boot time, the smaller your prediction window needs to be. Ex. If your boot time is five minutes, you need to predict what your traffic will be in five minutes, but if you can boot in 20 seconds, you only need to predict 20 seconds ahead. By definition your predictions will be more accurate the smaller the window is. But! Autoscaling serves two purposes. One is to address load spikes. The other is to reduce costs with scaling down. What this solution does is trade off some of the cost savings by prewarming the EBS volumes and then paying for them. This feels like a reasonable tradeoff if you can justify the cost with better auto-scaling. And if you're not autoscaling, it's still worth the cost if the trade off is having your engineers wait around for instance boots.

sfilmeyer 2y ago
>By definition your predictions will be more accurate the smaller the window is. Small nit, and this doesn't detract from your points. I don't think this is universally true by definition, even if it is almost always true. You could come up with some rare conditions where your traffic at t+5 minutes is actually easier to predict than at t+20 seconds. Of course, even in that case you're better off (ceteris paribus) being able to spin things up in 20 seconds.
- jedberg 2y ago
  I can come up with a lot of examples where it is easier to predict further out[0], but that also means I can predict them 20 seconds out. :) [0] For example I can tell you exactly when spikes will happen to Netflix's servers on Saturday morning (because the kids all get up at the same time). And I can tell you there will be spikes on the hour during prime time as people shift from linear TV to streaming (or at least they did a lot more 10 years ago!). I can also tell you when spikes to Alexa will be because I already know what times peoples alarms are set for.
  - foota 2y ago
    Hm... I think there are things with one sided uncertainty, that make the ops example possible. For example, things rarely start before they're scheduled, so if you know something is supposed to happen at 5 and last an hour, it's reasonable to assume it will be happening at 5:10, so if it's 4:59 right now, you're more certain that it will happen in 11 minutes than in one minute. Of course, as time progresses your uncertainty curve flattens out (at 5:09 you're pretty certain it'll happen any moment now), but that's only in the future. Even then, it's quite reasonable to say that there's no time leading up to the event where you're more confident it will start imminently than you're confident it will be occurring further out.

paulddraper 2y ago

> From a billing perspective, AWS does not charge for the EC2 instance itself when stopped, as there's no physical hardware being reserved; a stopped instance is just the configuration that will be used when the instance is started next. Note that you do pay for the root EBS volume though, as it's still consuming storage. Shutdown standbys absolutely the way to do it. Does AWS offer anything for this, because it's very tedious to set this up.

tekla 2y ago
Warm pools
- paulddraper 2y ago
  yep, that's it, thank you kind person

mnutt 2y ago

They talk about the limitations of the EC2 autoscaler and mention calling LaunchInstances themselves, but are there any autoscaler service projects for EC2 ASGs out there? The AWS-provided one is slow (as they mention), annoyingly opaque, and has all kinds of limitations like not being able to use Warm Pools with multiple instance types etc.

mdaniel 2y ago
I am a little confused by your mention of "EC2 autoscaler" and then "EC2 ASG" autoscaler, but if I'm hearing you correctly and you'd want "self managed ASGs," then you may have some success adapting Keda <https://github.com/kedacore/keda#readme https://github.com/kedacore/keda#readme> (or your-favorite-event-driven-gizmo) to monitor the metrics that interest you and driving ec2.LaunchInstances on the other side, since as very best I can tell that's what ASGs are doing just using their serverless-event-something-or-other versus your serverless-event-something-or-other. I would suspect you could even continue to use the existing ec2.LaunchTemplate as the "stamp out copies of these" system, since there doesn't appear to be anything especially ASG-y about them, just that is the only(?) consumer thus far After having typed all that out, I recalled that Open Stack exists and thus they may get you ever further toward your goal since they are trying to be on-prem AWS: https://docs.openstack.org/auto-scaling-sig/latest/theory-of-auto-scaling.html#components-of-auto-scaling https://docs.openstack.org/auto-scaling-sig/latest/theory-of...
- flaminHotSpeedo 2y ago
  Yeah that's basically what asg does, you can see the createFleet requests in cloudtrail

deleted 2y ago

[deleted]

fduran 2y ago

So I've created ~300k ec2 instances with SadServers and my experience was that starting an ec2 VM from stopped took ~30 seconds and creating one from AMI took ~50 seconds. Recently I decided to actually look at boot times since I store in the db when the servers are requested and when they become ready and it turns out for me it's really bi-modal; some take about 15-20s and many take about 80s, see graph https://x.com/sadservers_com/status/1782081065672118367 https://x.com/sadservers_com/status/1782081065672118367 Pretty baffled by this (same region, same pretty much everything), any idea why?. Definitively going to try this trick in the article.

fletchowns 2y ago
Perhaps in one case you are getting a slice of a machine that is already running, versus AWS powering up a machine that was offline and getting a slice of that one?
- fduran 2y ago
  Yes, some internal (AWS operation) explanation like the one you suggest makes sense.
paranoidrobot 2y ago
My guess is probably related to AWS Spot capacity. The second and third spikes at 80 and 140 seconds lines up nicely with this kind of behavior. The second spike would be optimised workloads that can respond to spot interruption in under 60 seconds. The third spike would be Spot workloads that are being force-terminated. The reason it's falling on those bounds is because of whatever is trying to schedule your workload only re-checks for free capacity once a minute. I used to be able to spin up spot instances and basically never get interruptions. They'd stay on for weeks/months. In my experience, it used to be fairly safe to have Spot instances for most workloads. You'd almost never get Spot interruptions. Now, some regions and instance types are difficult to run Spot instances at all.
- fduran 2y ago
  Thanks, pot capacity being scheduled differently would explain the behavior. Almost all my ec2 instances are spot, and actually I can compare the distribution with the on-demand ones. My spot instances are very short lived (15-30 mins max) and AFAIK I've never seen a spot instance force-terminated (this would be hard to find I think).

uavoperator 2y ago

This is really only tangentially related to the article, but >If AWS responds that there is no current capacity for m7a instances, the instance is updated to a backup type (like m7i) and started again Any ideas why m7i would be chosen as the backup type rather than the other way around? m7a seems to be more expensive than m7i, so maybe there's some performance advantage or something else I'm missing that makes AMD CPU containing instances preferable to Intel ones?

crohr 2y ago
m7i CPU is in the same ballpark figure than m7a (https://runs-on.com/benchmarks/aws-ec2-instances/ https://runs-on.com/benchmarks/aws-ec2-instances/). When you look at the interruption percentage for m7a I think m7i (not m7i-flex if you don't want burstable instances) is probably the better choice. But I suppose it depends on availability in their specific zones.
tenplusfive 2y ago
At least with other instance types (m5,m6,t3) it was the case that the AMD processors were cheaper. As it turns out, this does not seem to be a general rule. It seems like performance wise the AMD processors are (in certain workloads) quite a bit faster than their Intel equivalent: https://www.phoronix.com/review/aws-m7a-ec2-benchmarks/2 https://www.phoronix.com/review/aws-m7a-ec2-benchmarks/2 (in later pages it seems to be a little bit more mixed)

broknbottle 2y ago

minimizing the image definitely helps. https://aws.amazon.com/blogs/apn/how-to-build-sparse-ebs-volumes-for-fun-and-easy-snapshotting/ https://aws.amazon.com/blogs/apn/how-to-build-sparse-ebs-vol... https://netflixtechblog.medium.com/datastore-flash-upgrades-187f1e4ef859 https://netflixtechblog.medium.com/datastore-flash-upgrades-... https://github.com/Netflix-Skunkworks/s3-flash-bootloader https://github.com/Netflix-Skunkworks/s3-flash-bootloader

orf 2y ago

It seems that you want to make your root volume as small as possible, and use it to only attach a pre-warmed pool of EBS volumes at launch time that contain the actual config/data you need? You can launch a stripped down distribution with what, a 200mb disk? Then attach the “useful” EBS volume, and “do stuff” with that - launch a container, or whatever.

elchief 2y ago

I've noticed that Amazon linux 2023 boots faster than Ubuntu too

albert_e 2y ago

AWS will (should) make this an optional feature. Often the technology is the easier part. The difficult part is how to name the feature intuitively, adding to an ocean of jargon and documentation, and making the configuration knobs intuitive both in UI and CLI/SDK. Amazon Simple Compute Service :) ?

kylegalbraith 2y ago
Other founder of Depot here. AWS is pretty close to this idea with their Warm Pools [0]. But for our use case, they're just too slow to react to changes. We observed 60s+ to notice a change and actually start the machine. That doesn't work when we need to launch the machine as quickly as possible in reaction to a pending GHA job. That said, I think this is a problem they could likely solve with that functionality, and we'd love to use it. [0] https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-warm-pools.html https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-au...
immibis 2y ago
AWS Lambda.
ldoughty 2y ago
The article talks about fio to warm the drive... That's basically fast snapshot restore[0]. This reduces the "first access" penalty for "dirty" blocks. This is probably the slowest part of the entire article (it's about 10 seconds per dirty GB to fio the disk). [0]: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-fast-snapshot-restore.html https://docs.aws.amazon.com/ebs/latest/userguide/ebs-fast-sn...

nathants 2y ago

in the us-west-2-lax-1a local zone, i just booted 100 r5.xlarge spot instances as fortnite like game servers[1]. 1 to be a central server, 99 to be fake players. the server broadcasts x100 write amplified data from every player to every player. the 101st serve is my local pc. the server broadcasts at 200 MB/s[2]. the whole setup costs me $3-4 usd/hour and by far the slowest part of boot is my game compiling on the central server, whether i store ccache data in s3 or not. i've booted this every day for the last 6 months, to test the game. if your system can't handle 30s vm boots, your system should improve. 1. https://r2.nathants.workers.dev/ec2_snitch.png https://r2.nathants.workers.dev/ec2_snitch.png 2. https://r2.nathants.workers.dev/ec2_boot.mp4 https://r2.nathants.workers.dev/ec2_boot.mp4

solatic 2y ago

Makes me wonder why Depot isn't moving to on-prem hardware. When you're reselling compute with a better API, you give up a substantial proportion of your profits to the hyperscaler while offering worse performance (due to being held hostage to the hyperscaler's design decisions, like lazy loading root EBS from S3). Surely an optimized approach here looks something like booting customer CI workloads directly from the hypervisor, using an ISO/squashfs/etc. stored directly on the hypervisor, where the only networked disks are the ones with the customers' BuildKit caches?

gtirloni 2y ago

> tl;dr — boot the instance once, shut the instance down, then boot it again when needed their own tldr should be at the top not middle of the article :)

deleted 2y ago

[deleted]

pid-1 2y ago

Can you have warm pools of spot instances?

thefaux 2y ago

Whenever I see the flaws in aws ux, I remember that they bill by the hour.

mediumsmart 2y ago

Very cool. How many seconds of the faster boot time fit into one regular second?