3 ms·

I don't get why they're using EBS here to begin with. EBS trades off cost and performance for durability. It's slow because it's a network-attached volume that'

I don't get why they're using EBS here to begin with. EBS trades off cost and performance for durability. It's slow because it's a network-attached volume that's most likely also replicated under the hood. You use this for data that you need high durability for. It looks like their use-case fetches all the data it needs from the network (in the form of the GH Actions runner getting the job from GitHub, and then pulling down Docker containers, etc). What they need is a minimal Linux install (Arch Linux would be good for this) in a squashfs/etc and the only thing in EBS should be an HTTP-aware boot loader like IPXE or a kernel+initrd capable of pulling down the squashfs from S3 and run it from memory. Local "scratchspace" storage for the build jobs can be provided by the ephemeral NVME drives which are also direct-attach and much faster than EBS.

jedberg 2y ago
By using EBS they don't have to wait for disk to fill from network on second+ boot.
- Nextgrid 2y ago
  Ah so they are keeping the machines around? Do they need to do that - does the GH runner actually persist anything worth keeping in between runs?
  - jedberg 2y ago
    They keep the instances in a "stopped" state, which means keeping the EBS volume around (and paying for it) but not paying for the instance (which could be another machine when turn it back on, which is why you can't load it into scratch space and then stop it). What's on the EBS is their docker image, so they don't have to load it back up again.
    - Nextgrid 2y ago
      Makes sense. I still think it would be cheaper to just reload it from S3 (straight into memory, not using EBS at all) on every boot. The entire OS shouldn't be more than a gigabyte which is quite fast to download as a bulk transfer straight into RAM.
      - jedberg 2y ago
        Yes it would be cheaper, but the whole point of this article is trading off cost for faster boot times. They address your points in the article, how it's faster to boot off a warm EBS instead of loading from scratch.

jedberg 2y ago

By using EBS they don't have to wait for disk to fill from network on second+ boot.

Nextgrid 2y ago
Ah so they are keeping the machines around? Do they need to do that - does the GH runner actually persist anything worth keeping in between runs?
- jedberg 2y ago
  They keep the instances in a "stopped" state, which means keeping the EBS volume around (and paying for it) but not paying for the instance (which could be another machine when turn it back on, which is why you can't load it into scratch space and then stop it). What's on the EBS is their docker image, so they don't have to load it back up again.
  - Nextgrid 2y ago
    Makes sense. I still think it would be cheaper to just reload it from S3 (straight into memory, not using EBS at all) on every boot. The entire OS shouldn't be more than a gigabyte which is quite fast to download as a bulk transfer straight into RAM.
    - jedberg 2y ago
      Yes it would be cheaper, but the whole point of this article is trading off cost for faster boot times. They address your points in the article, how it's faster to boot off a warm EBS instead of loading from scratch.