3 ms·
I don’t use EC2 enough to have played with this, but a big part here is the population of the AMI into the per-instance EBS volume. ISTM one could do much bett
by amluto 2y ago
I don’t use EC2 enough to have played with this, but a big part here is the population of the AMI into the per-instance EBS volume.
ISTM one could do much better with an immutable/atomic setup: set up an immutable read-only EBS volume, and have each instance share that volume and have a per-instance volume that starts out blank.
Actually pulling this off looks like it would be limited by the rules of EBS Multi-Attach. One could have fun experimenting with an extremely minimal boot AMI that streams a squashfs or similar file from S3 and unpacks it.
edit: contemplating a bit, unless you are willing to babysit your deployment and operate under serious constraints, EBS multi-attach looks like the wrong solution. I think the right approach would be build a very very small AMI that sets up a rootfs using s3fs or a similar technology and optionally puts an overlayfs on top. Alternatively, it could set up a block device backed by an S3 file and optionally use it as a base layer of a device-mapper stack. There’s plenty of room to optimize this.
- mdaniel 2y agoI believe they addressed this in their post because one cannot (currently?) `aws ec2 run-instances --volume-id vol-cafebabe`, rather one can only tell AWS what volume parameters to use when they create the root device. Your theory may still be sound about using some kind of super bare bones AMI but there will be no such outcome of "hey, friend, use this existing EBS as your root volume, don't create a new one"
- stingraycharles 2y agoIsn’t EBS multi-attach only available for the (very expensive) io1 / io2 volume types?
- amluto 2y agoHmm, it does look like it, although one could carefully use large IO. But the bigger issue might be durability. Most EBS types have rather low quoted durability, and, for a shared volume like this, that’s a problem. Using S3 instead would be better all around except for the smallish engineering effort and deployment effort needed. Getting a tool like mkosi to generate a boot-from-S3 setup should be straightforward. Converting most any bootable container should also be doable, even automatically. Converting an AMI would involve more heuristics and be more fragile, but it ought to work reliably with most modern Linux distros.
- Szpadel 2y agowe used s3fs in production. please don't use it, it's unreliable, unpredictable failure modes, can bring whole instance down. if you really need something like that use rclone mount
- attentive 2y agoThat's reinventing ebs/ami/snapshots. They are already doing it i.e. data goes lazily from s3 to ebs/ec2.
- amluto 2y agoIt’s not, though. The way to boot a transient OS (just like a transient instance of a container on a machine/instance with a container runtime) is to give userspace read-only access to the image. It can be outright read-only or it can be an actual efficient overlay mechanism (qcow, overlayfs, device-mapper snapshot, etc). EBS, as the article notes, can’t actually do a read-only mount of a snapshot at all, and it’s very inefficient at instantiating a volume from a snapshot.
- antihero 2y agoCould you make a snapshot of the booted instance then and boot other instances from that?
- amluto 2y agoThat seems like it would have exactly the same problem. The problem is that EBS volumes load very inefficiently from snapshots. (They’re also unnecessarily expensive: you pay for (number of instances times size) despite the fact that what you actually want is for each instance to read exactly the same data.) Fundamentally, EBS volumes are designed to work more or less like actual disks, whereas modern scaled workloads want something closer to net-booted diskless machines.