3 ms·

Perhaps someone could give me some advice? I work alongside a small team maintaining quite a lot of machines on AWS. They're struggling (IMHO) to manually appl

by vosper 5y ago

Perhaps someone could give me some advice? I work alongside a small team maintaining quite a lot of machines on AWS. They're struggling (IMHO) to manually apply all of the security patches their scanning tool identifies. My theory is that Amazon Linux gets patched frequently, and so they'd be better off spending time normalizing our EC2 infra so that every instance is running Amazon Linux, and then work on an easy rollout mechanism to deploy the latest version. Has anyone got any thoughts on this? It wouldn't obviate the need for patching completely, but I feel like AWS is already doing some of this work for us, so we should take advantage.

rusteh1 5y ago
Yes, one of the core benefits of a provider like AWS is that they provide tooling to treat individual instances as immutable entities that you simply replace without any interruption to your users. You should focus on expressing the infrastructure as code and using mechanisms like ASGs to roll out new instances based on the latest Amazon provided AMIs.
windowsworkstoo 5y ago
Yep we do this, works good - you can either trigger a server refresh from SNS (AWS notifies you of certain AMI updates) or we just rebuild our underlying fleet each week with the most current AL2 AMI
voidfunc 5y ago
Every mainstream upstream Linux vendor is continuously pushing updated AMIs. It shouldn’t really matter whether you solve this with Ubuntu or Amazon Linux or RHEL/CentOS. Sounds like you need a better process / automation for rolling updates. Either continuously rebuilt golden images or rolling security patches, or turning on your distros unattended upgrade mechanism could be solutions depending on your environment.
gtirloni 5y ago
If you can, definitely standardize on as few distros as possible. It'll make applying patches (and learning when things go wrong, because they will) much easier. We used to have all sorts of distros that people just felt like using without worrying about their maintainability. We kept fighting fires to keep everything running. Once we standardized on a single distro (CentOS at the time), everything started working much more smoothly. We could have picked Debian, Ubuntu, it doesn't matter. That being said, Amazon Linux 2 is pretty well maintained. Most things (all?) that work on RHEL, will work on it. You may need to use 3rd-party repos if you want really newer stuff (eg. PHP) but that's inherent to such LTS releases. That situation is expected to improve with the improvements that adopting Fedora brings in AL2022 but I need to catch up.
LilBytes 5y ago
For those few AMI's that are long lived, AWS SSM Patch Manager is your friend. Naturally take care to roll out patches in a rolling block, you don't want to apply a broken patch everywhere in the same day :) https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-patch.html https://docs.aws.amazon.com/systems-manager/latest/userguide...
- belval 5y ago
  I second this, we use it to manage a bigger fleet with a few hundred machines. One thing to keep in mind though is that it will not apply kernel updates (as those require a reboot) so you still need to account for it.
  - podc 5y ago
    How do you apply kernel updates? Since rebooting anyway do you just throw the ec2 instance away at this point and launch a new one with latest image (and thus kernel updates)?
  - mcbain 5y ago
    I haven’t tried this yet (our instances need other changes to get to this point) but AL2 can support live kernel patching: https://docs.aws.amazon.com/systems-manager/latest/userguide/kernel-live-patching.html https://docs.aws.amazon.com/systems-manager/latest/userguide...
hayd 5y ago
Recently we moved to use EC2 Image Builder, and it's been working great: https://aws.amazon.com/image-builder/ https://aws.amazon.com/image-builder/ Note: We were already mostly Launch Templates/ASGs, so updates are always new instances (rather than patching long-running ones).
- LilBytes 5y ago
  For the life of me I never got Image Builder working in a decent state. I opted for Packer and I've been very happy with it. Though with that said I'm still using AWS SSM Patch Manager for a few outliers that are long lived. Like. You, Okta AD agent that can only be programmatically installed using AHK. :-/
  - hayd 5y ago
    It was a little strange to set up, I remember it taking a while/a lot of experimentation... But in the end it's just running userdata, and/or "component" scripts, and baking that into the AMI. It's been happily updating and switching out Launch Template versions for our ASGs (for reasons each pipeline can only push to 5 LTs). I guess I should write up a blogpost, because... the documentation is kinda garbage. I never got around to using packer properly so can't compare.

by vosper 5y ago

rusteh1 5y ago

Yes, one of the core benefits of a provider like AWS is that they provide tooling to treat individual instances as immutable entities that you simply replace without any interruption to your users. You should focus on expressing the infrastructure as code and using mechanisms like ASGs to roll out new instances based on the latest Amazon provided AMIs.

windowsworkstoo 5y ago

Yep we do this, works good - you can either trigger a server refresh from SNS (AWS notifies you of certain AMI updates) or we just rebuild our underlying fleet each week with the most current AL2 AMI

voidfunc 5y ago

Every mainstream upstream Linux vendor is continuously pushing updated AMIs. It shouldn’t really matter whether you solve this with Ubuntu or Amazon Linux or RHEL/CentOS. Sounds like you need a better process / automation for rolling updates. Either continuously rebuilt golden images or rolling security patches, or turning on your distros unattended upgrade mechanism could be solutions depending on your environment.

gtirloni 5y ago

If you can, definitely standardize on as few distros as possible. It'll make applying patches (and learning when things go wrong, because they will) much easier. We used to have all sorts of distros that people just felt like using without worrying about their maintainability. We kept fighting fires to keep everything running. Once we standardized on a single distro (CentOS at the time), everything started working much more smoothly. We could have picked Debian, Ubuntu, it doesn't matter. That being said, Amazon Linux 2 is pretty well maintained. Most things (all?) that work on RHEL, will work on it. You may need to use 3rd-party repos if you want really newer stuff (eg. PHP) but that's inherent to such LTS releases. That situation is expected to improve with the improvements that adopting Fedora brings in AL2022 but I need to catch up.

LilBytes 5y ago

For those few AMI's that are long lived, AWS SSM Patch Manager is your friend. Naturally take care to roll out patches in a rolling block, you don't want to apply a broken patch everywhere in the same day :) https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-patch.html https://docs.aws.amazon.com/systems-manager/latest/userguide...

belval 5y ago
I second this, we use it to manage a bigger fleet with a few hundred machines. One thing to keep in mind though is that it will not apply kernel updates (as those require a reboot) so you still need to account for it.
- podc 5y ago
  How do you apply kernel updates? Since rebooting anyway do you just throw the ec2 instance away at this point and launch a new one with latest image (and thus kernel updates)?
- mcbain 5y ago
  I haven’t tried this yet (our instances need other changes to get to this point) but AL2 can support live kernel patching: https://docs.aws.amazon.com/systems-manager/latest/userguide/kernel-live-patching.html https://docs.aws.amazon.com/systems-manager/latest/userguide...

hayd 5y ago

Recently we moved to use EC2 Image Builder, and it's been working great: https://aws.amazon.com/image-builder/ https://aws.amazon.com/image-builder/ Note: We were already mostly Launch Templates/ASGs, so updates are always new instances (rather than patching long-running ones).

LilBytes 5y ago
For the life of me I never got Image Builder working in a decent state. I opted for Packer and I've been very happy with it. Though with that said I'm still using AWS SSM Patch Manager for a few outliers that are long lived. Like. You, Okta AD agent that can only be programmatically installed using AHK. :-/
- hayd 5y ago
  It was a little strange to set up, I remember it taking a while/a lot of experimentation... But in the end it's just running userdata, and/or "component" scripts, and baking that into the AMI. It's been happily updating and switching out Launch Template versions for our ASGs (for reasons each pipeline can only push to 5 LTs). I guess I should write up a blogpost, because... the documentation is kinda garbage. I never got around to using packer properly so can't compare.