33 ms·

Bottlerocket, an open source Linux distribution built to run containers

347 points by sshroot 6y ago

LeSaucy 6y ago
Look out Rancher!
- chromedev 6y ago
  Rancher has been around a long time and isn't cloud-specific. Until this distro can prove itself useful outside of the AWS ecosystem, then Rancher will still dominate.
  - throwaway3neu94 6y ago
    Bottlerocket seems more like a competitor to RancherOS, which is abandonware. (https://github.com/rancher/os/issues/3000 https://github.com/rancher/os/issues/3000) I liked RancherOS's very few moving parts approach (even your shell is a Docker container). Hope Bottlerocket will be somewhat similar.
    - chromedev 6y ago
      I don't see any confirmation based on that link that it is abandoned. There was an update in June.
      - skolsuper 6y ago
        > RancherOS 1.x is currently in a maintain-only-as-essential mode. That is to say, it is no longer being actively maintained at a code level other than addressing critical or security fixes. quoting support.rancher.com, taken from the 4th comment down in the linked Github issue
        chromedev 6y ago
        It looks like that link is broken and I can't find anywhere on their website that states this anymore
solatic 6y ago
As strong as the engineering behind Bottlerocket seems to be, I'm not entirely sure who they built it for, except as a foundational component for AWS's managed offerings. If you, as an AWS customer, decide to fully embrace AWS lock-in, then why would you run this yourself on an EC2 instance instead of running ECS or EKS? If you're trying to avoid AWS lock-in, why would you choose an OS that's locking you into AWS Systems Manager and Amazon Linux 2 for debugging needs?
- acdha 6y ago
  There are varying levels of lock-in. A Linux distribution to run containers is a lot easier to replace than, say, a database with a proprietary query language custom semantics.
- captn3m0 6y ago
  >sure who they built it for Anytime AWS finds many of its customers using something outside of AWS, they want to build it in-house. AWS customers were using CoreOS a lot, hence this.
- coder543 6y ago
  If you're using an ECS or EKS Cluster, you still have to run some operating system on the ECS Container Instances. The containers have to run somewhere. There are a variety of options currently, including one from Amazon, but Bottlerocket seems designed to be a next-generation OS for container instances. It's extremely minimal, and designed to be as secure as possible, with transactional automatic updates. Ideally, I would just deploy Bottlerocket as the underlying OS on my ECS Cluster, and then never have to actually do any management of the Instance OSes whatsoever. I would deploy containers on top of my ECS Cluster, Bottlerocket would keep itself up to date and secure, and my containers would live happily ever after. It would hopefully feel more similar to using Fargate than not, except without paying the higher price for Fargate, and having access to the wider variety of hardware configurations available to regular ECS. Amazon has specifically said this in the Bottlerocket repo: > Bottlerocket is architected such that different cloud environments and container orchestrators can be supported in the future. It's mainly useful in concert with ECS or EKS right now, but it is architected to be useful in other places as well. I'm excited about Bottlerocket as a project, and I'm glad it's open source instead of just an opaque AMI that you can use on ECS or EKS.
senthilnayagam 6y ago
I remember reading last week linux plumber conference agreed to allow rust in linux kernel . but these guys have built an OS in rust.
- chromedev 6y ago
  Did you even read the blog post? This is running Linux, and only the AWS software-components running on it are largely written in Rust. It is still Linux and the OS is not written in Rust, nor the container daemon and probably none of the non-AWS software packages.
  - pjmlp 6y ago
    Most available userspace seems to be a mix of Rust and Go, except for existing stuff written in C like LinuxSE.
    - chromedev 6y ago
      What userspace do you see written in Rust?
      - pjmlp 6y ago
        From this thread https://news.ycombinator.com/item?id=24346300 https://news.ycombinator.com/item?id=24346300
        chromedev 6y ago
        That honestly sounds terrible, and apparently only works on ECS. What a shame. I was thinking that AWS was actually going to provide something useful to the open source community.
- djeiasbsbo 6y ago
  Well, they use a linux kernel which is written in C and some assembly. All of the overlying components are apparently written in rust. These components still interact with the system calls exposed by the kernel. An example of a kernel itself written in rust would be the Redox microkernel. It seems to be able to run a graphical environment but it hasn't been updated in a while.
haunter 6y ago
Wasn't Amazon Linux 2 something similar? Or I'm mixing it up https://aws.amazon.com/amazon-linux-2/ https://aws.amazon.com/amazon-linux-2/
- tootie 6y ago
  Seems like a similar goal, but the difference is that Bottlerocket is targeted at containers and Amazon Linux is targeted at EC2.
- acdha 6y ago
  Amazon Linux 2 is an AWS-optimized Linux distribution but it's a full distribution you're used to — RPM-based, generally compatible with RHEL/Centos, etc. — and that has the usual benefits and costs. You can customize it as much as any other Linux distribution but you're taking on the corresponding level of effort to secure and operate it. If all you run are containers, that's overhead for things you're not really using and it also means that you're going to be slower to update things like the kernel and system libraries since there's more to test and backwards compatibility is a big concern.
daxfohl 6y ago
So difference between this and Firecracker would be that the latter is boot-speed and overhead optimized, and this one is a bit heavier but more capable? If choosing between this and say Kata Containers plus Firecracker, the latter would be more secure because of VM isolation but this would be more efficient because multiple pods could go in a single VM? Is Bottlerocket secure enough to host multi-tenant workloads within the same VM?
- chromedev 6y ago
  Firecracker and Kata containers are ways to run containers inside lightweight VMs that boot directly into the Kernel, they are not Linux distros in themselves. You are trying to compare apples to oranges. You could run Firecracker or Kata containers on top of something like Bottlerocket, however Bottlerocket is geared more towards the container-sandbox and isolation crowd while Kata/Firecracker is for those who think you can only get isolation using VMs.
- kapilvt 6y ago
  multi-tenant workloads in the same vm.. thats sort of depends on your definition of 'secure', generally speaking for most enterprises the answer is no, the linux kernel is a huge surface attack space, firecracker and gvisor are different approaches at mitigating that. firecracker via a minimal hypervisor with legacy qemu stripped and a minimal exposed sys call/hypercall interface.. gvisor via reimplementing sys calls in golang intermediaries. both have tradeoffs, but my two cents would be around firecracker since its able to use more upstream kernel.
adolph 6y ago
Firecracker, Bottlerocket, starting to see a trend here https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/ https://aws.amazon.com/blogs/aws/firecracker-lightweight-vir...
- statictype 6y ago
  Both are written in Rust. Looks like Amazon is investing in the language quite a bit.
- b-rian 6y ago
  You mean, this trend? https://www.linuxha.com/bottlerocket/ https://www.linuxha.com/bottlerocket/ https://www.linuxha.com/bottlerocket/#firecracker_info https://www.linuxha.com/bottlerocket/#firecracker_info
aex 6y ago
Free project idea: A Qubes OS alternative built on Bottlerocket.
- chromedev 6y ago
  Why Bottlerocket? Why not just use Alpine, Void, NixOS, Arch, etc instead?
  - geek_at 6y ago
    I've fallen in love with Alpine. I'm using it on my bare bone servers running from a ram disk (USB drive) hosting Docker containers and even qemu VMs. All on encrypted zfs volumes. Totally love it
    - pcw888 6y ago
      Can you recommend any tutorials or blog posts on some of this? Really interested in it. Otherwise I'll search for it, thanks for the information.
      - chromedev 6y ago
        Here are some links for you that I think would be useful for a related setup, however I can't speak for the user that posted that. The headless installation should follow the same principles of using a RAMDisk though. https://wiki.alpinelinux.org/wiki/Raspberry_Pi_-_Headless_Installation https://wiki.alpinelinux.org/wiki/Raspberry_Pi_-_Headless_In... https://wiki.alpinelinux.org/wiki/Alpine_Linux_with_root_on_ZFS_with_native_encryption https://wiki.alpinelinux.org/wiki/Alpine_Linux_with_root_on_... https://wiki.alpinelinux.org/wiki/Docker https://wiki.alpinelinux.org/wiki/Docker https://wiki.alpinelinux.org/wiki/Install_Alpine_in_Qemu https://wiki.alpinelinux.org/wiki/Install_Alpine_in_Qemu
        pcw888 6y ago
        that's great, thank you!
DyslexicAtheist 6y ago
while "Amazon drivers are hanging smartphones in trees to get more work"[0] and while "Amazon is Hiring an Intelligence Analyst to Track 'Labor Organizing Threats'" and while "Amazon deletes job listings for analysts to track ‘labor organizing threats’"[2] ... in this thread we are celebrating these lizards for their Tech innovation. What's more scary than the damage to society the employees hiding in these companies do, is the cognitive dissonance we experience here of highly skilled individuals (who should know better) telling themselves "technology is neutral". [0] https://news.ycombinator.com/item?id=24342540 https://news.ycombinator.com/item?id=24342540 [1] https://news.ycombinator.com/item?id=24343361 https://news.ycombinator.com/item?id=24343361 [2] https://news.ycombinator.com/item?id=24345259 https://news.ycombinator.com/item?id=24345259
dpryden 6y ago
I'm confused about how the documentation recommends using a Kubernetes operator to manage OS updates. That seems weird and backwards to me. I would rather see an immutable OS AMI in an auto-scaled group, and just replace the node instance whenever there is an update. I can see a place for managing OS updates on an instance, but that seems more like "pets" than "cattle"... and I've always treated Kubernetes nodes like cattle, not pets. Isn't that the most common approach anyway?
- abhiyerra 6y ago
  I agree with you. This seems more complex than just having a auto scale group that auto rotates nodes after a certain amount of time and just picking a new update when the node launches.
  - NathanKP 6y ago
    I can provide a little background on this. In general yes I would recommend that you just use an ASG and roll out a new AMI. However that approach can be very expensive and time-consuming at truly massive scale (1000's or even 10's of thousands of machines). Bottlerocket is built in part based on our experiences operating AWS Fargate, which obviously has as one of its needs the ability to patch a colossal number of hosts which are running people's containers, without downtime or disrupting their containers. Bottlerocket is designed to ensure that this is both efficient and safe. We aren't the only ones with this need. Many large orgs also have tremendous fleets, and its unacceptable to cause significant disruption by rotating at the host level. Another aspect to consider is stateful workloads that are using the local disks. Bottlerocket lets you safely update your host if you are running something like a database or other stateful system where you don't really want to move your data around. Not everyone will need to use this updating mechanism, but I think it will be very attractive to many of the larger organizations with a lot of infrastructure.
    - GauntletWizard 6y ago
      It seems to me, not to be combative, that if Fargate can't afford the "noschedule: node is old" overhead and customers of Fargate can't handle their containers restarting on a regular basis, there's something wrong with your management engine or with their design and implementation. Much of the point of containerization is that you can roll containers often and run enough of them that you never have a single point of failure. What part of that assumption is broken that destroying machines regularly doesn't work?
moondev 6y ago
Cluster API does both! The operator rolls out immutable machine images that make up clusters. It's badass.
spicyusername 6y ago
So basically Amazon CoreOS.
trishankdatadog 6y ago
Little-known fact: like Google Fuchsia, Bottlerocket uses The Update Framework (TUF)[1][2] to securely update itself! [1] https://theupdateframework.io/ https://theupdateframework.io/ [2] https://github.com/awslabs/tough https://github.com/awslabs/tough
- kapilvt 6y ago
  tough is one of the actual oss gems behind bottle rocket thats reusable in non aws contexts.. bottle rocket probably can be but likely that will always be a second class citizen, and afaics completely undocumented for usage outside of aws atm.
  - jhaynes 6y ago
    Really glad you like our tough (TUF) library! As far as running in non-AWS contexts, we haven't had time to head down that path yet, but as I mentioned in an earlier post, we've tried to build Bottlerocket in a way that it can be extended to work outside of AWS either as VMs or even bare metal. In fact, a few of the engineers on the team have been playing with getting it running on their RaspberryPi's at home :)
waheoo 6y ago
I wish the push to containerise everything would just die. I need the pieces of my system to work together, not against one another, not contend for files and permissions.
robomaker2 6y ago
At my previous company we discarded the AWS-Linux distro and used rancherOS for container hosting because the version of yum they used was too flaky. They were unwilling to move to DNF to try and fix it. We've long badgered them for something like this (rancher style AWS-Linux dsitro) and they seem to have finally listened. Too bad, I moved to a different company and a different role to benefit from this. At least my old colleagues will be happy
jimmcslim 6y ago
So, is this available as an ISO? I currently run an Ubuntu VM in bhyve on my FreeNAS home-server to host various containers for experiments, etc... could I run this instead or is it tied to AWS?
- jhaynes 6y ago
  It isn't today, but we've built Bottlerocket in a way that it can be extended to work on bare metal or other places outside of AWS. There are a few issues on our github[1][2] that are similar that you're welcome to watch or contribute use cases to. [1] https://github.com/bottlerocket-os/bottlerocket/issues/841 https://github.com/bottlerocket-os/bottlerocket/issues/841 [2] https://github.com/bottlerocket-os/bottlerocket/issues/1097 https://github.com/bottlerocket-os/bottlerocket/issues/1097
- expertsteve 6y ago
  Talos (similar goals as bottlerocket) supports bare metal installs...
booleanbetrayal 6y ago
Can anyone at AWS comment on how this fits into the Fargate roadmap and Compute pricing? Presumably, a slimmer OS for things like EKS nodes could translate into some sort of Compute discount.
yarrel 6y ago
How exactly would they make a proprietary Linux distro?
viztor 6y ago
This reminds of CoreOS
peterwwillis 6y ago
I haven't dug into the engineering behind this yet, but my main concern with any custom Linux distribution is it often ends up as a waste of engineering. It's pretty easy to write your own Distro and pair it down to the essentials, and pairing it down that way allows you to strip out complexity, making it easier and more reliable to patch it. But that then means you are now maintaining this custom thing forever. If you're Amazon, that might be fine, but I suspect that this will be dropped when it is no longer profitable or a competing project supplants it - meaning in 5 years this thing might be gone. (A common theme of custom Linux distributions) And then there's troubleshooting. With a stripped-down distro, you will eventually need more tools to debug the thing, meaning you have to build and maintain packages to do that. Bottlerocket's answer to this is "run them in containers and use our API!", but I'm not sold on this. Have you ever tried to do debugging between host and container, or container to container? There's a lot of b.s. you have to hop through, and most Unix tools were not written with it in mind. I highly doubt that it will magically work for everything. If that's the case, then this "don't worry, because magic" idea is not really saving you work over maintaining a traditional OS. Moreover, you don't need a custom distro to do live patching. There are simple tricks you can use to juggle files and processes during a live patch, to say nothing of "checkpoint process && 'mv old_file new_file' && thaw process", etc. Kernels support live patching too. So if the argument is "well it's easier to patch", i'm not sure you're not trading away "easy" in one place for "pain in the ass" in another (see above). All of this also argues that it's just as effective to treat live-patched systems as you treat immutable infrastructure, and I'm not convinced of that argument either. The former is just more complex, and complexity attracts failures. Ultimately I think what you'll find is Bottlerocket will get a niche following, but also some people will get annoyed by it and go back to regular distros which already have well defined methods.
0_gravitas 6y ago
So, I'm a little confused; is this not what NixOS is all about, or is there a difference? (as my question probably suggests, im not all that knowledgeable about nix)
andrewrynhard 6y ago
For a project similar to bottlerocket, checkout https://github.com/talos-systems/talos https://github.com/talos-systems/talos. It is geared for cloud, VMware, and bare metal users. We have integrations with Cluster API for each, with the bare metal provider being our implementation: https://github.com/talos-systems/sidero https://github.com/talos-systems/sidero. Full disclosure, I am the CTO of Talos Systems.

347 points by sshroot 6y ago

LeSaucy 6y ago

Look out Rancher!

chromedev 6y ago
Rancher has been around a long time and isn't cloud-specific. Until this distro can prove itself useful outside of the AWS ecosystem, then Rancher will still dominate.
- throwaway3neu94 6y ago
  Bottlerocket seems more like a competitor to RancherOS, which is abandonware. (https://github.com/rancher/os/issues/3000 https://github.com/rancher/os/issues/3000) I liked RancherOS's very few moving parts approach (even your shell is a Docker container). Hope Bottlerocket will be somewhat similar.
  - chromedev 6y ago
    I don't see any confirmation based on that link that it is abandoned. There was an update in June.
    - skolsuper 6y ago
      > RancherOS 1.x is currently in a maintain-only-as-essential mode. That is to say, it is no longer being actively maintained at a code level other than addressing critical or security fixes. quoting support.rancher.com, taken from the 4th comment down in the linked Github issue
      - chromedev 6y ago
        It looks like that link is broken and I can't find anywhere on their website that states this anymore

solatic 6y ago

As strong as the engineering behind Bottlerocket seems to be, I'm not entirely sure who they built it for, except as a foundational component for AWS's managed offerings. If you, as an AWS customer, decide to fully embrace AWS lock-in, then why would you run this yourself on an EC2 instance instead of running ECS or EKS? If you're trying to avoid AWS lock-in, why would you choose an OS that's locking you into AWS Systems Manager and Amazon Linux 2 for debugging needs?

acdha 6y ago
There are varying levels of lock-in. A Linux distribution to run containers is a lot easier to replace than, say, a database with a proprietary query language custom semantics.
captn3m0 6y ago
>sure who they built it for Anytime AWS finds many of its customers using something outside of AWS, they want to build it in-house. AWS customers were using CoreOS a lot, hence this.
coder543 6y ago
If you're using an ECS or EKS Cluster, you still have to run some operating system on the ECS Container Instances. The containers have to run somewhere. There are a variety of options currently, including one from Amazon, but Bottlerocket seems designed to be a next-generation OS for container instances. It's extremely minimal, and designed to be as secure as possible, with transactional automatic updates. Ideally, I would just deploy Bottlerocket as the underlying OS on my ECS Cluster, and then never have to actually do any management of the Instance OSes whatsoever. I would deploy containers on top of my ECS Cluster, Bottlerocket would keep itself up to date and secure, and my containers would live happily ever after. It would hopefully feel more similar to using Fargate than not, except without paying the higher price for Fargate, and having access to the wider variety of hardware configurations available to regular ECS. Amazon has specifically said this in the Bottlerocket repo: > Bottlerocket is architected such that different cloud environments and container orchestrators can be supported in the future. It's mainly useful in concert with ECS or EKS right now, but it is architected to be useful in other places as well. I'm excited about Bottlerocket as a project, and I'm glad it's open source instead of just an opaque AMI that you can use on ECS or EKS.

senthilnayagam 6y ago

I remember reading last week linux plumber conference agreed to allow rust in linux kernel . but these guys have built an OS in rust.

chromedev 6y ago
Did you even read the blog post? This is running Linux, and only the AWS software-components running on it are largely written in Rust. It is still Linux and the OS is not written in Rust, nor the container daemon and probably none of the non-AWS software packages.
- pjmlp 6y ago
  Most available userspace seems to be a mix of Rust and Go, except for existing stuff written in C like LinuxSE.
  - chromedev 6y ago
    What userspace do you see written in Rust?
    - pjmlp 6y ago
      From this thread https://news.ycombinator.com/item?id=24346300 https://news.ycombinator.com/item?id=24346300
      - chromedev 6y ago
        That honestly sounds terrible, and apparently only works on ECS. What a shame. I was thinking that AWS was actually going to provide something useful to the open source community.
djeiasbsbo 6y ago
Well, they use a linux kernel which is written in C and some assembly. All of the overlying components are apparently written in rust. These components still interact with the system calls exposed by the kernel. An example of a kernel itself written in rust would be the Redox microkernel. It seems to be able to run a graphical environment but it hasn't been updated in a while.

haunter 6y ago

Wasn't Amazon Linux 2 something similar? Or I'm mixing it up https://aws.amazon.com/amazon-linux-2/ https://aws.amazon.com/amazon-linux-2/

tootie 6y ago
Seems like a similar goal, but the difference is that Bottlerocket is targeted at containers and Amazon Linux is targeted at EC2.
acdha 6y ago
Amazon Linux 2 is an AWS-optimized Linux distribution but it's a full distribution you're used to — RPM-based, generally compatible with RHEL/Centos, etc. — and that has the usual benefits and costs. You can customize it as much as any other Linux distribution but you're taking on the corresponding level of effort to secure and operate it. If all you run are containers, that's overhead for things you're not really using and it also means that you're going to be slower to update things like the kernel and system libraries since there's more to test and backwards compatibility is a big concern.

daxfohl 6y ago

So difference between this and Firecracker would be that the latter is boot-speed and overhead optimized, and this one is a bit heavier but more capable? If choosing between this and say Kata Containers plus Firecracker, the latter would be more secure because of VM isolation but this would be more efficient because multiple pods could go in a single VM? Is Bottlerocket secure enough to host multi-tenant workloads within the same VM?

chromedev 6y ago
Firecracker and Kata containers are ways to run containers inside lightweight VMs that boot directly into the Kernel, they are not Linux distros in themselves. You are trying to compare apples to oranges. You could run Firecracker or Kata containers on top of something like Bottlerocket, however Bottlerocket is geared more towards the container-sandbox and isolation crowd while Kata/Firecracker is for those who think you can only get isolation using VMs.
kapilvt 6y ago
multi-tenant workloads in the same vm.. thats sort of depends on your definition of 'secure', generally speaking for most enterprises the answer is no, the linux kernel is a huge surface attack space, firecracker and gvisor are different approaches at mitigating that. firecracker via a minimal hypervisor with legacy qemu stripped and a minimal exposed sys call/hypercall interface.. gvisor via reimplementing sys calls in golang intermediaries. both have tradeoffs, but my two cents would be around firecracker since its able to use more upstream kernel.

adolph 6y ago

Firecracker, Bottlerocket, starting to see a trend here https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/ https://aws.amazon.com/blogs/aws/firecracker-lightweight-vir...

statictype 6y ago
Both are written in Rust. Looks like Amazon is investing in the language quite a bit.
b-rian 6y ago
You mean, this trend? https://www.linuxha.com/bottlerocket/ https://www.linuxha.com/bottlerocket/ https://www.linuxha.com/bottlerocket/#firecracker_info https://www.linuxha.com/bottlerocket/#firecracker_info

aex 6y ago

Free project idea: A Qubes OS alternative built on Bottlerocket.

chromedev 6y ago
Why Bottlerocket? Why not just use Alpine, Void, NixOS, Arch, etc instead?
- geek_at 6y ago
  I've fallen in love with Alpine. I'm using it on my bare bone servers running from a ram disk (USB drive) hosting Docker containers and even qemu VMs. All on encrypted zfs volumes. Totally love it
  - pcw888 6y ago
    Can you recommend any tutorials or blog posts on some of this? Really interested in it. Otherwise I'll search for it, thanks for the information.
    - chromedev 6y ago
      Here are some links for you that I think would be useful for a related setup, however I can't speak for the user that posted that. The headless installation should follow the same principles of using a RAMDisk though. https://wiki.alpinelinux.org/wiki/Raspberry_Pi_-_Headless_Installation https://wiki.alpinelinux.org/wiki/Raspberry_Pi_-_Headless_In... https://wiki.alpinelinux.org/wiki/Alpine_Linux_with_root_on_ZFS_with_native_encryption https://wiki.alpinelinux.org/wiki/Alpine_Linux_with_root_on_... https://wiki.alpinelinux.org/wiki/Docker https://wiki.alpinelinux.org/wiki/Docker https://wiki.alpinelinux.org/wiki/Install_Alpine_in_Qemu https://wiki.alpinelinux.org/wiki/Install_Alpine_in_Qemu
      - pcw888 6y ago
        that's great, thank you!

DyslexicAtheist 6y ago

while "Amazon drivers are hanging smartphones in trees to get more work"[0] and while "Amazon is Hiring an Intelligence Analyst to Track 'Labor Organizing Threats'" and while "Amazon deletes job listings for analysts to track ‘labor organizing threats’"[2] ... in this thread we are celebrating these lizards for their Tech innovation. What's more scary than the damage to society the employees hiding in these companies do, is the cognitive dissonance we experience here of highly skilled individuals (who should know better) telling themselves "technology is neutral". [0] https://news.ycombinator.com/item?id=24342540 https://news.ycombinator.com/item?id=24342540 [1] https://news.ycombinator.com/item?id=24343361 https://news.ycombinator.com/item?id=24343361 [2] https://news.ycombinator.com/item?id=24345259 https://news.ycombinator.com/item?id=24345259

dpryden 6y ago

I'm confused about how the documentation recommends using a Kubernetes operator to manage OS updates. That seems weird and backwards to me. I would rather see an immutable OS AMI in an auto-scaled group, and just replace the node instance whenever there is an update. I can see a place for managing OS updates on an instance, but that seems more like "pets" than "cattle"... and I've always treated Kubernetes nodes like cattle, not pets. Isn't that the most common approach anyway?

abhiyerra 6y ago
I agree with you. This seems more complex than just having a auto scale group that auto rotates nodes after a certain amount of time and just picking a new update when the node launches.
- NathanKP 6y ago
  I can provide a little background on this. In general yes I would recommend that you just use an ASG and roll out a new AMI. However that approach can be very expensive and time-consuming at truly massive scale (1000's or even 10's of thousands of machines). Bottlerocket is built in part based on our experiences operating AWS Fargate, which obviously has as one of its needs the ability to patch a colossal number of hosts which are running people's containers, without downtime or disrupting their containers. Bottlerocket is designed to ensure that this is both efficient and safe. We aren't the only ones with this need. Many large orgs also have tremendous fleets, and its unacceptable to cause significant disruption by rotating at the host level. Another aspect to consider is stateful workloads that are using the local disks. Bottlerocket lets you safely update your host if you are running something like a database or other stateful system where you don't really want to move your data around. Not everyone will need to use this updating mechanism, but I think it will be very attractive to many of the larger organizations with a lot of infrastructure.
  - GauntletWizard 6y ago
    It seems to me, not to be combative, that if Fargate can't afford the "noschedule: node is old" overhead and customers of Fargate can't handle their containers restarting on a regular basis, there's something wrong with your management engine or with their design and implementation. Much of the point of containerization is that you can roll containers often and run enough of them that you never have a single point of failure. What part of that assumption is broken that destroying machines regularly doesn't work?

moondev 6y ago

Cluster API does both! The operator rolls out immutable machine images that make up clusters. It's badass.

spicyusername 6y ago

So basically Amazon CoreOS.

trishankdatadog 6y ago

Little-known fact: like Google Fuchsia, Bottlerocket uses The Update Framework (TUF)[1][2] to securely update itself! [1] https://theupdateframework.io/ https://theupdateframework.io/ [2] https://github.com/awslabs/tough https://github.com/awslabs/tough

kapilvt 6y ago
tough is one of the actual oss gems behind bottle rocket thats reusable in non aws contexts.. bottle rocket probably can be but likely that will always be a second class citizen, and afaics completely undocumented for usage outside of aws atm.
- jhaynes 6y ago
  Really glad you like our tough (TUF) library! As far as running in non-AWS contexts, we haven't had time to head down that path yet, but as I mentioned in an earlier post, we've tried to build Bottlerocket in a way that it can be extended to work outside of AWS either as VMs or even bare metal. In fact, a few of the engineers on the team have been playing with getting it running on their RaspberryPi's at home :)

waheoo 6y ago

I wish the push to containerise everything would just die. I need the pieces of my system to work together, not against one another, not contend for files and permissions.

robomaker2 6y ago

At my previous company we discarded the AWS-Linux distro and used rancherOS for container hosting because the version of yum they used was too flaky. They were unwilling to move to DNF to try and fix it. We've long badgered them for something like this (rancher style AWS-Linux dsitro) and they seem to have finally listened. Too bad, I moved to a different company and a different role to benefit from this. At least my old colleagues will be happy

jimmcslim 6y ago

So, is this available as an ISO? I currently run an Ubuntu VM in bhyve on my FreeNAS home-server to host various containers for experiments, etc... could I run this instead or is it tied to AWS?

jhaynes 6y ago
It isn't today, but we've built Bottlerocket in a way that it can be extended to work on bare metal or other places outside of AWS. There are a few issues on our github[1][2] that are similar that you're welcome to watch or contribute use cases to. [1] https://github.com/bottlerocket-os/bottlerocket/issues/841 https://github.com/bottlerocket-os/bottlerocket/issues/841 [2] https://github.com/bottlerocket-os/bottlerocket/issues/1097 https://github.com/bottlerocket-os/bottlerocket/issues/1097
expertsteve 6y ago
Talos (similar goals as bottlerocket) supports bare metal installs...

booleanbetrayal 6y ago

Can anyone at AWS comment on how this fits into the Fargate roadmap and Compute pricing? Presumably, a slimmer OS for things like EKS nodes could translate into some sort of Compute discount.

yarrel 6y ago

How exactly would they make a proprietary Linux distro?

viztor 6y ago

This reminds of CoreOS

peterwwillis 6y ago

I haven't dug into the engineering behind this yet, but my main concern with any custom Linux distribution is it often ends up as a waste of engineering. It's pretty easy to write your own Distro and pair it down to the essentials, and pairing it down that way allows you to strip out complexity, making it easier and more reliable to patch it. But that then means you are now maintaining this custom thing forever. If you're Amazon, that might be fine, but I suspect that this will be dropped when it is no longer profitable or a competing project supplants it - meaning in 5 years this thing might be gone. (A common theme of custom Linux distributions) And then there's troubleshooting. With a stripped-down distro, you will eventually need more tools to debug the thing, meaning you have to build and maintain packages to do that. Bottlerocket's answer to this is "run them in containers and use our API!", but I'm not sold on this. Have you ever tried to do debugging between host and container, or container to container? There's a lot of b.s. you have to hop through, and most Unix tools were not written with it in mind. I highly doubt that it will magically work for everything. If that's the case, then this "don't worry, because magic" idea is not really saving you work over maintaining a traditional OS. Moreover, you don't need a custom distro to do live patching. There are simple tricks you can use to juggle files and processes during a live patch, to say nothing of "checkpoint process && 'mv old_file new_file' && thaw process", etc. Kernels support live patching too. So if the argument is "well it's easier to patch", i'm not sure you're not trading away "easy" in one place for "pain in the ass" in another (see above). All of this also argues that it's just as effective to treat live-patched systems as you treat immutable infrastructure, and I'm not convinced of that argument either. The former is just more complex, and complexity attracts failures. Ultimately I think what you'll find is Bottlerocket will get a niche following, but also some people will get annoyed by it and go back to regular distros which already have well defined methods.

0_gravitas 6y ago

So, I'm a little confused; is this not what NixOS is all about, or is there a difference? (as my question probably suggests, im not all that knowledgeable about nix)

andrewrynhard 6y ago

For a project similar to bottlerocket, checkout https://github.com/talos-systems/talos https://github.com/talos-systems/talos. It is geared for cloud, VMware, and bare metal users. We have integrations with Cluster API for each, with the bare metal provider being our implementation: https://github.com/talos-systems/sidero https://github.com/talos-systems/sidero. Full disclosure, I am the CTO of Talos Systems.